07 Module · Why Ollama

The prompt is
a data transfer.

The interesting question about AI in a European company is rarely capability. It is jurisdiction — and once you accept that framing, the architecture follows.

First released: 2023
Written in: Go
Endpoint: OpenAI-compatible
Licence: MIT

A prompt containing an unreleased product plan, a patient record or a client's source code is a data transfer. Calling it a feature does not change what it is.

This is the conversation we keep having. An engineering team adopts a hosted assistant because it is obviously useful. Six months later somebody asks what has been pasted into it, and the honest answer is that nobody knows — because the interface made it frictionless and nothing logged it.

The usual response is a policy: a document telling people what they may not paste. Policies of that shape have a poor record against a tool that is genuinely helpful and one keystroke away.

The architectural response is to move the model inside the perimeter. Then the policy question stops being "what may leave?" and becomes "what, specifically, is worth sending out?" — a much smaller question, with a much shorter list.

Argumenten · The case

Seven reasons,
in order of how often they matter.

Ranked by what decides the procurement conversation, not by what is most interesting technically.

01 Nothing crosses the perimeter

Prompts, source code, documents and customer data stay on hardware you own. The data-protection conversation becomes a description of the network rather than an argument about a sub-processor's sub-processor.

02 It speaks the protocol you already use

An OpenAI-compatible endpoint means existing editors, scripts and libraries point at your own machine with one changed base URL. Nothing above it needs rewriting, and nothing is locked to us.

03 The cost is electricity, not tokens

One GPU serves a team all day for the price of running it. Nobody rations usage, nobody watches a meter, and the finance conversation happens once at procurement rather than monthly.

04 Model management is a package manager

ollama pull, and the weights are on disk. Pinning a specific model version is trivial, which matters when a hosted model changes underneath you and last month's prompts quietly stop working.

05 Latency is a local network

No round trip, no rate limit, no provider incident taking your tooling down. For editor integrations, where the model runs on every keystroke pause, this is the difference between useful and abandoned.

06 Retrieval over your own documents

Embedding an internal corpus locally means the index never leaves either. A question answered against your own handbook, contracts or codebase — with nothing uploaded to build the index.

07 No training-data question

There is no clause to read about whether your inputs are retained or used for training, because there is no counterparty. That absence is the entire product.

Jämförelse · Side by side

Against a hosted API

We are not pretending these are equivalent. The point of the table is to show exactly where the trade sits.

Concern	Hosted frontier API	Ollama, on your GPU
Raw capability	Higher — meaningfully so	Good, and closing
Where the prompt goes	Their infrastructure	Your rack
Data-protection paperwork	DPA, SCCs, a transfer assessment	None — no transfer
Cost model	Per token, forever	Capital, then electricity
Cost at heavy team use	Grows with adoption	Flat
Latency	Internet round trip	Local network
Availability	Their status page	Your status page
Model stability	Changes underneath you	Pinned until you move it
Context window	Very large	Smaller, and VRAM-bound
Upfront work	An API key	Hardware, sizing, deployment

Hur vi driftar · Our install

What a Chenborg Ollama actually contains.

Most of the value is in the sizing and the wiring, not the install. Ollama takes an afternoon; choosing hardware that matches the models your team will really use, and connecting it to where the work happens, is the engagement.

GPU host sized against your actual workload, procured or rented at a European provider
Ollama declared in NixOS, so the machine is reproducible like every other one
An OpenAI-compatible endpoint on your WireGuard network — never published
Retrieval over your own repositories and documents, indexed locally
Integrations where the work is: editors, Emacs, Redmine's REST API, internal search
A written acceptable-use and data-handling note your DPO can actually sign

The Ollama engagement in detail

ollama.nix

# The GPU host, declared once.
services.ollama = {
  enable            = true;
  acceleration      = "cuda";
  host              = "10.44.0.9";  # wg0 only
  loadModels        = [
    "qwen3:32b"
    "nomic-embed-text"
  ];
  environmentVariables = {
    OLLAMA_KEEP_ALIVE = "30m";
  };
};

# not reachable from the internet,
# by construction rather than by policy
networking.firewall.interfaces.wg0
  .allowedTCPPorts = [ 11434 ];

Ärligt talat · Honestly

The gap is real, and here it is.

Open-weight models running on one GPU are not frontier models. Anyone who tells you otherwise is selling hardware. What matters is whether the gap falls on work you actually do.

Hard reasoning is where you feel it. Long multi-step problems, subtle architectural questions, unusual mathematics — a frontier model is meaningfully better and it is not close.
Context windows are smaller and VRAM-bound. Feeding an entire large repository into one prompt is a hosted-model habit. Locally you use retrieval instead, which is more work to build.
Somebody must own the machine. A GPU host needs patching, monitoring and capacity planning. It is a server, with everything that implies.
Hardware is capital. A GPU is a real purchase, and if adoption fails it is a real write-off. For a small team experimenting, a hosted API with a strict policy may honestly be the better first step.
The frontier moves. Local models are improving quickly, but so is the top. Expect to revisit the sizing every eighteen months rather than treating it as settled.

Where the gap does not matter is the bulk of daily work: summarising, classifying, drafting, rewriting, explaining unfamiliar code, and answering questions against internal documents. That is most of it, and it is the part that touches your most sensitive material.

So the honest recommendation is usually both: local by default for anything touching your own data, and a clear, written line about the specific work that may leave the building. Drawing that line is part of what we deliver.