Ollama

Ollama runs open-weight language models on hardware you own. It is the reason we can answer “can we use AI?” without the answer being “yes, by sending your source code to another jurisdiction”.

Why we still choose it

Because the interesting question about AI in a European company is rarely capability — it is jurisdiction. A prompt containing an unreleased product plan, a patient record or a client’s source code is a data transfer, and calling it a feature does not change that.

Ollama makes the local option genuinely practical: model management that works like a package manager, and an OpenAI-compatible endpoint, so existing tooling points at your own machine with a changed base URL and nothing else.

How we run it

Declaratively on NixOS, on a GPU host sized to the models you actually need, exposed on your WireGuard network so nothing is published. Then connected to where the work happens: editors, Emacs, Redmine’s REST API, internal search, and retrieval over your own document store.

The monthly cost is electricity rather than tokens, and the compliance conversation is short.

The honest limit

Local open-weight models are not frontier models. For summarising, classifying, drafting and answering questions about internal documents, the gap does not matter much. For the hardest reasoning work it does. We help draw a clear line about which work may leave the building rather than pretending the line does not exist.