Chapter 5: Ollama and Local Sovereignty¶

If you have never run a large language model locally, the first time you do it is a small shock. Not because it is difficult. Because it is easy. You install Ollama, pull a model, and within twenty minutes you have a capable LLM answering questions without a network request leaving your machine. The shock is that it works this well on consumer hardware.

This chapter is about that setup in production: the specific configuration on inference-host-1, the models Bob uses, and the patterns that make local inference operationally reliable.

Why Local Inference Wins for Regulated Data¶

The argument for cloud LLMs is convenience and capability. The largest models available through APIs are larger than anything you can reasonably run locally on a single GPU. If you need GPT-4-class capability for a general-purpose assistant, the cloud is still ahead on raw benchmark scores.

For SQL Server diagnostic analysis, that argument does not apply. The task is constrained. You are not asking the model to write poetry or translate Mandarin legal documents. You are giving it a structured diagnostic payload, wait stats, query plans, DMV snapshots, and asking it to reason within a well-defined domain. For that task, a well-tuned 14B to 32B parameter model running locally is competitive with much larger cloud models, and in some cases beats them because the local model can be prompted with more context without worrying about token costs.

At Dogsbarking, where the work from May 2024 through March 2026 involved managing deduplication pipelines for enterprise Salesforce clients, the data moving through the system was contact records, transaction history, and account data from companies operating under GDPR. Using GPT-4 or Claude 3 to assist with entity resolution code was fine, that was developer tooling, and the code itself was not customer data. But the actual deduplication work happened on records. If you wanted an LLM to help analyze why two records were not matching correctly, you had to show it the records. Showing it the records meant sending personally identifiable information to an external API. Under GDPR Article 28, that triggers a Data Processing Agreement with the AI vendor, a review of their subprocessor list, documentation in your Article 30 records of processing, and possibly notification to the relevant supervisory authority depending on the data category. For a client in the EU, that is not a theoretical compliance exercise. It is a real legal obligation with real consequences if you skip it.

At Entergy via ComTec, from May 2020 through April 2024, the constraint was NERC CIP. North American Electric Reliability Corporation Critical Infrastructure Protection standards classify information about bulk electric system assets, configurations, and operational state as Protected Cyber Asset data. Sending a SQL query plan that references a table named GridOperationsHist or TransmissionLineMonitor to an external API is a CIP-002 and CIP-004 conversation at minimum. The standard does not list "LLM inference API calls" as a prohibited channel because NERC CIP was not written with LLM APIs in mind. But the underlying principle, that protected asset information should not traverse uncontrolled external paths, is clear. During the audit period, 100% compliance meant no ambiguous interpretations. If there was a reasonable argument that something violated the standard, you did not do it.

The local inference setup eliminates both problems by eliminating the external transmission entirely.

Three concrete data-residency wins for organizations in regulated industries:

Audit trail completeness. When query plans and T-SQL text never leave your network, your audit trail is entirely within your control. There is no third-party data processor to include in your GDPR data processing records, no cloud provider's retention policy to worry about, no subpoena that can reach data you never sent anywhere. Your CISO can sign off on "we process this data on our own hardware" with confidence.
Zero token-cost exposure. A cloud LLM charges per token. Bob runs monitoring cycles continuously, potentially dozens of times per hour across multiple SQL Server instances. At cloud API pricing, a high-frequency monitoring use case becomes expensive quickly. The marginal cost of one more monitoring cycle on inference-host-1 is negligible after the hardware is purchased.
Network isolation for sensitive schemas. Database schema names, stored procedure names, and column names appear in query plans and diagnostic output. Even when data values are excluded from the diagnostic payload, schema information often has sensitivity. In financial services, healthcare, or any environment where schema design is considered proprietary, sending that information to an external API is a risk that requires a vendor agreement, privacy review, and ongoing monitoring. With local inference, the schema stays local.

It is worth being honest about where cloud LLMs still belong in this picture. At Dogsbarking, I used GitHub Copilot, GPT-4, Claude 3, and Gemini daily for developer work: writing Python, generating T-SQL patterns, reviewing code logic, drafting documentation. Those tools are excellent for that purpose. The code they help produce is not sensitive. The prompts are not customer data. The seam between "AI as developer tool" and "AI in the production data path" is important.

The distinction is about what data you are putting in the model's context. Code patterns and technical questions are fine for external APIs. Production query plans with table names that reveal regulated schemas are not. Bob sits in the production data path. That is where the line is.

The `inference-host-1` Configuration¶

The primary inference host runs Ubuntu 25.10 with Ollama installed as a systemd service. Here is the unit file in production:

# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Server
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=ollama
Group=ollama
ExecStart=/usr/local/bin/ollama serve
Restart=always
RestartSec=3
Environment=OLLAMA_HOST=0.0.0.0:11434
Environment=OLLAMA_ORIGINS=*
Environment=OLLAMA_MODELS=/var/lib/ollama/models
Environment=OLLAMA_NUM_PARALLEL=2
Environment=OLLAMA_MAX_LOADED_MODELS=2

[Install]
WantedBy=multi-user.target
# source: bob@c549c88

The .env file that Bob's agent reads to locate the inference host:

// ollama_config.json
{
  "url": "http://10.0.0.70:11434",
  "model": "gemma4:26b",
  "timeout": 900,
  "num_ctx": 16384,
  "temperature": 0.7,
  "keep_alive": "10m"
}
// source: bob@c549c88

The failover chain lives in llm_config.json:

// llm_config.json (failover chain)
{
  "failover_chain": [
    {"provider": "ollama", "priority": 0, "url": "http://10.0.0.70:11434", "model": "gemma4:26b"},
    {"provider": "ollama", "priority": 1, "url": "http://10.0.0.71:11434", "model": "qwen2.5-14b:latest"}
  ],
  "default_cooldown_seconds": 60,
  "max_cooldown_seconds": 900
}
// source: bob@c549c88

The one-line health check for inference-host-1:

curl -sf http://10.0.0.70:11434/api/tags | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK', len(d['models']), 'models')" || echo "FAIL"
# source: bob@c549c88

This returns something like OK 8 models when the service is healthy. The CI pipeline runs this check before any test that requires inference.

Model Discovery with `/api/tags`¶

Ollama's /api/tags endpoint returns a JSON list of every model that has been pulled to the host. Bob queries this at startup to verify the models it needs are present and to emit a warning if the preferred model is missing and a fallback is available.

import httpx
import os

def discover_models(host: str) -> dict[str, dict]:
    """Return a dict of model_name -> model_metadata from the Ollama host."""
    response = httpx.get(f"{host}/api/tags", timeout=10)
    response.raise_for_status()
    models = {}
    for m in response.json().get("models", []):
        models[m["name"]] = {
            "size": m.get("size", 0),
            "modified_at": m.get("modified_at", ""),
            "details": m.get("details", {}),
        }
    return models

# source: bob@c549c88

At startup, Bob calls discover_models on the primary host. If the primary is unreachable, the _ollama_call() function automatically walks the llm_config.json failover chain, trying each server in priority order. Each server has its own default model configured separately: gemma4:26b on .70, qwen2.5-14b on .71. The two hosts carry different model sets. The active server and model are tracked globally so the dashboard can show which host is currently serving inference.

Hot-Swapping Models Without Restarting the Agent¶

A key operational requirement for production monitoring is the ability to change which model is being used without stopping the agent. This matters during A/B testing of model responses, when a better model is pulled to the host, or when a model is generating poor-quality diagnostics and needs to be replaced with a fallback.

Bob reads its active model from ollama_config.json. Updating that file and using the /api/settings/ollama REST endpoint, or the web dashboard's Models tab, is enough to change the active model. The change takes effect on the next LLM call.

The hot-swap sequence:

# 1. Pull the new model (can run while agent is active)
curl -X POST http://10.0.0.70:11434/api/pull \
  -d '{"name": "gemma4:27b"}'

# 2. Verify it appears in the model list
curl -sf http://10.0.0.70:11434/api/tags | \
  python3 -c "import sys,json; [print(m['name']) for m in json.load(sys.stdin)['models']]"

# 3. Update Bob's model via the REST API
curl -X POST http://localhost:8000/api/settings/ollama \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4:27b", "url": "http://10.0.0.70:11434"}'

# source: bob@c549c88

The model pull in step 1 happens in the background. Ollama supports concurrent pulls while inference is running. The old model remains loaded in memory until the pull completes and you send the reload signal. There is no gap in monitoring coverage during the swap.

This pattern also works for downgrading. If gemma4:26b is producing inconsistent diagnostic output for your specific workload and an alternate model performs better, the swap takes about two minutes.

There is a failure mode in the hot-swap process worth understanding. If you issue the settings API call before the model pull in step 1 has fully completed, Bob may attempt to call an incomplete model file. The failure looks like a context window error rather than a file corruption problem. The fix is to verify the model appears in /api/tags before updating Bob's configuration. Step 2 in the sequence handles this. If you skip it to save time, that is the failure you get.

What Was Tried and Rejected¶

gemma4:26b did not arrive as the default on day one. The model selection went through several iterations, and the rejections tell you as much as the final choice.

Earlier builds used qwen2.5 variants during the architecture's testing phase. Qwen2.5 models handle structured output cleanly and are fast on the RTX 3090. The series remains in the failover chain: qwen2.5-14b runs on inference-host-2 and is the automatic failover model when the primary is unreachable. The Qwen3-Coder variant is used specifically for Bob's evolution growth goals, the ones where Bob is actually writing Python code to extend his own capabilities, because code-specialized models produce better results for that narrow task.

Smaller models in the 7B–8B range were tested on inference-host-2 at the beginning, when the RTX 3090 was not yet in the picture. They are fast on the RTX 3060 and produce coherent structured output. The smaller models are adequate for simple triage, the kind where you have one dominant signal and a clear recommendation falls out. They lose resolution on multi-factor problems where the correct recommendation requires holding several concurrent findings in working context and reasoning about their interaction. Deadlock analysis is the clearest example: a deadlock involves at least two processes, two resources, and an ordering problem. Getting the victim chain right requires reasoning about all of those together, and the smaller models would frequently identify the victim correctly but get the contention source wrong.

The VRAM economics on the RTX 3090 work out cleanly. gemma4:26b fits comfortably in 24 GB of VRAM with headroom for a full diagnostic context window. The OLLAMA_MAX_LOADED_MODELS=2 setting is available if you want to keep a secondary model resident, but in practice the primary model stays loaded continuously because the load time from disk is several minutes and the monitoring workload does not justify evicting it between cycles.

Model Selection Philosophy¶

The production setup on inference-host-1 runs gemma4:26b as the default model for all agent work: chat, goal evaluation, monitoring diagnostics, and the Forge evolution pipeline. The reasoning: the model fits comfortably in 24 GB of VRAM, performs well on structured reasoning tasks, handles SQL constructs reliably, and produces output that is consistent enough to parse programmatically. Bob's own evolution pipeline has made 34 verified self-modifications to its reason() function while running against this model, so the model and the agent have effectively co-evolved.

The failover chain configured in llm_config.json:

gemma4:26b on 10.0.0.70:11434 (primary)
qwen2.5-14b on 10.0.0.71:11434 (automatic failover)

On inference-host-2 at .71, the RTX 3060 has 12 GB VRAM. The practical ceiling there is qwen2.5-14b at Q4. It is not as capable as the 26B model for complex multi-signal reasoning, but it is adequate for most monitoring tasks and sufficient for alert triage.

The Sovereignty Argument¶

When Bob's inference runs on hardware you own, you know exactly what the system is doing. You can inspect every inference request. You can log every prompt and every response. You can audit the model weights to verify they are what you downloaded. You can run in an air-gapped environment if required. You can guarantee to a regulator, auditor, or skeptical CISO that the diagnostic data for your SQL Server instances has not been transmitted to any third party.

You cannot do those things with a cloud API.

The counterargument is that cloud models are more capable. That is true at the frontier. It is less true for domain-specific tasks at production query volumes. For the specific problem Bob solves, the local models are good enough, the hardware is affordable, and the sovereignty is complete.

The next chapter is where the pieces come together: the three-layer agent that takes sensor data from SQL Server, routes it through Ollama for analysis, and acts on what it finds.