Chapter 4: The Infrastructure¶
Bob does not run in the cloud, by design. The off-cloud placement is the architecture, not a limitation to work around.
This chapter documents the physical infrastructure Bob runs on: the hardware, the hypervisor, the network, the storage. Every IP address in this chapter is real. Every hostname is production. If something in this chapter contradicts what you see when you SSH into the relevant host, the chapter is wrong and should be corrected.
The Network¶
The lab runs on a single /24 subnet: 10.0.0.0/24. The router is OPNsense at .1, running FreeBSD with Unbound for DNS and a full stateful firewall. DHCP is centralized on OPNsense. DNS resolution is handled by two BIND 9.20.11 forward-only resolvers: dns-primary at .222 and dns-secondary at .221, both running as Proxmox VMs.
The primary hypervisor is Proxmox VE 8.4.1 at 10.0.0.2. It runs on an i9-13900K with 94 GB of RAM and is the host for most of the services Bob depends on. The VM inventory relevant to Bob is listed below.
NAS storage lives on nas at 10.0.0.20: Ubuntu 25.10 with Samba acting as an Active Directory domain controller, four NICs bonded into bond0 for throughput, and 37 TB of RAID for bulk storage.
The SQL Server target Bob monitors runs on db-host at 10.0.0.14. This is a Windows Server 2012 host with SQL Server listening on the non-standard port 50003. The BookOfBob metadata database lives here alongside the production databases Bob monitors.
OPNsense at .1 handles more than routing. It runs Unbound DNS, which means internal resolution is fast and fully under local control. No DNS query for an internal hostname leaves the LAN. DHCP leases are managed centrally here too, which matters for monitoring: static assignments for every host the agent touches, since an IP address change at 3 AM because a DHCP lease expired is exactly the kind of silent failure that produces a 4 AM page.
The two BIND resolvers at .222 and .221 are forward-only. They receive DNS queries from LAN hosts, forward to Unbound on OPNsense or upstream resolvers, and cache results. Redundant DNS is infrastructure hygiene. If the agent host cannot resolve db-host.local because a single resolver VM is restarting, Bob goes blind. Two resolvers cost four vCPUs and 8 GB of RAM on Proxmox. That is a cheap insurance premium.
The AI Hosts¶
Bob's inference layer runs on two dedicated AI servers.
inference-host-1 at 10.0.0.70 is the primary inference host. Hardware: Ryzen 9 9950X processor, RTX 3090 GPU with 24 GB VRAM, 182 GB system RAM. This machine runs Ubuntu 25.10 with Ollama managed as a systemd service, Open WebUI for browser-based model interaction, and two text-to-speech containers. The RTX 3090 is critical here: 24 GB of VRAM is enough to run gemma4:26b, Bob's primary model, without falling back to CPU inference. CPU fallback on a model this size is usable for experimentation and painful for production monitoring.
inference-host-2 at 10.0.0.71 is the secondary inference host. Hardware: i7-11700F, RTX 3060 with 12 GB VRAM, 125 GB system RAM, Ubuntu 24.04.4. Ollama runs as a systemd service here with fifteen models available, including qwen2.5-14b which serves as Bob's automatic failover model. The RTX 3060 handles models up to about 13B parameters comfortably at Q4 quantization. Larger models require careful quantization choices to fit in 12 GB.
Both hosts expose Ollama at the standard port 11434. Bob's _ollama_call() function implements automatic failover: if the primary host at .70 is unreachable, it transparently tries the next entry in the llm_config.json failover chain, which points to qwen2.5-14b on .71. Each server has its own default model since different servers host different model sets. No human intervention is needed for a failover event.
The Ryzen 9 9950X in inference-host-1 is worth a note because the CPU choice matters more than it seems. Ollama offloads model layers to GPU VRAM as much as possible, but when a model's total parameter footprint exceeds available VRAM, the overflow runs on CPU. For gemma4:26b, Bob's primary model, the model fits in 24 GB VRAM with room to spare, so CPU inference is not a normal-path concern. Where the CPU matters is in tokenization, in the embedding generation for the context window, and in parallel requests: when two monitoring cycles happen to coincide, the second one does not wait for the first to complete GPU inference if the CPU can handle the non-GPU work in parallel. The 9950X's 32 threads handle that without contention. An older eight-core machine would not.
inference-host-2 at .71 tells a different story. The i7-11700F is a capable processor but it predates the efficiency cores that make the 9950X fast at mixed workloads. The RTX 3060 with 12 GB VRAM is the binding constraint: gemma4:26b does not fit. The failover model qwen2.5-14b at Q4 quantization uses about 9 GB of VRAM, leaving 3 GB for context and system overhead. That 3 GB margin is tight. Running concurrent requests on the 3060 with the 14B model loaded will cause model layers to spill to system RAM, and the performance penalty is significant enough that in practice, the fallback host should not be expected to match inference-host-1's throughput or response latency.
That asymmetry is fine for a failover scenario. Bob running on inference-host-2 is slower and less capable than Bob running on inference-host-1, but it is running. The alternative to a slower fallback is no monitoring when the primary is down.
The Hypervisor and Supporting VMs¶
Proxmox at 10.0.0.2 hosts the VMs that provide Bob's supporting infrastructure. The table below shows the VMs that matter to this deployment.
| VM ID | Name | vCPUs | RAM | IP | Role |
|---|---|---|---|---|---|
| 101 | dns-primary | 4 | 4 GB | .222 | BIND 9 primary DNS |
| 102 | dns-secondary | 4 | 4 GB | .221 | BIND 9 secondary DNS |
| 103 | gitea | 4 | 16 GB | .80 | Source control, CI |
| 104 | webserver | 8 | 32 GB | .50 | Apache, web serving |
| 106 | n8n | 4 | 8 GB | .60 | Workflow automation |
The Gitea VM at 10.0.0.80 is where Bob's source lives: http://10.0.0.80:3000/hyp3rsoft/bob. Gitea's SSH port is 2222, not the standard 22. The CI runner (gitea-runner) lives on the same VM and handles the build pipeline.
Bob agent VMs are provisioned on Proxmox as needed. The agent itself is lightweight enough to share a VM with other services in a lab context. For production deployments, dedicated VM allocation is recommended.
The n8n workflow automation VM at .60 is not part of Bob's primary architecture, but it handles the glue work: webhook routing, notification formatting, and the scheduled tasks that do not belong in the agent loop. When Bob fires an alert, the destination is an n8n webhook endpoint. n8n decides whether that alert becomes an Asterisk call, a formatted message to a chat channel, or a ticket in the issue tracker. Keeping that routing logic out of the agent code means it can be changed without deploying a new version of Bob.
The webserver VM at .50 hosts the Bob documentation site, built from the manuscript's Markdown via the Gitea CI pipeline and served under Apache. That is not operational infrastructure for Bob, but it is part of the project. Cloudflare terminates SSL at the edge and proxies to the Apache vhost, which means the documentation is publicly accessible without exposing the LAN.
The Proxmox Decision¶
The hypervisor choice is Proxmox VE 8.4.1 running on the i9-13900K machine at 10.0.0.2. That decision deserves an explanation because it is not the obvious choice and the alternatives are real.
The first alternative is bare-metal Docker: run everything as containers directly on the host operating system, no hypervisor. This is simpler initially. It is also the choice that produces the most pain at scale. Container-based isolation is not VM-based isolation. A runaway container process can consume resources that affect other containers on the same host. A hypervisor with defined vCPU and memory limits on each VM does not have that problem. For a home lab running one or two services, Docker on bare metal is fine. For a lab running eight VMs with different security boundaries and resource requirements, the hypervisor overhead is worth it.
The second alternative is ESXi. VMware's hypervisor is the enterprise standard for a reason: it is stable, well-documented, and most corporate IT departments already know it. The reason it is not the choice here is licensing. VMware's acquisition by Broadcom in 2024 changed the licensing model in ways that eliminated the free tier and made single-node home lab deployments economically unreasonable. Proxmox is open source, actively developed, and free for single nodes without a support contract.
The third alternative is k3s or another lightweight Kubernetes distribution. This is the forward-looking choice for anyone building microservices at scale, and if the project's trajectory was toward hundreds of small services, it would be the right call. Bob is not hundreds of small services. It is a handful of well-defined components that do not benefit from the orchestration overhead of Kubernetes. Using k3s here would be choosing the tool that will impress people reading your architecture diagram over the tool that is right for the problem.
Proxmox on a 94 GB, 32-thread machine gives the lab a vCPU overcommit ratio of 1.06x across 34 allocated vCPUs, barely overcommit. The VMs mostly idle, which means the ratio on active threads at any given moment is much lower. RAM is the tighter constraint: 77 GB allocated against 94 GB available. That leaves 17 GB for the hypervisor overhead and any burst. It is tight but manageable because the AI inference hosts are dedicated hardware, not VMs, so the RAM-hungry inference workload never touches Proxmox's RAM budget.
The Network Topology¶
graph TD
inet[Internet] --> cf[Cloudflare Tunnel]
cf --> ws[webserver .50]
opn[OPNsense .1] --> prox[Proxmox .2]
opn --> nas[nas .20]
opn --> ai1[inference-host-1 .70]
opn --> ai2[inference-host-2 .71]
opn --> win[db-host .14]
prox --> gitea[Gitea VM .80]
prox --> web[webserver VM .50]
ai1 -- "Ollama :11434" --> bob[Bob Agent]
win -- "SQL :50003" --> bob
gitea -- "source control" --> bob
bob --> win
bob --> ai1Sizing for Bob¶
The infrastructure above is a production setup at the high end of what you need for a single-instance SQL Server monitoring deployment. The table below gives honest numbers for three deployment sizes.
| Tier | CPU | GPU / VRAM | RAM | Storage | Use Case |
|---|---|---|---|---|---|
| Minimum | 8-core (any modern x86) | RTX 3060 / 12 GB | 32 GB | 500 GB SSD | Single SQL Server instance, models ≤ 13B, response time 5–30 s |
| Recommended | 16-core (Ryzen 7 / i9) | RTX 3080/3090 / 16–24 GB | 64 GB | 1 TB NVMe | 2–4 SQL Server instances, models up to 32B Q4, response time 2–8 s |
| Production | 16-core+ (Ryzen 9 9950X) | RTX 3090 / 24 GB | 128 GB+ | 2 TB NVMe RAID | 5+ instances, concurrent monitoring, redundant inference host |
A few notes on the numbers.
The VRAM figure is the constraint that matters most. System RAM can be large or small; Ollama uses system RAM as overflow when a model layer spills off the GPU, but at inference time you want as much as possible on the GPU. A model layer on CPU is roughly 10x slower than on GPU. For a monitoring agent that might need to respond within a few seconds of an alert firing, 10x slower means the difference between "useful" and "too late to matter."
The minimum tier is genuinely minimum. A single RTX 3060 with 12 GB VRAM will run qwen2.5-14b or a similarly-sized model at Q4 quantization with acceptable latency for batch monitoring tasks. It will struggle if you try to run a 26B+ model or run concurrent requests. For a self-hoster who wants to try this on a weekend, the minimum tier works. For production, reach for the recommended tier.
Storage matters less than you might expect for inference, because model weights are read once at load time and then the model lives in GPU/system RAM. The storage budget is mostly for the model files themselves, which range from about 4 GB for a 7B model at Q4 to about 20 GB for a 32B model at Q4. Keep 200 GB free for a comfortable model library with room to grow.
The "production" tier listed above maps to the actual inference-host-1 hardware. It is not aspirational. It is what I am running.
The minimum tier deserves more attention because it is where most people starting out will land. An RTX 3060 with 12 GB VRAM and qwen2.5-14b or a comparable 14B model will work. The responses are slower and the diagnostic quality on complex query plans is lower than with the 26B model running on the production tier. The failure mode you will hit first at the minimum tier is not a crash or an error. It is the model returning a finding that misses the root cause of a complex problem. Wait stat correlation with query plan analysis is where smaller models struggle most. If your SQL Server workload is straightforward, mostly OLTP, well-indexed, the 14B model is adequate. If you are running complex analytical workloads with large parallel queries and frequent plan regressions, the minimum tier will produce diagnostic output that sounds plausible but requires more human verification.
The recommended tier at 64 GB RAM matters for a reason separate from inference. The Proxmox snapshots that the remediation pattern depends on require disk space proportional to VM size. If your SQL Server VM has 16 GB RAM and a 200 GB disk, a pre-fix snapshot will consume roughly the changed-page delta, which is small for most index maintenance operations. But if you are running multiple agents against multiple VMs and each monitoring cycle produces a snapshot, disk consumption grows. 1 TB NVMe with fast write throughput keeps the snapshot operation fast enough that it does not become a bottleneck in the remediation flow.
The lab did not start at the production tier. At month zero, Bob ran on the development workstation at 10.0.0.100, a desktop-class machine with an RTX GPU and enough RAM to run the 14B model. That was where the 200-line prototype lived. By month six, the prototype had grown into something that needed its own host, and inference-host-2 at .71 came online as the primary inference host. The RTX 3060 was enough to validate the architecture. By month twelve, the diagnostic quality limitations of the 14B model were clear enough that the case for dedicated hardware was easy to make. inference-host-1 at .70, Ryzen 9 9950X, RTX 3090, 182 GB RAM, came online around month fourteen and has been the primary host since. The two months between twelve and fourteen were the procurement and setup time.
That arc, workstation to dedicated mid-range hardware to dedicated high-end hardware, mirrors the arc of what Bob could do. At month zero, it could diagnose. At month six, it could recommend. At month twelve, it could act in narrow, low-risk cases. By month eighteen, the auto-apply scope had expanded to cover the full index maintenance category, and the architecture was stable enough to document.
Network Security Notes¶
The setup described here is a LAN deployment with no exposure of inference endpoints to the internet. Ollama at 10.0.0.70:11434 is not firewalled from the internal LAN (all LAN hosts can reach it), which is an open finding from the NETWORK_SCAN.md audit. For an organization deploying Bob in a regulated environment, the recommendation is to firewall 11434 to permit only the specific hosts that need it: the agent host, the management workstation, and the Gitea CI runner.
The SQL Server host at .14 listens on 50003. The non-standard port is a defense-in-depth measure, not a security boundary. Proper SQL Server security requires strong service account credentials, auditing enabled, and network access controlled at the firewall, not just the port number.
Authentication between Bob and the SQL Server uses a dedicated service account with the minimum permissions required: VIEW SERVER STATE for DMV access, VIEW DATABASE STATE on monitored databases, and write access to BookOfBob for logging. No sysadmin membership. This is important for the same reason it always is: the blast radius of a credential compromise should be as small as possible.
The MCP server that connects Bob's reasoning layer to its actuators uses authenticated tool calls. The anonymous bind finding from NETWORK_SCAN.md applies to the LDAP service on nas, not to the Bob toolchain, but it is worth noting: if you are building on this infrastructure, audit every service that accepts anonymous authentication and close those gaps before putting an agent on the network that can take action based on what it finds.
The Data Residency Guarantee¶
Every byte of diagnostic data that passes through Bob stays inside 10.0.0.0/24.
Query plans, wait stats, deadlock graphs, T-SQL text: none of it leaves the LAN. The Ollama instance receives this data as part of the inference prompt and processes it entirely in memory on inference-host-1. There is no telemetry to external services. There is no model-training feedback loop that uploads your data somewhere. The models themselves are static weights that were downloaded once and run locally.
This is not difficult to verify. Run tcpdump -i eth0 not src 10.0.0.0/24 and not dst 10.0.0.0/24 on the agent host during a monitoring cycle. You should see nothing. If you see outbound connections on the inference ports, something is misconfigured.
The data residency guarantee is the thing that makes Bob viable for regulated environments. It is also the thing that makes it interesting as an architecture. The system's intelligence is local. It requires good local hardware, which is cheap compared to the alternative, and good local models, which are free.
Chapter 5 documents the Ollama side of this in detail: the specific models, the configuration, the systemd setup, and the hot-swap pattern for switching models without restarting the agent.