Chapter 8: Toward Self-Healing¶
The phrase "self-healing infrastructure" gets used loosely. It appears in vendor decks alongside phrases like "zero-touch operations" and "autonomous remediation," and like most marketing language, it describes something real and hides what building it actually costs.
A self-healing system detects a specific class of problems, diagnoses them correctly, applies a known-good fix without human intervention, verifies the fix worked, and rolls back safely if it did not. Everything short of that complete loop is automation, which is valuable but different.
Bob, as of this writing, executes parts of that loop. Not all of it. This chapter is honest about where the line sits today and what it would take to move it.
What Bob Does Today¶
Bob's current operational loop is this: probes run on a 60-second cycle, collecting wait stats, query performance data, and memory pressure from the monitored SQL Server instance. Every cycle, the Layer 2 reasoning engine builds a diagnostic prompt, sends it to Ollama on inference-host-1 at 10.0.0.70, and parses the structured JSON response. If severity is NONE or LOW, the finding is logged and the cycle ends. If severity is MEDIUM, an alert fires and a human reviews the finding. If severity is HIGH or CRITICAL, the agent proceeds toward the actuator layer.
The actuator layer has a gatekeeper: the confidence check. Bob will not auto-apply any fix the model rates below HIGH confidence. And even within high-confidence findings, the current auto-apply scope is narrow on purpose: only index maintenance operations, ALTER INDEX ... REBUILD and UPDATE STATISTICS, run automatically. Everything else, query rewrites, configuration changes, service restarts, requires a human to read the finding and give explicit approval through the management interface.
The Proxmox snapshot always comes first. This is enforced in the remediation tool code: the tool will refuse to execute if no snapshot call appears in the current remediation log for this session. That check runs against the remediation log on db-host at 10.0.0.14:50003. No snapshot record means no apply. Period.
This architecture means that today Bob reliably handles one class of problem autonomously: index fragmentation. High-fragmentation indexes rebuild overnight without anyone waking up to do it. Everything else is assisted: Bob identifies it, explains it, and waits.
The Failure Domain Map¶
Understanding where Bob sits on the self-healing spectrum requires understanding what can go wrong and what the system does when it does. The failure domains from Architecture §2.5 are the frame:
LLM host down. If inference-host-1 at .70 is unreachable, Bob's _ollama_call() function automatically walks the llm_config.json failover chain. No human configuration reload required. On the RTX 3060, the model ceiling drops. gemma4:26b cannot fit in 12 GB of VRAM. The fallback model on .71 is qwen2.5-14b. Diagnostic quality degrades under heavy query plan loads, but monitoring continues. The heartbeat loop also has a circuit breaker: after three consecutive LLM failures, it stops attempting goal execution until the LLM recovers. This prevents cascading failures from a runaway retry loop. If both inference hosts are unreachable, Bob's heartbeat circuit breaker trips, monitoring probes continue, no autonomous actions are taken, and an alert fires via the notification system. The system fails safe, not silent.
MCP tool error. When a single tool call fails, the agent falls back to read-only mode for that capability. If the SQL diagnostic MCP call fails, the probe returns a partial snapshot. If the Proxmox snapshot call fails, the remediation gate blocks: no snapshot record means no apply. The agent escalates and waits. This is the right behavior. A monitoring system that applies fixes when it cannot snapshot first is more dangerous than a monitoring system that does nothing.
SQL fix misfires. The snapshot pre-apply requirement is the primary defense here. The secondary defense is the transaction boundary in execute_remediation: the fix runs inside BEGIN TRANSACTION / COMMIT TRANSACTION. If the SQL itself executes without a runtime error but produces a wrong result, the transaction commits, the snapshot remains available for manual rollback, and the post-apply verification step should catch the problem. Post-apply verification currently checks whether the targeted metric improved within five minutes. If the metric is the same or worse, Bob marks the remediation as VERIFY_FAILED, fires an alert, and logs the snapshot ID for manual rollback. The snapshot name format is bob-pre-fix-YYYYMMDD-HHMMSS, easily identifiable in the Proxmox interface at 10.0.0.2.
Network partition. If the agent host cannot reach the SQL Server or the Proxmox API, probes fail with connection errors. The agent treats a connection failure like a probe failure: partial data, elevated default severity, alert to the operations channel. Network-level detection beyond simple reachability is provided by Bob's network_scan tool (nmap-based) and the multi-host SSH monitoring that probes dns-primary at .222, dns-secondary at .221, and other configured hosts every monitoring cycle. A partition that isolates the agent from SQL Server but leaves the agent reachable is logged. A partition that takes the agent itself offline is not detectable from within the agent, which is expected: monitoring cannot monitor itself. The notification system at P0 severity handles the escalation path for events the agent cannot self-report.
flowchart TD
probe[Probe cycle starts]
probe --> infer{Inference host reachable?}
infer -- Yes --> reason[Layer 2 reasoning]
infer -- No --> retry[Retry x3 with backoff]
retry -- Primary still down --> fallback[Switch to inference-host-2 .71]
retry -- Primary recovered --> reason
fallback -- Fallback also down --> readonly[Read-only mode + alert]
fallback -- Fallback ok --> reason
reason --> severity{Severity?}
severity -- NONE/LOW --> log[Log and continue]
severity -- MEDIUM --> alert[Alert + human review]
severity -- HIGH/CRITICAL --> gate{Confidence HIGH?}
gate -- No --> alert
gate -- Yes --> snap{Snapshot successful?}
snap -- No --> alert
snap -- Yes --> apply[Apply fix in transaction]
apply --> verify{Metric improved?}
verify -- Yes --> done[Log success]
verify -- No --> escalate[Alert + log snapshot ID for manual rollback]What Already Self-Heals: The Forge Pipeline¶
The gap between where Bob is today and a fully self-healing system is not a gap in the agent logic. Several pieces of that loop are already running in production, and they are worth understanding before describing what still requires human judgment.
The most significant self-healing mechanism is one most DBAs would not think to look for: Bob rewrites his own source code. The Forge evolution pipeline runs in a separate process (run_evolution.py) on a 30-cycle batch schedule managed by a systemd timer. Each cycle follows seven steps (DIAGNOSE, TARGET, GENERATE, VALIDATE, EVALUATE, APPLY, REFLECT) and uses approximately 16–18 LLM calls to challenge Bob's own reasoning, generate candidate improvements, and apply the best one if it passes all safety gates.
The APPLY step has 8 sequential safety layers before any patch reaches disk: non-empty check, existence verification, uniqueness check, syntax compilation, essential component verification, SHA-256 hash verification of 16 sacred functions that can never be modified, semantic invariant checks, and a timestamped backup creation. After the patch is written, two runtime checks run: 7 behavioral smoke tests in a subprocess with a 15-second timeout, and a scoring gauntlet of 3 real LLM-graded questions with a 120-second timeout. The patch must score an average of 7.0 or above, with no more than a 1.5-point drop from the recent 5-cycle average. Any failure at any layer triggers automatic rollback from the timestamped backup.
In production, the Forge pipeline has completed 1,027+ cycles and applied 34 verified modifications, all targeting the reason() function that drives Bob's diagnostic reasoning. Current diagnostic scores run 9.8–10.0 out of 10 on domain-relevant sysadmin questions. Those scores are what justify the autonomous monitoring posture: the model has been continuously validated against real questions, not just run in good faith.
The three maturity levels below apply to the SQL-specific remediation path: the path from finding an actionable SQL Server problem to applying a fix autonomously.
Level 1: Alert-only. This is where Bob started and where it ran for approximately three months. Every finding is logged. High-severity findings fire P0/P1 alerts via the notification system. Nothing applies automatically. The DBA reads the finding, decides what to do, and acts manually. This level produces the training data for Level 2: you accumulate a history of findings and their resolutions, which tells you how accurate the model's recommendations are before you give it any authority to act on them.
Level 2: Suggest-with-approval. The reasoning layer generates a specific fix candidate and presents it to the DBA through the Kanban board in the web dashboard. The DBA reviews the finding, the confidence score, the proposed SQL, and the snapshot status, then approves or denies the Kanban card. This level is where you build trust in the model's judgment for specific fix categories. Index maintenance recommendations were in this level for about two months before moving to Level 3. Every approval or denial is logged to outcomes.json, so you can look back and see that out of forty-seven index maintenance recommendations, forty-three were approved and applied successfully, three were rejected by the DBA as low-priority, and one was rejected because the model recommended rebuilding an index on a table that was actively receiving a bulk load.
Level 3: Apply-with-rollback. The current production level for index maintenance. The confidence gate, the snapshot pre-condition, and the post-apply verification step are what make this safe rather than reckless. The transition from Level 2 to Level 3 for a given fix category requires two things: a track record from Level 2 that shows a false-positive rate below roughly 5%, and an operational window constraint that restricts auto-apply to hours when the risk of blocking is lowest.
The gating mechanism between levels is not a technical threshold. It is a decision by the DBA running the system, based on the evidence accumulated at the previous level. There is no algorithm that tells you when to promote a fix category from suggest to apply. You decide when you trust the track record you have built. That subjectivity is not a design flaw. The track record is the argument, not the algorithm.
This is how the roadmap is sequenced for the SQL remediation path:
Phase 1: Expand the auto-apply scope carefully. Index maintenance is conservative and low-risk. The next category is statistics updates, which are already on the auto-apply list. After that: automatically killing long-running blocking sessions that have been blocking for more than a configurable threshold (default: 15 minutes) and have no open transaction of their own. That action is reversible in practice (the session will reconnect if the application is designed correctly) but not reversible in the transactional sense. It requires confidence from both the model and an operational window check: do not auto-kill sessions during business hours unless severity is CRITICAL.
Phase 2: Cross-instance correlation. Today Bob monitors one SQL Server instance. The architecture supports multiple via mssql_instances.json, but the reasoning layer currently treats each instance independently. Cross-instance correlation means: if three instances all show the same wait type spiking at the same time, that is probably not a per-instance SQL problem, it is probably an infrastructure problem, a storage controller, a network segment, a shared service. Detecting and naming that correlation requires a reasoning step that runs across instances rather than within one.
Phase 3: SQL-specific evolution goals. Bob already has an autonomous evolution goal system (goals g013–g020 in goals.json) that has successfully built 8 new capabilities by writing Python code to disk and verifying they work. The next frontier is SQL Server-specific growth goals: an index fragmentation analyzer, a TempDB health monitor, a Query Store reader. These would follow the same pattern as the existing evolution goals: defined in goals.json, evaluated by the heartbeat using Qwen3-Coder, verified by file existence and import check. The SQL feature gap analysis in docs/SQL_MONITOR_FEATURE_GAPS.md already has 20 goal definitions ready to feed into this pipeline.
The Question of Trust¶
There is a principle in reliability engineering called the automation trust gradient. The idea is that you should extend autonomy to a system incrementally, proportional to the track record. You do not hand a new system the keys on day one. You give it narrow authority, watch what it does with that authority, and expand the scope as confidence accumulates.
Bob started with zero auto-apply authority. Every finding was routed to a human. That phase lasted about three months. In three months of watching, the high-confidence index maintenance recommendations were correct roughly ninety percent of the time. The ten percent failures were mostly cases where the fragmentation was real but the index was on a table being actively loaded, so the rebuild caused brief blocking. Not disasters, but not clean either.
The solution was an operational window check: auto-apply index maintenance only between midnight and 5 AM. That check dropped the failure rate to near zero over the next two months. With that track record established, index maintenance moved to full autonomy.
That is the model for every subsequent expansion: narrow scope, operational window check, watch the failure rate, adjust. Automate what the track record supports, nothing more.
There is a category of fix that will never enter Level 3, not because the model cannot reason about it correctly, but because the judgment call it requires is inherently contextual in ways the model cannot fully see.
Schema changes are the clearest example. If Bob detects that a missing index would eliminate a 90% table scan on a critical query, the recommendation is correct. The risk is in the execution: adding an index on a large table during business hours blocks writes for the duration of the build. The model can know that index builds block writes. It cannot know whether a blocking event right now will affect a payment processing window, a report that a VP is waiting for, or a data load from an external partner that has contractual SLA implications. That context lives in the organization, not in the DMVs. A DBA who picks up the phone and asks before scheduling the index build has access to that context. An automated agent does not.
The same argument applies to query rewrites. A plan regression where the optimizer chose a hash join over a nested loop join because statistics are stale is a diagnosable, fixable problem. Bob can identify it. An UPDATE STATISTICS call might fix it. But if the query is in a stored procedure that belongs to a third-party application whose vendor has a support clause that prohibits schema modifications without their sign-off, the correct action is to email the vendor, not to rewrite the query. Bob cannot know that the stored procedure has a vendor lock. The DBA can.
The self-healing system that is worth building is one that handles the routine clearly and escalates the complex cleanly. The boundary between those two categories shifts over time as the model's track record grows and as the system's knowledge of the environment deepens. But it never disappears entirely. There will always be a class of judgment that belongs to a human with organizational context that no monitoring system can fully encode.
A DBA who has been in the field long enough has internalized this gradient through painful experience. Automation that outstrips its track record produces incidents. Incidents produce rollbacks, not just of code but of organizational trust in automation generally. Once an organization has been burned by a system that did too much too fast, getting approval for the next automation initiative is a year-long political project.
Bob is designed not to earn that kind of reputation. The aggressive gating, the snapshot requirement, the confidence floor, the operational windows: none of these are bureaucratic overhead. They are how you build a track record.
The next chapter is where the personal becomes the communal. The architecture documented in Part 2 is not meant to stay in one server room. The question Chapter 9 addresses is how to share it in a way that actually works: not as a GitHub repo that nobody runs, but as a living community practice.
What Bob does for one DBA's infrastructure scales. The bottleneck is the trust gradient, not the technology, multiplied across every organization that would need to build its own track record from scratch. Chapter 9 is about shortcutting that process without cutting corners.