Chapter 1: The Report Problem¶
The report was 214 pages long.
I had not set out to build something that large. It grew the way most technical debt grows: incrementally, each addition justified at the time.
I had the PDF open on one monitor and SQL Server Management Studio on the other. It was a Tuesday in March, some year where the word "digital transformation" was still being used unironically in executive slide decks. The report was the weekly SQL Server health check for a mid-size financial services client. I had written the script that generated it. I had scheduled the job. I had configured the SMTP relay so it landed in fourteen inboxes every Monday morning.
Nobody read it.
I knew this because I had buried a joke on page 87, something about a cat photo and a deadlock, and not a single person said anything. Page 87 was probably where most readers gave up, assuming they made it past the table of contents at all.
That is the report problem. Not the content. The content was fine. The content was accurate, well-structured, timestamped, cross-referenced. The problem was that the format had grown to match the anxiety of the person generating it. Every time I missed something in a previous report and an incident happened, I added a new section. Missed an index fragmentation spike? Add a fragmentation section. Missed a growing tempdb? Add a tempdb section. Missed a suspicious query regression? Add a top-query-by-CPU section. Do that for two years and you have a 214-page document that nobody opens until something breaks.
At which point the report is useless anyway, because you are in incident mode and you do not have time to read 214 pages.
The DBA's job is variance detection. Is this server behaving differently today than it was last Tuesday? If yes, why? If the why is benign, note it and move on. If the why is a problem, fix it or escalate it before anyone downstream notices.
The report was supposed to serve variance detection. It failed at this for two reasons.
The first reason is volume. 214 pages of data is not information. It is noise with structure. Humans are not good at reading 214 pages of time-series charts and synthesizing the signal. We habituate. The eye moves over the page looking for something that looks wrong, and when nothing jumps out immediately, we mark the email read and go to our next meeting. The report was doing the cognitive work of data collection but not the cognitive work of interpretation. That interpretation work was still landing on the DBA.
The second reason is timing. The report ran at 6 AM Monday. By the time I read it Tuesday morning, the data was thirty hours old. If something was quietly getting worse across the weekend, I was not going to catch it at 6 AM on a Monday. I was going to catch it at 2 PM on a Tuesday when an application team called saying their batch job had been timing out since Sunday.
This is a solvable problem. It just requires thinking about it as a software problem rather than a documentation problem.
SQL Server exposes a remarkable amount of diagnostic information in real time. Dynamic Management Views, or DMVs, let you query the engine's internal state the same way you query a table. You can ask the server what its longest-running queries are. You can ask what its wait stats look like. You can ask what queries have changed execution plans since last week. You can ask whether any of your indexes have not been used in thirty days and are therefore dead weight.
None of this requires a scheduled batch job. You can ask these questions continuously.
The old approach was: run a batch job, collect data, format a report, email it to fourteen people, hope someone reads it. The new approach is: observe continuously, detect variance, and act on the variance immediately. The difference between the two is roughly the difference between weather forecasting via newspaper (read yesterday's weather report this morning) and weather forecasting via radar (see what's happening right now).
The problem with continuous monitoring is not technical. The problem is the interpretation layer. You can instrument everything. SQL Server will happily export metrics to a table, to a flat file, to a monitoring agent, to a Prometheus scrape endpoint if you set one up. But interpreting that stream in real time, correlating wait stats with query plans across application activity windows, still requires someone who understands what the numbers mean.
Historically that someone was a DBA. A human, reachable by phone, whose job it was to know which signals matter and which are noise.
That still works at a certain scale. It stops working when you have more SQL Server instances than DBAs, which is most organizations over a certain size. It stops working at 2 AM. It stops working when your experienced DBA leaves and takes fifteen years of pattern-matching knowledge with them.
I want to be precise about what I was frustrated with, because the frustration is the seed of Bob.
I was not frustrated with SQL Server. SQL Server is a well-built piece of software with genuinely good diagnostic tooling. I was not frustrated with monitoring in general. I was not even frustrated with the report per se.
I was frustrated with the gap between the diagnostic information that already existed in the system and the humans-in-the-loop required to interpret it. The information was there. The system knew it had a problem. The problem was sitting in a DMV somewhere, expressible in four lines of T-SQL. What was missing was something that could read that DMV, understand what it meant, and say something useful about it without me having to be awake.
That is a reasoning problem. Data collection was solved. Alerting thresholds were solved, badly, but solved. The missing piece was the ability to look at a cluster of signals, a query regression here, a wait stat anomaly there, an index usage change somewhere else, and synthesize a diagnostic that would make sense to a senior DBA reading it cold.
In 2022 you could not do that automatically. You could write rule engines, expert systems, heuristics. But those systems were brittle. They knew what you taught them. They could not generalize from a pattern they had not seen before. Every new failure mode required a new rule.
In 2024 you could do it differently.
Large language models turned out to be surprisingly good at understanding SQL Server diagnostic data. Not because they were trained on SQL Server specifically, though they were exposed to plenty of SQL and plenty of SQL Server documentation. The reason was more general than that. A well-trained model could read a query plan XML, understand what the shapes of slow plans look like, and generate a plausible diagnostic hypothesis. It could read a wait stats snapshot, understand what CXPACKET waits mean at high volume, and suggest whether you were looking at a parallelism problem or a legitimate parallel scan on a large table.
It was not magic. It was wrong sometimes. It hallucinated occasionally. The first few months of working with LLMs on SQL diagnostic problems were an education in knowing when to trust the output and when to verify it independently.
But it was dramatically better than a rule engine. It generalized. It could reason about a combination of signals it had never seen in exactly that configuration before. It could explain its reasoning in plain English that a non-DBA developer or a manager could read and act on without a translation layer.
That was the insight. AI would not replace DBAs at the level of actual system design and incident response, but the interpretation gap, the distance between raw diagnostic data and actionable human language, was exactly the kind of gap a language model could bridge.
The question was how to build that bridge without handing your SQL Server's internal state to an external API.
That question is why Bob exists. And it is why Bob runs locally on a machine I own, on a network I control, using models that have never heard of your query plans and never will tell anyone about them.
There is a version of this story where I say: I saw AI coming and I was ahead of the curve. That version is false.
The honest version is: I had a problem that annoyed me for years, I kept a mental note of it, and then a technology arrived that looked like it might fit the hole. I did not move fast. I spent about six months reading about transformer architectures and local LLM options before I wrote a single line of code in that direction. I ran Ollama locally on a laptop first, queried it by hand with SQL fragments I copied and pasted, and watched whether the responses were useful.
They were useful enough. Not production-ready. Not reliable enough to act on without review. But useful enough to know the direction was right.
The cloud question came up immediately. OpenAI's API was obviously capable. Claude was available. I was using both at work, at Dogsbarking.com, for semantic mapping and entity resolution in Salesforce deduplication pipelines. GPT-4 and Claude 3 were handling complex reasoning tasks across large data sets, and doing it reliably enough that I was leaning on them for production work.
So why not use the cloud for Bob?
The answer was not ideological. It was the client list. The SQL Server environments I was thinking about when I designed Bob were not toy databases. They were production systems in industries where data governance is not optional: financial services, healthcare, utilities. At Lumeris in 2017 I was managing PHI under HIPAA. At Entergy through 2024 I was maintaining NERC CIP compliance. If I sent a query plan from one of those environments to an external API, I would need to explain to a compliance officer exactly which data left the network, under which data processing agreement, with which retention policy.
That conversation is possible but slow, and every organization that wants Bob to help them needs to go through a vendor review process for each AI provider in the chain. Cloud LLMs are not self-certifying for HIPAA or NERC CIP. You need a business associate agreement, a data processing addendum, or a vendor risk assessment before you can say that the AI assistant in your monitoring stack is compliant.
Local inference skips all of that. The data stays inside the network. There is no vendor. There is no data processing agreement to negotiate. There is nothing to include in your audit documentation beyond "we process this data on our own hardware using open-weight models." Your CISO can sign off on that sentence without a six-month review cycle.
That was the practical argument for local inference. It did not require me to have strong feelings about cloud AI in general. I use cloud AI. I use it at work when the compliance constraints permit it. The point is that "when compliance constraints permit it" is a significant qualifier for the environments where monitoring AI is most valuable.
Building Bob took eighteen months to reach its current form. Not eighteen months of continuous full-time work, but eighteen months from the first working prototype to the production system documented in Part 2. That time includes multiple architectural restarts, a period of six months where the project sat idle because other work took priority, and the slow accumulation of operational track record that is the only way to know whether the reasoning layer is actually trustworthy.
That duration matters because it sets an honest expectation. The architecture in Part 2 is not the first design. It is the design that survived contact with real operational conditions. The first design did not have a rollback mechanism. The second design had a rollback mechanism but no confidence gate on the actuator. The third design, which is roughly what is documented here, has both. Each iteration was motivated by something going wrong in a way that the previous design could not handle gracefully.
Chapter 2 covers what that development process looked like from the inside: the shift from running queries manually to building a system that could do it without me, and what the term "vibe coding" actually means when you are writing software that touches production data.
What I want to leave you with from this chapter is the problem statement, clearly. Because if you do not feel the problem, the solution is not going to make sense.
The report was 214 pages long. Nobody read it. A server would get sick, and then sicker, and then something would break, and then I would open the report and find the signals had been there for days. Sitting in a PDF. Waiting.
That is the report problem. That is what Bob is for.