Why AI SRE Tools Are Getting It Wrong: A Mental Model for Production Reliability
AI SRE tools compete on signal quality. Better dashboards, smarter alerts, richer metrics. But the reconstruction of meaning from those signals still happens manually, every time, by the engineer who
AI SRE tools have been competing on signal quality. More dashboards, smarter alerts, richer traces, anomaly detection that catches the thing before a human does. The category has been optimizing signal delivery for a decade and the headline problem is still the same: at 2am, none of that is the thing you actually need. You need to know what is happening, why, what it touches, and how it was fixed last time.
So the engineer becomes a translator. Under time pressure they reconstruct meaning from raw signals: a CPU graph, an error rate, a PageRank value, a deviation multiplier. The meaning was never shipped. It had to be rebuilt, by a human, every single time, usually by the one person who happened to remember the last outage.
That reconstruction tax is the real cost of an incident. Every AI SRE tool that improves the signal still leaves the reconstruction to the engineer. Better numbers do not eliminate the manual work of turning numbers into meaning. That is the wrong problem to optimize.
The right problem: build an operational memory you can query, where meaning is already attached, where the reconstruction has already happened, and where every answer traces back to real evidence.
The evidence problem
The foundation is an evidence discipline, and it is what keeps the product honest.
Every element NOFire renders is an evidence claim with three parts:
the display: what we show you (a badge, a node, a number, a timeline row, a sentence)
the assertion: the operational fact it claims
the source: the real data in your live system
The one rule the product never breaks: a display must never assert more than its source supports. A pill that says “healthy” when nothing is measured is a false claim. So is “100% confidence” on an empty state, or “Invalid Date” where a timestamp should be. These are not lint bugs; they are claims with no data behind them.
What ties a display honestly to its source is provenance, and it has exactly three states that must read differently:
Observed: hard fact. A live metric, an active trace edge, a firing alert. The display is the data.
Inferred: best estimate. A dependency derived from an alert label, a criticality score from graph centrality, a root-cause hypothesis from deployment logs. The display points at a guess.
Unknown: no data yet. We say so explicitly, in a way that looks different from a confident value.
Two rules follow from this.
Evidence-first. Displays anchor to observed sources first. The language model compiles meaning from that evidence; it cannot invent infrastructure states or claim certainty it does not have. Every LLM-generated sentence must cite the data node it came from.
Compiled, not raw. Raw data without context is noise at 2am: K8S_SERVICE 'payment': cluster_ip: None -> 10.96.218.151 tells a tired reader nothing. The product’s job is evidence compilation: turn raw signals into actionable claims. “Payment service exposed via ClusterIP” is the same fact, compiled into meaning. The reconstruction tax is just evidence compilation done by hand; NOFire ships the compilation instead.
The core mental model
There is one set of entities in your system: services, deployments, the things that run and talk to each other. Everything NOFire does is a way of knowing something about those entities, or a way of looking at them together.
That is the whole mental model. Two questions, and only two, define every view:
Where am I? Zoomed out on the map (all entities and their relationships), or zoomed in on a subject (one entity).
What am I asking? Which lens am I looking through.
There are three lenses, and they are the three tenses of knowing an entity:
Topology is the present, structural: what is, and what is wired to what. It comes from the live dependency graph and is always current.
Knowledge is the past: what we have learned. How much this service matters, what depends on it, how it has failed before, what fixed it.
State is the live present and near future: what is happening right now and what is about to happen.
Topology tells you what the system is. Knowledge tells you what we have learned. State tells you what it is doing.
The grid
A screen in NOFire is never a new product. It is a position times a lens.
Map (the whole system):
Topology: the dependency graph
Knowledge: the graph seen through criticality
State: system-wide situations.
Subject (one entity):
Topology: the service and its callers/callees
Knowledge: the service wiki and its ontology
State: the service’s live status and signals.
Six cells. Not six tools. One graph and one entity page, each viewable through three lenses. You move by zooming in and out and flip lenses without
The governing rule that keeps this from sprawling: different data is not a different destination. Topology and knowledge are different data about the same entity. They belong as lenses on one surface, not separate places. The only thing that earns its own destination is a different job, never just a different data source.
The subject layer: an operational memory per service
For every service, NOFire synthesizes a living wiki. It is not hand-written and it does not rot in a doc; it is compiled from what the system already knows, and every claim carries explicit provenance.
Two axes, never conflated
A service is described on two orthogonal axes, because they answer different questions and confusing them is how dashboards lie:
Criticality is how much it matters: inferred from graph centrality, blast radius, and single-point-of-failure analysis.
Readiness is how well it is operated: from observable signals (owner, metrics, alerting, resilience).
They are a grid. The dangerous cell is tier-1 and at-risk: a service that matters and is poorly run. That cell is the whole point of having both axes.
The compiled change timeline
The timeline is the clearest example of the whole model in one component. Its timestamps and grouping are deterministic: same-service changes within a few minutes collapse into one logical change, headlined by the most significant one. Its descriptions are compiled into plain language. The result reads like memory, not a syslog:
June 2026
2026-06-09 20:38 UTC · rollout: Rolled out flagd 2.2.0-flagd-ui
Scaled to 1 replica · Created in cluster
No human had to notice the temporal connection between those events at 2am. The structure was already in the graph.
Recovery knowledge
The “when this breaks” section is the runbook the last outage wrote for you: known mitigations from past incident resolutions, each cited. Data with no provenance is just an assertion; data you can click back to a real incident is something you can trust at 2am and act on at design time.
From declared to observed
A dependency graph built from configuration tells you what can talk to what. It is a starting point, not the truth. Network observability closes the gap: real traffic tells you which calls actually happen, how often, how fast, and where they fail. That upgrades the topology lens from declared to observed, exactly the provenance distinction from before. An edge carrying live traffic is a runtime dependency; a declared edge with no traffic is a phantom worth flagging.
The same observed signals feed the state lens. An error count on a real edge is the raw material of a meaningful state, not just another line on a chart.
The state layer: meaning, not numbers
The deepest part of the model is the state lens, and it applies the same evidence-compilation approach to live data: emit the interpreted claim where the judgment can happen, instead of shipping a raw number for something downstream to reinterpret.
A severity score becomes severe anomaly rather than 7.3. An influence score becomes root cause candidate rather than a centrality value. A forecast becomes threshold breach imminent rather than a slope. Signs aggregate over time into a status per entity. Statuses combine with the dependency graph into a system-wide situation: localized, cascading, or projected to escalate.
The point is not prediction for its own sake. It is grounded, explainable meaning. Every sign traces back to the signal that produced it, every situation to the entities and the graph that formed it. A system that says “this will degrade in 28 minutes, here is the cascade path, and here is the incident it resembles” is useful precisely because you can see why it believes that.
The control layer: enforcement outside the context window
Knowing what is happening is not enough. When an agent acts on that knowledge, a layer must govern what it is actually allowed to do, and that layer cannot live inside the agent’s context.
Context can be manipulated. A log line, a ticket, a tool output: any input an agent reads is a surface something can write to. An agent that enforces its own limits is an agent that can be argued out of them. The control plane in NOFire is out-of-band: it sits in the network path, the identity layer, and the execution environment, not in the model’s reasoning. The agent cannot renegotiate a decision made at that layer because it cannot see it.
The result is one execution path for every actor (human, CI pipeline, or agent) with the same context, the same policy gates, and the same audit trail. Adding an agent to your system does not add a new, unguarded way to touch production. It adds another actor on the same governed path.
The same memory that grounds a root-cause investigation grounds the control decision. Before an action executes, the control plane reads the entity’s criticality, its blast radius, its past incidents, and its owner. A low-risk action on a tier-3 service with no recent changes is different from the same action on a tier-1 service in an active incident. The context that makes the diagnosis right is the same context that makes the enforcement smart.
AI reasons. Humans decide. The memory ensures both decisions are made with the same facts.
The memory is active
This is not a wiki you visit. The memory is wired into the work. When an investigation targets a service, its compiled wiki is loaded directly into the agent’s context, so root-cause analysis reasons with the service’s history, dependencies, and known mitigations rather than rediscovering them. Knowledge becomes the agent’s working memory, not a tab someone forgot to open.
From there the loop continues into action. The same readiness signals that draw the risks-and-gaps read-out can drive proposals: identify the gaps that matter, and, gated by human approval, open tickets in the team’s tracker. Knowledge becomes risk becomes action, with provenance the whole way, and nothing outward-facing happens without a person saying yes.
One memory, every stage of the lifecycle
Because all of this is one queryable memory, it is not only an incident tool. The same memory answers different questions in different tenses across the whole lifecycle:
Plan: query what actually caused last quarter’s incidents, so reliability work is backed by evidence.
Build: understand the real blast radius of a change before it merges.
Deploy: judge a release against how this service normally behaves and what usually breaks here.
Operate: ground root-cause analysis in the change timeline and known mitigations, so the answer arrives in minutes.
Onboard: give a new engineer ownership, history, and dependency maps on day one.
These are not five products. They are one memory asked five questions: past, conditional, comparative, causal, and present.
The flywheel
Every incident that resolves writes its mitigation and its learnings back into the memory as new, cited references. The enriched memory grounds the next investigation and makes the next answer faster and more confident. Better answers mean fewer surprises, which means the next change and the next deploy are safer too.
incident resolves -> mitigation + learning written into memory
^ |
grounds next RCA <- memory gets richer (new cited references)
^ |
2am gets faster <- design and deploy get safer
The asset that makes 2am better is the same asset that makes a Tuesday afternoon design review better. Every outage makes the system smarter instead of just tired.
The principles
A display never asserts more than its source. No confidence on nothing, no green without a signal, no value without provenance.
Provenance, always: observed, inferred, or unknown. The three states must look different. Unknown is a first-class answer, not an empty field.
Evidence-first. Every claim, score, and situation traces to real data. The LLM compiles; it does not invent. Explainability is the moat.
Compiled, not raw. Ship interpreted claims, not raw values someone has to decode. Deterministic where it can be; compiled into plain language where that adds meaning.
One subject, many lenses. One map, many lenses. A new surface is justified by a new position or a new job, never a new data source.
Control closes the loop. Every agent action is evaluated out-of-band against the same live model and memory. The governance layer and the context layer are one system.
A walk through one service
Open payment. The facts panel says Important, tier-2, readiness Partial, owner backend (inferred, from its alert’s team label). Risks and gaps, at the top: no SLO, no recovery docs. The change timeline, compiled: a June rollout that also scaled and updated config, one logical entry, exact UTC. Observability: the selector to scope it, and a foldable Errors block with a runnable error-rate query. When this breaks: the mitigation from its last incident, plus the linked learnings. Flip to topology and you see only the services it calls and is called by. Every line on that page is a sign you can click back to its evidence.
That is the difference between an alert count and understanding.
Where this lands
Operational clarity comes first: a system you can understand, one entity or the whole map, through topology, knowledge, and state, with evidence behind every word. Control closes the loop: every action any actor takes, whether human, pipeline, or agent, runs through the same governed path against the same live model. Proactive reliability is what clarity and control together make possible: a system that remembers, that sees its own trajectory, and that tells you in time.
Not a wall of numbers to reconstruct. One memory, in every tense, where the meaning is already attached and the boundary already holds.
Follow me on
Contact me!
If you want to start adopting a culture of reliability and AI, feel free to Contact me.
Better alerts help, but reliability still depends on whether someone can reconstruct what happened, see what changed, and know which action is safe under pressure. AI SRE tooling has to support that judgement, not just surface more signal.