SLOs for Agents.

Eval scores are a model-quality metric. They are not a reliability metric. Most agent programs ship before they understand the difference — and the on-call engineer at three a.m. is the one who learns it for them.

The signals are not the same

The Google SRE team gave the industry the four golden signals — latency, traffic, errors, saturation — and a generation of services were instrumented against them. There is no equivalent for agents yet.

The reason is not that agents are too new. The reason is that the existing signals do not translate. A retry is not an error — it is often the correct response to a transient failure. A refusal is not a failure — it is often the correct response to ambiguity. A correct answer at the wrong moment, or against the wrong tool, is not a success in any sense the customer cares about.

Agents need their own signals, and most teams do not yet know what they are.

What the gap actually costs

Most enterprise agent programs measure model quality and then ship to production hoping reliability follows. It does not. MIT Project NANDA’s 2025 implementation research found that just 5% of integrated enterprise AI pilots were extracting millions in value while the vast majority showed no measurable P&L impact. Reliability is not the only reason pilots fail, but it is one of the places the gap becomes visible.

The agent works in demo. It does not work at three a.m., under load, on a slightly malformed request, against a downstream dependency that returned a partial response. The eval score does not catch that. Reliability is the gap between the eval and the customer, and it is where most agent programs lose their value.

The cost is paid by the on-call engineer at three a.m. and by the customer the next morning. It is also paid by the next program — once the first agent has produced an outage that the team cannot explain, the engineering organisation quietly reverts to manual workflows. The agent is technically still deployed. It is also no longer trusted. That state is worse than not having shipped at all.

Six SLOs that matter

These are the six SLO categories I think every team shipping a production agent should be writing against. Some of them have analogs in traditional service reliability. None of them are imported wholesale.

1. Resolution rate. Of all requests received, what percentage reach an acceptable terminal state? Not “no error returned” — actually solved, or correctly escalated to a human, or correctly refused. This is the agent equivalent of availability, but with a sharper definition: an agent that returns no error but punts every request to a fallback is not available, even if its uptime dashboard is green.

2. Time to resolution. From request received to terminal state, including all retries and escalations, measured at p50, p95, and p99. Single-call latency is the wrong number — end-to-end is what the customer experiences. An agent that retries five times in two seconds is not slow. An agent that returns “success” in two hundred milliseconds while having quietly handed off to a human is fast on the wrong metric.

3. Cost per resolution. Tokens, tool calls, downstream service costs, and compute, all aggregated per resolved request rather than per agent invocation. This is the metric the CFO will eventually demand, and it is the one that catches retries-as-cost-amplifiers before they show up in the bill. A 2% increase in retries can blow a budget by 20% if it is not measured at the resolution level.

4. Refusal correctness. When the agent declines to act, was it right to decline? An agent that refuses when genuinely uncertain is doing its job. An agent that refuses on tasks it should have completed is broken, even if its refusal looked polite. This SLO is tracked as a quality dimension on the refusal stream specifically, separate from the main quality metric, because refusals tend to hide failure modes that aggregated quality scores miss.

5. Quality regression. A held-out test set evaluated continuously against the deployed agent, with the SLO defined on the delta over time rather than the absolute score. The agent that scored 92% at launch and now scores 89% has regressed, and a 3% regression that goes undetected for two months is exactly how production agents silently degrade. The eval suite is not just a pre-deployment gate. It is a production signal.

6. Behavioural drift. Distribution shift in inputs, outputs, or model behaviour over time, independent of quality scores. A drift signal can fire while quality looks stable — and it is often the first indication that something around the agent has changed: new request types from a feature launch upstream, new tool failures from a dependency upgrade, new edge cases the eval suite has not yet learned to cover. Drift is a leading indicator. Treat it as a first-class signal, not a quarterly report.

A first SLO sheet

A production review should be able to read something this concrete:

Signal	Initial target	Alert when
Resolution rate	92% of eligible requests reach solved, escalated, or correctly refused states	Seven-day average drops below 88%
Time to resolution	p95 under 45 seconds, including retries and tool calls	p95 exceeds 60 seconds for two consecutive hours
Cost per resolution	Within 15% of the approved unit-cost model	Daily cost per resolved request exceeds the model by 25%
Refusal correctness	95% of refusals judged appropriate in sample review	Two review samples in a row fall below 90%
Quality regression	No more than 2% decline against the held-out suite	Regression exceeds 3% from launch baseline
Behavioural drift	No unreviewed input class exceeds 5% of traffic	New class exceeds 5% without evaluation coverage

The numbers will change by domain. The shape should not. Each SLO names a terminal state, a measurement window, and a response threshold.

What to do with this

These six SLOs do not require new tools. They require treating the agent as a service with its own definitions of success, error, and degradation, and then instrumenting accordingly.

Most of the signal comes from the orchestration layer — the layer that decides what runs when, retries what fails, and records what happened. If the orchestration layer was built for prototyping rather than production, the signals will not be there. The retry counter is missing. The terminal-state classification is missing. The cost rollup is missing. Build the orchestration for the SLOs you intend to write, not the other way around.

Reliability failures also show up downstream in the business case. They appear in the agent ROI floor calculation as the ship-probability discount: an unreliable agent does not earn what its eval scores projected.

When to talk to us

The first on-call incident is the wrong moment to discover your SLO definitions. Talk to us before launch, while there is still time to build the orchestration that exposes the right signals.

Reliability is what determines whether the agent is trusted. Trust is what determines whether it ships.

References

Google SRE, Monitoring Distributed Systems — The Four Golden Signals, in Site Reliability Engineering (2016). sre.google
MIT Project NANDA, The GenAI Divide: State of AI in Business 2025 (2025). mlq.ai PDF

Request a consultation