PagerDuty’s SRE Agent Benchmark Shows Why Incident Context Matters

Incident response happens with partial evidence: one alert fires, a deployment just changed, and a confident message in the call may be wrong. PagerDuty launched Paige, its SRE agent, alongside a benchmark report that describes how it compares incident diagnoses. The report’s constraints are as important as its ranking because they show how far a simulated incident is from an on-call rotation.

What the benchmark measures

PagerDuty says its benchmark injects realistic faults into Kubernetes microservices and asks agents to work through them. It scores five dimensions: finding the faulty component, diagnosing the cause, identifying a mitigation, grounding conclusions in evidence, and explaining the reasoning. A panel of judges built on different models evaluates the answers. ^[2]

The report’s latest complete comparison covers 17 scenarios. PagerDuty reports its SRE Agent first, with Claude Code (Opus 5.5) second and Codex (Sol 5.6) third. It says the results are means per scenario, lower is better for latency and misdirection, and the reported sweep was a single trial. It also reports roughly two percentage points of sweep-to-sweep variability in earlier runs. These are company-reported results, not an independent assessment of real production incidents. ^[2]

PagerDuty says its product and test system evolved together. Early benchmarking found that a complex multi-agent investigation path performed worse on a subset of scenarios and was slower; the company changed the topology and data structure. It acknowledges that this creates a risk of benchmark influence, while saying it focused on general approaches instead of specific faults. That disclosure is useful: a benchmark should reveal the system’s development loop, not only its final score. ^[2]

The missing context changes the result

The product is designed to incorporate signals as an incident unfolds, including call transcripts, Slack or Teams messages and additional alerts. PagerDuty excluded those inputs from the comparative benchmark so each agent could be tested under similar conditions. It separately replayed its own past incidents and estimated that processed transcript data was associated with up to a five-point increase in diagnosis scores. That separate estimate is not the same controlled comparison as the 17-scenario sweep. ^[2]

This creates a practical evaluation question: what context does an organization’s agent actually receive, and can it distinguish a verified system signal from an untested human hypothesis? More context can help connect a deployment to a new error pattern. It can also amplify a mistaken statement or a stale runbook.

PagerDuty’s report names other limitations: the test load does not match production volume or historical complexity, one trial per agent is not a statistical evaluation, failed runs are excluded from quality scores, and remediation is a smaller part of the benchmark than triage and diagnosis. It also says it has not completed a full comparison between LLM judges and human scoring. ^[2]

The launch announcement describes Paige and the broader PD Reliability Platform as generally available from November 3. That establishes product availability per PagerDuty’s statement; it does not establish that the benchmark predicts performance in a particular customer’s environment. ^[1]

Build a test that resembles your own on-call

A team evaluating an SRE agent should use representative service maps, alerts, deploy events and runbooks, while keeping the exercise isolated from real customer traffic. Include both clear faults and misleading signals. Ask reviewers to verify each diagnosis against the underlying telemetry rather than grading a confident explanation as correct.

Measure more than the score after a successful run. Record completion rate, time to a supported diagnosis, wrong turns, source quality, escalation behavior, and whether a proposed mitigation stays within the agent’s permissions. If the agent can trigger remediation, separately test the authorization boundary and make rollback behavior explicit. These are evaluation recommendations, not features established by PagerDuty’s benchmark.

Then test whether the result changes when you add your own incident context. Keep a frozen set of scenarios for regression testing, and avoid sending sensitive transcripts to a system until retention, access controls and redaction have been reviewed. A benchmark can help a team ask better questions; it cannot choose the acceptable risk for that service.

For related coverage, see how teams should verify AI-generated technical documents and the hidden risks of trusted developer tools.