Grafana's o11y-bench 1.1 checks what SRE agents actually changed — not just what they claim

On October 28, 2026, Grafana Labs released o11y-bench 1.1 and added its open observability-agent benchmark to Fireworks' Specialized Intelligence Index. The benchmark runs an agent through 64 tasks against a real, locally started Grafana observability stack with synthetic Prometheus metrics, Loki logs and Tempo traces. Version 1.1 changes how results are checked: reported values are verified against environment facts, trace references are read back from Tempo, and dashboard changes are compared with saved before-and-after state. Failed or empty ground-truth queries become grading errors rather than silently counting toward a pass. A reproducible scenario clock and stored facts permit regrading without rerunning the agent. The repository adds Fireworks inference support and makes task specifications, graders and run artefacts public. Because the grading and task calibration changed, 1.1 scores are not comparable with 1.0 scores.

Developers choosing a model for incident triage need to know whether an agent actually identifies the right service, time range and cause, not merely whether it can write plausible PromQL or describe a convincing incident narrative. Grafana's new outcome checks make it harder for a confident but incorrect answer or a broken dashboard edit to pass. The public harness also lets small teams test the same candidate agent with different model providers, track cost and repeatability, and investigate failures before granting production privileges. The trade-off is that benchmark success measures performance on a controlled synthetic environment, not a guarantee of reliability in a live on-call incident or a specific organisation's data.

The benchmark covers Prometheus/PromQL, Loki/LogQL, Tempo/TraceQL, multi-step incident investigations, Grafana API operations and dashboard creation or repair. An agent works against a running stack using Grafana MCP tools, not just a static question-and-answer dataset. This tests whether it can retrieve and interpret the right evidence and leave the system in the requested state.

For numeric answers, the verifier checks the values against facts from the environment. For trace claims, it retrieves the cited trace and checks support. For dashboard changes, it inspects saved state before and after the run. An empty or failed reference query is treated as a grading error. These are important safeguards against rewarding fluent guesses, although the benchmark still contains rubric-based checks and an LLM grading component.

Each task is attempted three times. Grafana reports both Pass^3, the average score across three attempts, and Pass@3, which measures whether any attempt succeeded. The repository stores task specifications, trajectories, grading artefacts and a scenario clock so results can be examined and regraded. The team says a no-tools control failed all 64 tasks, but that is a developer-reported validation check, not an independent audit.

Fireworks' Specialized Intelligence Index now includes o11y-bench results, and the default benchmark agent can call Fireworks serverless models or customer deployments. The public leaderboard reports scores, reasoning settings, token usage and estimated cost per trial. Teams should compare runs using the same benchmark version and configuration, and account for the extra cost of running their own model and grading calls.

The telemetry is synthetic, the scenarios are finite, and agents can behave differently when faced with noisy production data, partial permissions, changing dashboards and real incident pressure. A result obtained with one model, agent harness, tool set or effort level cannot be assumed to transfer to another. Scores from 1.0 and 1.1 are explicitly not comparable. Teams still need representative internal cases, human review, scoped credentials and a rollback path before enabling agent-driven production changes.