# Grafana's o11y-bench 1.1 checks what SRE agents actually changed — not just what they claim

Grafana's open 64-task observability benchmark now verifies agent answers against telemetry and dashboard state, supports repeatable regrading and appears in Fireworks' specialist model index. Its new scores cannot be compared directly with version 1.0.

The benchmark runs agents against Prometheus, Loki, Tempo and Grafana, checks actual outcomes across three attempts, and makes the task and grading code inspectable. Synthetic tests and vendor-run leaderboard scores still need local validation.

- Status: Active
- Published: 2026-11-02T17:33:32+13:00
- Updated: 2026-11-02T17:33:32+13:00
- Categories: Artificial Intelligence, Cloud & Infrastructure, AI Agents, Observability
- Tags: AI agents, AI evals, Grafana, Observability
- Canonical HTML: https://beyondthe.news/dossiers/grafana-o11y-bench-1-1-grounded-sre-agent-evaluation

## What changed

On October 28, 2026, Grafana Labs released o11y-bench 1.1 and added its open observability-agent benchmark to Fireworks' Specialized Intelligence Index. The benchmark runs an agent through 64 tasks against a real, locally started Grafana observability stack with synthetic Prometheus metrics, Loki logs and Tempo traces. Version 1.1 changes how results are checked: reported values are verified against environment facts, trace references are read back from Tempo, and dashboard changes are compared with saved before-and-after state. Failed or empty ground-truth queries become grading errors rather than silently counting toward a pass. A reproducible scenario clock and stored facts permit regrading without rerunning the agent. The repository adds Fireworks inference support and makes task specifications, graders and run artefacts public. Because the grading and task calibration changed, 1.1 scores are not comparable with 1.0 scores.

## Why it matters

Developers choosing a model for incident triage need to know whether an agent actually identifies the right service, time range and cause, not merely whether it can write plausible PromQL or describe a convincing incident narrative. Grafana's new outcome checks make it harder for a confident but incorrect answer or a broken dashboard edit to pass. The public harness also lets small teams test the same candidate agent with different model providers, track cost and repeatability, and investigate failures before granting production privileges. The trade-off is that benchmark success measures performance on a controlled synthetic environment, not a guarantee of reliability in a live on-call incident or a specific organisation's data.

## Sixty-four tasks exercise more than query syntax

The benchmark covers Prometheus/PromQL, Loki/LogQL, Tempo/TraceQL, multi-step incident investigations, Grafana API operations and dashboard creation or repair. An agent works against a running stack using Grafana MCP tools, not just a static question-and-answer dataset. This tests whether it can retrieve and interpret the right evidence and leave the system in the requested state.

## Version 1.1 grades facts and persisted changes

For numeric answers, the verifier checks the values against facts from the environment. For trace claims, it retrieves the cited trace and checks support. For dashboard changes, it inspects saved state before and after the run. An empty or failed reference query is treated as a grading error. These are important safeguards against rewarding fluent guesses, although the benchmark still contains rubric-based checks and an LLM grading component.

## Repeatability and regrading are part of the design

Each task is attempted three times. Grafana reports both Pass^3, the average score across three attempts, and Pass@3, which measures whether any attempt succeeded. The repository stores task specifications, trajectories, grading artefacts and a scenario clock so results can be examined and regraded. The team says a no-tools control failed all 64 tasks, but that is a developer-reported validation check, not an independent audit.

## The Fireworks index adds a comparison surface

Fireworks' Specialized Intelligence Index now includes o11y-bench results, and the default benchmark agent can call Fireworks serverless models or customer deployments. The public leaderboard reports scores, reasoning settings, token usage and estimated cost per trial. Teams should compare runs using the same benchmark version and configuration, and account for the extra cost of running their own model and grading calls.

## Why a high score is not an on-call approval

The telemetry is synthetic, the scenarios are finite, and agents can behave differently when faced with noisy production data, partial permissions, changing dashboards and real incident pressure. A result obtained with one model, agent harness, tool set or effort level cannot be assumed to transfer to another. Scores from 1.0 and 1.1 are explicitly not comparable. Teams still need representative internal cases, human review, scoped credentials and a rollback path before enabling agent-driven production changes.

## Key details

- Announced October 28, 2026; version 1.1 is an update to an existing open benchmark, not a new Grafana Cloud product.
- 64 tasks in six observability/SRE areas run against Prometheus, Loki, Tempo and Grafana.
- Agents can use Grafana MCP tooling; default harness runs on Harbor.
- Version 1.1 validates cited traces, numeric facts and saved dashboard state, and flags invalid ground truth.
- Three attempts per task support Pass^3 and Pass@3 reliability comparisons.
- Task definitions, grading logic, environment and run artefacts are available in the public GitHub repository.
- Fireworks' Specialized Intelligence Index now includes o11y-bench, and the default agent supports Fireworks inference.
- Version 1.1 results must not be directly compared with version 1.0.

## Builder takeaways

- Use the benchmark to shortlist model and agent combinations for observability workflows, not to authorise autonomous incident response.
- Run the same harness and benchmark version across candidates, with matched effort settings and token/cost accounting.
- Inspect trajectories and verifier artefacts when an agent passes or fails; do not rely on a single aggregate score.
- Add organisation-specific incidents, permissions, dashboards and failure modes to internal tests before deployment.
- Keep human approval and restricted credentials for changes to production dashboards or incident systems.

## What to watch

- Independent reruns of 1.1 results with different harnesses and model configurations.
- How Fireworks' published results distinguish vendor runs from independently submitted results.
- Additional real-world incident tasks and stronger validation of long-running or multi-agent SRE workflows.
- Whether teams demonstrate that higher o11y-bench scores predict fewer operational mistakes in production.

## Uncertainties

- Grafana and partners produced the release claims and initial benchmark results; independent reproduction of version 1.1 was not established in this research pass.
- All 64 tasks use synthetic but realistic telemetry, not representative live incidents.
- Rubric and LLM-based grading can still introduce judgement errors despite new deterministic checks.
- Benchmark cost depends on provider prices, model effort, retries and grading overhead.
- Model rankings may change with benchmark revisions and cannot be compared directly with version 1.0.

## Sources

- [Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1](https://grafana.com/blog/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1/) — Grafana Labs · primary · 2026-10-28T00:00:00+13:00. Official release explanation, 64-task scope, grading improvements, version incompatibility and Fireworks integration.
- [grafana/o11y-bench repository and benchmark instructions](https://github.com/grafana/o11y-bench) — Grafana Labs · primary repository. Inspectable task specifications, runner, grader, saved facts, scenario clock, regrading, provider support and limitations.
- [o11y-bench public leaderboard](https://o11ybench.ai/) — o11y-bench · primary data. Live version-specific scores and cost-per-trial comparison; leaderboard results are not independently audited.
- [o11y-bench: Grafana Just Built the Exam Your AI SRE Agent Will Fail](https://glancerai.com/blog/o11y-bench-grafana-ai-agent-observability-benchmark) — Glancer AI · independent analysis · 2026-07-20T00:00:00+12:00. Independent background on earlier benchmark and Pass^3 semantics; predates version 1.1 and does not verify its new scores.