AI Agent Benchmarks 2026: GAIA, SWE-Bench, OSWorld, Tau²-Bench, WebArena and the New Moore’s Law

Last updated: May 2026 · By Ignacy Kwiećień, founder & editor-in-chief, DecodeTheFuture.org

What is an AI agent benchmark?

An AI agent benchmark is a fixed set of tasks plus an automated grader that lets you compare two agents on the same problem. It is not a regular LLM benchmark like MMLU. Static QA benchmarks ask “given this prompt, return this string”; agent benchmarks ask “given this goal, navigate an environment, call tools, recover from errors, and arrive at the target end-state.” That difference is what makes them harder to score and easier to game.

In 2026 most serious agent benchmarks have three parts: (1) a sandboxed environment — a Linux VM, a real browser, a customer-service simulator; (2) a list of goals expressed in natural language with hidden “ground truth” end-states; (3) a deterministic grader that compares the post-task state against the target. Some still use an LLM-as-judge for partial credit, which, as the methodology section below shows, is the single largest source of noise in published numbers.

This article is the third spoke of the DTF AI Agents cluster. For the foundational picture see What is an AI Agent? Complete Guide for 2026; for how the agent itself is wired together, see AI Agent Architecture Explained; for orchestration patterns, Multi-Agent Systems Explained.

Which 6 AI agent benchmarks matter in 2026?

Hundreds of agent eval suites have been published since 2023. Six of them carry almost all the signal in 2026: they have credible task curation, automated grading, an active leaderboard, and at least one frontier lab citing them in shipping documentation. Everything else is either downstream of these six or too narrow to compare across vendors.

Two practical notes before the deep dives. First, scoreboard timestamps decay: top numbers shift weekly as new model releases land. Second, none of the six is a pure model evaluation — they all measure system performance, where the agent loop, retrieval, and tool definitions matter as much as the underlying model. The same Claude or GPT can score 30 points apart on GAIA depending on the scaffolding around it.

GAIA — the general-assistant benchmark

GAIA was introduced by Meta and Hugging Face in late 2023 as the first agent benchmark that asks questions a human assistant could plausibly receive at work: “find the second author of paper X, look up their h-index on Google Scholar, calculate the percentile against the median in the field.” Tasks span three difficulty levels and require some combination of web search, file parsing, math, and code. Humans score around 92%; pre-agent LLMs scored under 10%.

The headline number on GAIA depends entirely on which leaderboard you read. The Princeton HAL “scaffolded” leaderboard, which allows agents to use full tool stacks, currently shows Claude Sonnet 4.5 at 74.6%, with Anthropic occupying the top six positions. The “bare model” leaderboard, which strips scaffolding and tests the model’s intrinsic agentic ability, has GPT-5 Mini at 44.8%. The Steel.dev system-level leaderboard shows OPS-Agentic-Search at 92.36%. That 30-to-50 point spread on the same tasks is the single most important fact about agent benchmarks in 2026 — framework value frequently dwarfs model differences.

SWE-Bench Verified — the coding agent gold standard

SWE-Bench was released in 2024 by Princeton; OpenAI then audited a 500-issue subset and re-released it as SWE-Bench Verified. Each task is a real GitHub issue from one of 12 popular Python repos (Django, sympy, scikit-learn, etc.) plus the original maintainer’s patch as ground truth. The agent must produce a code change that passes the maintainer’s actual test suite.

The May 2026 standings: Claude Mythos Preview leads at 93.9%, GPT-5.5 at 88.7% (released April 23, 2026), Claude Opus 4.7 Adaptive at 87.6% (April 16, 2026), GPT-5.3 Codex at 85%, Claude Opus 4.6 at 80.8%, Claude Sonnet 4.6 at 79.6%, with DeepSeek V4 Pro Max and Gemini 3.1 Pro tied at 80.6%. Practical caveat: the OpenAI audit released in February 2026 found that 59.4% of the hardest Verified tasks have test suites that wouldn’t actually catch the intended bug, so the upper end of the leaderboard is partly an artifact of weak tests rather than agent capability. Independent contamination analysis estimates inflation of 5–15 points on post-2023 models.

OSWorld — computer use, not just code

OSWorld (NeurIPS 2024, XLang Lab) drops an agent into a real Ubuntu virtual machine with screenshots and keyboard / mouse access. Tasks include “extract this column from a downloaded CSV and email the result,” “rebuild this LibreOffice slide from a PDF,” “configure VS Code to use ESLint.” There are 369 tasks split across multiple OS apps and web apps, all with deterministic post-condition checks (file diffs, registry state, network calls).

OSWorld is the most honest of the six because no amount of prompt-engineering can fake a file actually being saved. May 2026 standings: Claude Mythos Preview at 79.6%, GPT-5.4 at 75.0%, Claude Opus 4.6 at 72.7%, Claude Sonnet 4.6 at 72.5%. The human baseline ranges from 72–84% depending on category — meaning frontier agents have hit roughly human level on the average task, while still failing on the 20% that require fine motor skills or unusual menus.

Tau²-Bench — the policy adherence test

Tau²-Bench (Sierra Research, 2024 with major 2026 expansion) is the most enterprise-relevant of the six. It simulates customer-service domains — retail, airline, telecom — where another LLM plays the user and the agent must complete the task and respect a written policy. An agent that books the right flight but waives a non-waivable change fee fails the task. That maps directly onto how regulated industries actually deploy agents.

The benchmark introduces two ideas worth borrowing for any private eval: a dual-control architecture (the user is also a simulated agent with its own tools and database, so the conversation actually changes shared world-state), and the pass^k metric, which measures how often the agent succeeds across k repeated trials — not just once. Single-run pass rates can hide variance of 20+ points; pass^4 or pass^8 expose it. The 2026 update grew to 38 model entries (April 13, 2026) and added voice and knowledge-retrieval domains.

WebArena — realistic browser tasks

WebArena (CMU, 2023) hosts self-contained replicas of real web apps — a OneStopShop e-commerce site, a Reddit clone, a GitLab clone, a CMS, an OpenStreetMap instance — and asks the agent to complete 812 multi-step browser tasks. Examples: “find a hat under $30, leave a 5-star review, message the seller asking about shipping.” Grading is post-condition based: did the right database state change?

The May 2026 leaderboard is unusually tight: Claude Mythos Preview at 68.7%, GPT-5.4 Pro at 65.8%, Claude Opus 4.6 at 64.5% — a 4.2-point spread between #1 and #3, and a 16.6-point spread across the top 10. Two years ago the state of the art was 14%; the jump to 60%+ is the most dramatic capability expansion in any agent benchmark. ServiceNow released WebArena Verified in 2025 to address contamination, since the original 2023 tasks have leaked into post-cutoff training data.

METR HCAST + Time Horizons — the new Moore’s Law

METR’s contribution is structural rather than score-based. Instead of asking “how many tasks does the agent complete?”, they ask “what is the longest task length, in human minutes, at which the agent succeeds 50% of the time?” The benchmark, HCAST (Human-Calibrated Autonomy Software Tasks) plus RE-Bench plus a set of shorter novel tasks, contains about 230 software tasks calibrated against human completion times.

The plot of “50% time horizon” against release date is the most-cited graph in agent research right now. Task length and success rate are correlated at R² = 0.83; the time horizon is doubling roughly every 4 months as of 2024–2025 (down from every 7 months over 2019–2025). Extrapolating: frontier agents in mid-2026 reliably finish ~2-hour software tasks at 50%; by 2027 a full 8-hour workday; 2028 a week; 2029 a month. The extrapolation breaks at some point — almost certainly — but the doubling has held through six model generations. METR’s January 2026 update (“Time Horizon 1.1”) confirmed the trend with refined methodology.

Why do agent leaderboards diverge so much?

If you look at GAIA across leaderboards in May 2026 you see scores between 44.8% and 92.36% on the same 466 tasks. That is not a measurement error — it is the entire point. The difference between leaderboards is what you are allowed to bring to the test:

• Bare model. Just the LLM with the simplest possible loop. No retrieval-augmented generation, no scratchpad, no domain-specific tools. Numbers tend to land 30–50 points lower.
• Scaffolded. The vendor’s published agent harness with multi-step planning, custom tools, retries. This is what users actually get when they call claude.ai or ChatGPT.
• Full system. Custom tool stacks, fine-tuned routers, curated retrieval indexes. This is what enterprise integrators ship. Numbers can exceed human baselines on “easy” tasks but rarely transfer to private workloads.

For a buying decision the bare-model number tells you about the model; the scaffolded number tells you about the vendor’s product; the system number tells you about the integrator. Confusing them is how teams end up disappointed when their pilot scores 30 points worse than the leaderboard claimed.

Five reasons agent benchmark scores lie

Even when leaderboards are honest about scaffolding, the published numbers overstate real-world capability. Five mechanisms explain almost all the gap between benchmark scores and pilot results.

1. Training-data contamination

Most agent benchmarks were public before frontier model cutoffs. SWE-Bench tasks have appeared verbatim in pre-training data; HumanEval problems are near-duplicates of LeetCode solutions; MMLU questions surface unedited in Common Crawl. The OpenAI February 2026 audit of SWE-Bench Verified found that 59.4% of the hardest tasks had tests that would pass even when the underlying bug was unfixed, and an independent analysis showed roughly one-third of all SWE-Bench issues contain solutions in the comments. Estimated inflation on post-2023 models: 5–15 points. Verified subsets (SWE-Bench Verified, OSWorld-Verified, WebArena-Verified) reduce but do not eliminate this.

2. LLM-as-judge noise

Many benchmarks fall back on an LLM to grade partial credit when ground truth is hard to encode. Multiple 2025–2026 audits found error rates above 50% in LLM judges, driven by three biases: position bias (preferring the answer shown first roughly 60% of the time), length bias (longer answers are scored higher regardless of quality), and agreeableness bias (over-accepting outputs without critical evaluation). Swapping the position of two identical answers flips the verdict in about 40% of pairs. Any score with an LLM judge in the loop should be reported with a confidence interval that almost no leaderboard provides.

3. Single-run reporting

Almost all leaderboards report a single trajectory per task. Agent variance per run is large — pass^4 scores often run 15–25 points below pass^1. A 90% benchmark score sometimes corresponds to 70% reliability in production, where the same task gets retried by different sessions. Tau²-Bench’s pass^k metric is the right answer here; few other benchmarks report it.

4. No cost or safety in scoring

Of the major agent benchmarks, 0 / 15 integrate cost-efficiency or safety into their primary scoring rubric. A score of 88% on SWE-Bench achieved with $50 of inference per task is treated as identical to one achieved with $0.50. 13 / 15 rely on binary success metrics, ignoring partial completion or graceful failure. For procurement decisions you have to compose a cost layer yourself: most teams divide pass-rate by dollars-per-task to get a real ROI ranking.

5. Search-time contamination

For benchmarks that allow web access (GAIA, WebArena), the open internet has been steadily polluted with leaderboard answers and walkthroughs. An agent that “browses to find the answer” is increasingly likely to land on a page that already contains the solution, especially for older tasks. This is hard to control for and quietly inflates GAIA-style scores by an unknown but probably non-trivial amount.

How to read 2026 leaderboards in five steps

A practical rubric for converting benchmark numbers into procurement signal:

1. Read the methodology page first. Bare model? Scaffolded? Verified subset? Pass^1 or pass^k? Without that, the score is meaningless.
2. Match the benchmark to the workload. Building a coding assistant? SWE-Bench Verified is necessary, OSWorld is a tie-breaker. Building a customer-service bot? Tau² first, GAIA second. Building a research assistant? GAIA + WebArena. Computer-use product? OSWorld is the only honest signal.
3. Look at three benchmarks, not one. Single-benchmark optimization is the agent equivalent of teaching to the test. A model that wins three out of six is the actual frontier; a model that wins one is sometimes a one-trick scaffolding pony.
4. Run a private 50-task private eval. Sample tasks from your real workload, gold-label them, and grade your top three candidates yourself. Ten engineering hours of private evals is worth more than ten weeks of leaderboard reading.
5. Track METR time horizons separately. If you are sizing a multi-year deployment, the time horizon doubling tells you when current limitations stop biting more reliably than any pass-rate trend.

Building your own evals — when to stop trusting public benchmarks

For any production agent, the moment to stop relying on public benchmarks is the moment you have 50 real workload tasks and a deterministic grader. The grader is the hard part — the agent’s “did I succeed?” cannot grade itself, and an LLM-as-judge re-introduces the noise discussed above.

The simplest pattern that survives contact with reality:

import json, statistics, time
from your_agent import run_agent
from your_graders import deterministic_check  # NOT an LLM

def evaluate(tasks: list[dict], k: int = 4) -> dict:
    runs = []
    for task in tasks:
        outcomes = []
        for trial in range(k):
            t0 = time.time()
            result = run_agent(task["prompt"], env=task["env"])
            ok = deterministic_check(result, task["target_state"])
            outcomes.append({
                "ok": ok,
                "tokens": result.tokens,
                "seconds": time.time() - t0,
            })
        pass_at_1 = outcomes[0]["ok"]
        pass_at_k = all(o["ok"] for o in outcomes)
        runs.append({"task": task["id"], "pass@1": pass_at_1, "pass^k": pass_at_k,
                     "median_tokens": statistics.median(o["tokens"] for o in outcomes)})
    return {
        "pass@1": sum(r["pass@1"] for r in runs) / len(runs),
        "pass^k": sum(r["pass^k"] for r in runs) / len(runs),
        "median_tokens": statistics.median(r["median_tokens"] for r in runs),
        "tasks": runs,
    }

Three properties make this honest where leaderboards aren’t: it is deterministic (no LLM judge), it reports both pass@1 and pass^k so variance is visible, and it tracks tokens-per-success so you can build the cost ranking that public benchmarks don’t.

Personal note — the deterministic verifier behind DTF

The agent that helps draft DTF articles uses a benchmark-of-one: scripts/check_seo.py, a 200-line Python script that asserts twelve specific properties of an article (title length, meta description length, slug format, AIO box markup, FAQ JSON-LD shape, SVG accessibility metadata, bibliography wrapper, internal-link count). The agent writes the article; the script grades it deterministically; the agent retries on any error.

That is the same pattern as a serious agent benchmark, scaled down to one workload. There is no LLM judge anywhere — the grader is a function that returns 0 or 1 with no opinions. The result, after about thirty articles, is that the agent now passes 12/12 on the first try roughly 80% of the time, exactly because the eval is honest. If I’d graded it with another LLM I would have shipped a lot of broken articles convinced they were fine. The general lesson: if your agent is doing real work, build a deterministic eval before you build features.

What benchmarks miss — directions for 2027

Three gaps in the current benchmark landscape are likely to drive the next research wave:

• Long-horizon reliability. Tau²’s pass^k is a start, but no current benchmark tests an agent over a multi-day session with persistent memory. METR’s time-horizon work is the closest proxy.
• Cost-aware scoring. The first benchmark that publishes a Pareto frontier of pass-rate vs dollars-per-task will do for agents what MLPerf did for inference. Galileo and a handful of academic groups are working on this in 2026.
• Safety + policy under adversarial users. Tau² tests benign policy adherence; almost nothing tests prompt injection, indirect injection, or jailbreaks at the task-completion level. OWASP’s LLM Top 10 (2025) covers the threats but not the benchmark format. Expect a “SafeBench-Agent” entrant before the end of 2026.

For the architectural picture of how memory, tools, and orchestration shape what an agent can be evaluated on at all, return to our AI Agent Architecture Explained and Multi-Agent Systems Explained spokes, or to the cluster hub. For the broader market context, our best AI coding assistants 2026 review interprets SWE-Bench Verified for buying decisions, and ChatGPT vs Claude vs Gemini 2026 compares the underlying frontier models.

FAQ — AI agent benchmarks 2026