HomeArtificial IntelligenceAI Agent Benchmarks 2026: Leaderboards for SWE-bench, OSWorld, GAIA, Tau²-Bench, Terminal-Bench and...

AI Agent Benchmarks 2026: Leaderboards for SWE-bench, OSWorld, GAIA, Tau²-Bench, Terminal-Bench and METR

Last updated: 13 October 2026 (all leaderboards re-checked that day) · By Ignacy Kwiećień, founder & editor-in-chief, DecodeTheFuture.org

The AI agent benchmarks worth tracking in 2026 are SWE-bench Pro and Terminal-Bench (coding agents), OSWorld-Verified and the new OSWorld 2.0 (computer use), WebArena (browser agents), GAIA and BrowseComp (research and agentic search), τ²/τ³-bench (customer-service tool use) and METR’s time horizon (long autonomous tasks). As of 13 October 2026, GAIA (94.0%) and OSWorld-Verified (90.2%) top scores sit above their published human baselines, so they no longer separate frontier agents; the benchmarks that still do are OSWorld 2.0 (best 20.6%), SWE-bench Pro (best 61.5%) and Terminal-Bench 4.0 (best 64.9%). OpenAI stopped reporting SWE-bench Verified in February 2026 because of contamination and flawed tests, and METR stopped updating its time-horizon page in September 2026.

SWE-bench Pro Terminal-Bench 4.0 OSWorld 2.0 GAIA τ²-bench WebArena METR

AI agent benchmarks 2026: top scores as of 13 October 2026

The table lists the highest score on each official or maintainer-run leaderboard, plus the date that score was posted. It does not use vendor launch posts, because vendors run their own harnesses, and those numbers are often higher than anything on the public leaderboards. When a benchmark has no public leaderboard we say so and give no number. Please re-check the links before you quote a figure: these tables change every week.

Benchmark (agent type) Top listed score Entry Date of entry / page Status
SWE-bench Pro, public set (coding) 61.5% ± 3.1 Muse Spark 1.1, mini-swe-agent harness Page checked 13 Oct 2026 (no update date shown) ~30% tasks flagged
Terminal-Bench 4.0 (terminal / coding) 64.85% ± 3.11 Claude Code + Claude Opus 5.5 (max effort) Entry dated 22 Sep 2026; board updated 6 Oct 2026 Active
SWE-bench Verified (coding) 79.2% Sonar Foundation Agent and live-SWE-agent, both on Claude Opus 4.5 (tie) Latest entry on board: 26 Feb 2026 Contaminated
OSWorld-Verified (computer use) 90.19% Intelligence-Indeed Agent (agentic framework, 100 steps) 25 Jul 2026 Above human baseline
OSWorld 2.0 (long computer-use workflows) 20.6% binary / 54.8% partial Claude Opus 4.8, max thinking, 500-step limit Project page checked 13 Oct 2026 (paper 28 Jun 2026) Unsolved
GAIA, test set (general assistant / search) 94.0% Shadow Harness V8.01 (multi-model system) 12 Oct 2026 Above human baseline
τ²-bench, text (customer-service tools) 87.9% pass^1 Qwen3.5-397B-A17B Page checked 13 Oct 2026 Near ceiling
τ³-Banking (knowledge retrieval + policy) 55.2% pass^1 Qwen 3.8 Max Page checked 13 Oct 2026 Active
WebArena (browser) 74.3% WebTactix with DeepSeek V3.2 Feb 2026 (newest row: Jun 2026) Near human 78.24%
METR time horizon (long tasks) ~12.0 h (50% horizon) Claude Opus 4.6. Claude Mythos Preview (early) measured ~17.4 h, but METR flags anything above 16 h as unreliable Page frozen 8 Sep 2026 Suite topped out
BrowseComp (agentic search) — No official public leaderboard. The scores in circulation are vendor self-reports. — Active

How these were collected: we read each leaderboard’s own data on 13 October 2026. That means the JSON on swebench.com, the OSWorld-Verified results spreadsheet, the GAIA results_public dataset on Hugging Face, the Terminal-Bench 4.0 board, METR’s benchmark_results_1_1.yaml and the WebArena sheet. Model names are copied exactly as each board lists them.

The pattern is more useful than any single number. The older public suites (GAIA, OSWorld-Verified, τ²-bench text, WebArena) now top out near or above human baselines. They can tell you whether an agent is competent. They can no longer tell you which frontier agent is better. The suites that still spread models apart are the newer and longer ones: OSWorld 2.0, SWE-bench Pro, Terminal-Bench 4.0 and τ³-Banking.

What is an AI agent benchmark?

An AI agent benchmark is a fixed set of tasks with an automated grader, so you can compare two agents on the same problems. It is different from a static LLM benchmark like MMLU. A static benchmark gives a prompt and expects a particular answer string. An agent benchmark gives a goal and expects the agent to work through an environment, call tools, recover from errors and leave the environment in a target end-state. Scoring that is harder, and gaming it is easier.

Most serious agent benchmarks have three parts:

  1. A sandboxed environment: a Linux VM, a self-hosted website, a terminal container or a simulated customer.
  2. Goals written in natural language, each with a hidden ground-truth end-state.
  3. A grader that checks the final state against that ground truth.

Some suites still use an LLM judge for partial credit or for tasks on live websites. That is a major source of noise, and the methodology section below explains why.

This article is part of the DTF AI Agents cluster. For the basics, start with What is an AI Agent?. For how agents are wired together, see AI Agent Architecture Explained. Coordination patterns are covered in Multi-Agent Systems Explained.

Which benchmark fits which agent type?

Start from the agent you are building, not from whichever benchmark is in the news. Here is how each type of agent maps to the suites that still give useful signal in October 2026:

Agent type Primary benchmark Second opinion Avoid as a headline
Coding agent working in a repoSWE-bench Pro (public + commercial results)Terminal-Bench 4.0SWE-bench Verified
Terminal / DevOps / data-wrangling agentTerminal-Bench 4.0METR time horizon (trend only)Single-run vendor claims
Desktop computer-use agentOSWorld 2.0OSWorld-Verified (as a minimum bar)OSWorld-Verified ranks at the top
Browser / web-automation agentWebArena-Verified (or its 258-task hard subset)Online-Mind2Web (live sites)Original WebArena with LLM-judged tasks
Research / agentic search agentBrowseCompGAIA (test set, Level 3)GAIA validation-set scores
Customer-service agent with policiesτ²-bench with pass^kτ³-Banking (knowledge base)Pass^1 alone
Voice agentτ³-VoiceYour own call recordingsText-only scores
Long-running autonomous agentMETR time horizonOSWorld 2.0 (250+ steps per task)Any short-task pass rate

Coding agents: SWE-bench Verified, SWE-bench Pro and Terminal-Bench

SWE-bench Verified: contaminated and no longer reported by OpenAI

SWE-bench (Princeton, 2023) gives an agent a real GitHub issue from one of 12 Python repositories. The agent passes only if the project’s hidden tests pass after its patch. In August 2024 OpenAI released SWE-bench Verified, a 500-task subset that expert engineers had checked by hand. For two years it was the standard coding-agent number in model launches.

That ended on 23 February 2026, when OpenAI stopped reporting SWE-bench Verified. OpenAI audited 138 problems that o3 did not consistently solve, about 27.6% of the set. It found:

  • 59.4% of those problems had flawed tests or problem descriptions. Tests that are too narrow (35.5%) reject correct solutions. Tests that are too wide (18.8%) check behaviour the task never asked for.
  • Every frontier model OpenAI tested could reproduce gold patches or problem details for some tasks. That is evidence of training-data contamination.

An earlier independent study, SWE-Bench+, looked at the original SWE-bench. It found that 32.67% of successful patches had the solution sitting in the issue text or comments, and that 31.08% of passed patches got through only because the tests were weak.

The official swebench.com leaderboard is still online. Its top Verified score is 79.2%, a tie between Sonar Foundation Agent and live-SWE-agent, both running Claude Opus 4.5. The newest entry is dated 26 February 2026. On the bash-only track, which runs every model through the same minimal mini-SWE-agent harness, Claude Opus 4.5 (high) leads with 76.8%. Some 2026 launch posts report higher Verified scores from the vendor’s own harness. Those are self-reported, and none of them appears on the official board.

SWE-bench Pro: harder, but also audited

SWE-bench Pro comes from Scale AI (September 2025). It has 1,865 long-horizon problems from 41 repositories, split three ways: a 731-task public set, a held-out set, and a commercial set built from startups’ private codebases. Scale designed it to resist contamination. On the public leaderboard checked on 13 October 2026, the top entries are:

  1. Muse Spark 1.1: 61.5% ± 3.1
  2. GPT-5.4 (xHigh): 59.1% ± 3.6
  3. Muse Spark: 55.0%
  4. Claude Opus 4.6 (thinking): 51.9%

All four ran on the mini-swe-agent harness.

There is a caveat here too. On 8 July 2026 OpenAI published an audit of SWE-bench Pro. Its automated pipeline flagged 200 tasks (27.4%) as broken. Human engineers flagged 249 (34.1%). OpenAI estimates that about 30% of Pro tasks are broken, and it formally withdrew its earlier advice to switch to Pro. Read Pro scores as useful for rough ranking, with ±3-point error bars and a known share of bad tasks underneath.

Terminal-Bench 4.0: the most actively maintained coding board

Terminal-Bench (Stanford and the Laude Institute, paper January 2026) gives agents hard, realistic tasks inside a command-line container, each with a verifier. Version 2.0 shipped in November 2025. Version 4.0 followed on 28 August 2026 and changed how the benchmark is maintained:

  • It is now a continuous benchmark with semantic versioning.
  • Every task gets a flat 8-hour agent timeout.
  • 8 tasks were removed: 2 because they were saturated, 2 because of refusals, 2 because their solutions were public, and 2 for quality or platform issues. 19 more were fixed.

On the 4.0 leaderboard (last updated 6 October 2026), the top entries are:

  1. Claude Code + Claude Opus 5.5 (max effort): 64.85% ± 3.11
  2. Claude Code + Sonnet 5.5 (max): 61.82% ± 2.94
  3. Codex + GPT-6 Astra (max): 58.18% (tied with GPT-6.1 Sol)

Every row reports cost and token counts. Few agent boards do, and you need those figures to compare cost-effectiveness. Our best AI coding agents 2026 guide turns these numbers into tool choices.

Computer-use agents: OSWorld leaderboard 2026

OSWorld (NeurIPS 2024, XLang Lab) runs agents inside a real Ubuntu VM. The agent gets screenshots and keyboard and mouse control, and works across LibreOffice, Chrome, GIMP, Thunderbird, VS Code and multi-app workflows. There are 369 tasks, and 361 if you exclude the ones that need Google Drive. Each task has an execution-based checker that inspects the real end-state. The original paper put humans at 72.36% and the best model at 12.24%.

OSWorld-Verified (28 July 2025) fixed the tasks the community had reported and moved evaluation to AWS. In the maintainers’ verified results, the top entries as of 13 October 2026 are:

  • 90.19%: Intelligence-Indeed Agent, an agentic framework, 100 steps, 25 July 2026.
  • 85.96%: claude-fable-5, the best general model entry, listed by Anthropic on 1 August 2026.
  • 83.64%: Pointer Agent with Opus 4.7.
  • 83.39%: claude-opus-5.

16 of the 149 verified entries now score above the 72.36% human baseline. So OSWorld-Verified tells you whether a computer-use agent is competent, but it can no longer rank frontier agents. The site also keeps a separate “self-reported” tab. Keep those numbers apart from the verified ones.

The maintainers’ answer is OSWorld 2.0, published 28 June 2026. It has 108 long-horizon workflows across 7 professional domains, running on 31 self-hosted websites. The median human completion time is about 1.6 hours, and tasks average more than 250 agent steps and 27.25 scoring checkpoints. The project page names Claude Opus 4.8 (max thinking, 500-step limit) as the top scorer, with 20.6% binary completion and 54.8% partial score. Claude Opus 4.7 reaches 18.2%. GPT-5.5 uses the fewest tokens but plateaus around 14%. If you search for an “OSWorld benchmark leaderboard 2026”, these two boards answer different questions. OSWorld-Verified asks whether the agent can do typical desktop tasks. OSWorld 2.0 asks whether it can finish an hour-plus workflow without supervision.

Browser agent benchmarks 2026: WebArena and successors

WebArena (CMU, 2023) hosts working replicas of an online shop, a forum, a GitLab instance, a CMS and a map service. It asks agents to complete 812 multi-step tasks on them. The 2023 paper reported 14.41% for the best GPT-4 agent and 78.24% for humans. The maintainers’ leaderboard sheet is now led by WebTactix with DeepSeek V3.2 at 74.3% (February 2026), then OpAgent at 71.6% and ColorBrowserAgent at 71.2%. The sheet has a “Result Source” column, and several rows are marked self-reported.

Three newer benchmarks fill the gaps:

  • WebArena-Verified (ServiceNow). All 812 tasks, reference answers and evaluators were manually reviewed. LLM-judge and substring scoring were replaced with deterministic comparison. A 258-task “hard” subset makes runs cheaper. It became available on PyPI in January 2026. It has no public leaderboard yet, so it is best for your own runs.
  • Online-Mind2Web (OSU, 2025). 300 tasks on 136 live websites. Its paper found that earlier results were over-optimistic. It uses an LLM judge that agrees with humans about 85% of the time, so allow for judge noise.
  • BrowseComp, covered below. It tests whether a browsing agent can find hard-to-locate facts, not whether it can operate a website.

Agentic search benchmarks: GAIA and BrowseComp

GAIA leaderboard 2026

GAIA (Meta and Hugging Face, 2023) has 466 assistant-style questions that need browsing, file handling, code and reasoning. In the original paper, humans scored 92% and GPT-4 with plugins scored 15%. Answers to about 300 test questions are kept hidden to power the official leaderboard. There are two GAIA leaderboards, and they are easy to confuse:

  • Official test-set leaderboard (Hugging Face). As of 13 October 2026 it is led by Shadow Harness V8.01 at 94.0%, submitted 12 October 2026. Close behind are Ops-Agentic-Search-2.0 at 93.7% (22 September 2026) and CustomGPT.ai Research Lab v44 at 93.4%. Most of these are multi-model systems, not single models. The board also shows heavy hill-climbing: one team made 29 “Shadow” submissions between 9 September and 12 October 2026, rising from 73.4% to 94.0%. Repeated submissions like that raise the risk of overfitting to the hidden test set.
  • Princeton HAL (hal.cs.princeton.edu/gaia). This board runs standard scaffolds on the 165-question validation set. Its best result is the HAL Generalist Agent with Claude Sonnet 4.5 at 74.55%. HAL has paused adding new models. On 5 October 2026 it also noted that its reported costs are overstated, by up to 2× for some models.

A score above 92% on a set whose questions have been public since 2023 does not show super-human research ability. It shows that GAIA is saturated as a ranking tool. Level-3 accuracy and cost per task are still worth comparing.

BrowseComp

BrowseComp (OpenAI, April 2025) has 1,266 questions. Each answer is short and easy to check but buried deep on the web, so the agent has to search persistently and creatively. It is the benchmark most often cited for deep-research and agentic search. However, we found no maintainer-run public BrowseComp leaderboard. The numbers you see come from model launch posts, each using that vendor’s own browsing tools. So we give no “top score” here. When comparing vendors, check whether they used the same search tool and whether they blocked pages that contain the answers.

Tool-calling and customer-service agents: τ-bench family

τ-bench (Sierra, June 2024) simulates a customer through an LLM and asks the agent to resolve the request with API tools while following a written policy. Grading compares the final database state with the goal. It introduced the pass^k metric, which only counts a task if the agent succeeds on all k independent trials. In the 2024 paper, gpt-4o solved fewer than 50% of tasks, and its retail pass^8 was below 25%. That gap is why single-run scores mislead.

The family has grown several times since:

  • τ²-bench (June 2025) added a telecom domain with “dual control”, where the user can also act on the environment.
  • A February 2026 audit fixed more than 50 retail and airline tasks.
  • March 2026 brought τ-Knowledge, a banking domain with about 700 documents, and τ-Voice, which tests full-duplex voice agents.

The leaders on taubench.com as of 13 October 2026, all by pass^1:

  • τ²-bench text: Qwen3.5-397B-A17B at 87.9%, Gemini 3.0 Pro at 85.4%, Claude Opus 4.5 at 85.3%.
  • τ³-Banking: Qwen 3.8 Max at 55.2%, Claude Opus 5 at 48.7%, Grok 4.5 at 47.9%.
  • τ³-Voice: gpt-live-1 at 81.7%.

Submissions are labelled Verified only when trajectories are provided and prompts are unmodified. The maintainers ask for at least 4 trials per domain, so pass^k can be computed. For an enterprise buyer, τ³-Banking with pass^4 is the most informative public number in this family.

METR time horizons: the “new Moore’s law” for agents

METR asks a different question from the other benchmarks. Instead of counting solved tasks, it asks how long a task, measured in skilled-human time, an agent can finish with 50% reliability. The task suite (HCAST, RE-Bench and short SWAA tasks) was expanded in Time Horizon 1.1 in January 2026. Here are METR’s published 1.1 estimates (metr.org/time-horizons, data file benchmark_results_1_1.yaml):

Model (release)50% time horizon95% CI
GPT-4 (Mar 2023)~4 min2–8 min
o3 (Apr 2025)~2.0 h1.2–3.2 h
GPT-5 (Aug 2025)~3.4 h1.9–6.8 h
Claude Opus 4.5 (Nov 2025)~4.9 h2.7–10.4 h
GPT-5.2 (Dec 2025)~5.9 h3.3–13.6 h
Claude Opus 4.6 (Feb 2026)~12.0 h5.3–60.6 h
GPT-5.4 (Mar 2026)~5.7 h3.1–12.8 h
Claude Mythos Preview, early (Apr 2026)~17.4 h*8.5–55.1 h

* METR, 8 May 2026: “Measurements above 16 hrs are unreliable with our current task suite.” The page has said it is “no longer actively updated” since 8 September 2026.

METR 50% time horizon of frontier AI models, 2022–2026 Log-scale chart of METR Time Horizon 1.1 estimates: GPT-3.5 Turbo Instruct (Mar 2022) about 0.6 minutes, GPT-4 (Mar 2023) 4 minutes, Claude 3.5 Sonnet (Oct 2024) 20 minutes, o3 (Apr 2025) 2 hours, GPT-5 (Aug 2025) 3.4 hours, Claude Opus 4.5 (Nov 2025) 4.9 hours, Claude Opus 4.6 (Feb 2026) 12 hours, Claude Mythos Preview early (Apr 2026) 17.4 hours. A dashed line at 16 hours marks METR’s stated reliability limit of its current task suite. Doubling time since 2023 is about 129 days (95% CI 104 to 158 days). METR 50% time horizon of frontier AI models, 2022–2026 DecodeTheFuture.org AI agent benchmarks 2026, METR time horizon, Time Horizon 1.1, doubling time, frontier agents METR Time Horizon 1.1 point estimates for selected frontier models on a logarithmic scale, with METR’s 16-hour reliability limit. Data: metr.org benchmark_results_1_1.yaml, retrieved 13 October 2026. https://metr.org/time-horizons/ 2026-10-13 Diagram image/svg+xml en © DecodeTheFuture.org METR 50% time horizon · 2022–2026 Task length (human time) finished 50% of the time · log scale 1 min 10 min 1 h 4 h 16 h METR: above 16 h unreliable with current tasks 2022 2023 2024 2025 2026 GPT-3.5 Instruct · 0.6 min GPT-4 · 4 min Claude 3.5 Sonnet · 20 min o3 · 2.0 h GPT-5 · 3.4 h Claude Opus 4.6 · 12 h Mythos Preview (early) · 17.4 h* Doubling since 2023: ~129 days (95% CI 104–158) Unlabelled dots: GPT-4o, o1, Opus 4.5, GPT-5.2 · point estimates only Source: METR Time Horizon 1.1 data, retrieved 13 Oct 2026

METR’s data file puts the doubling time at about 129 days (95% CI 104–158) for models since 2023, and about 188 days over the whole series. Both figures exclude points above 16 hours. Three caveats matter more than the trend line:

  • A 50% horizon is not a reliability guarantee. In METR’s data, each model’s 80% horizon is roughly 2 to 10 times shorter.
  • The confidence intervals for the newest models span roughly a factor of ten.
  • The task suite has run out of long enough tasks, which is why METR flags anything above 16 h and has stopped updating the page.

Treat any extrapolation (“a work-week by year X”) as a forecast, not a measurement.

How to read an AI agent leaderboard: a 9-point checklist

Run through this list before you quote a leaderboard number in a deck or a buying decision. Each point comes from a real failure described above.

  1. Two dates. When was the entry submitted, and when was the board last updated? SWE-bench Verified has had no new entries since February 2026. METR’s page is frozen.
  2. Which split? Validation sets are public and easy to tune on (HAL’s GAIA uses 165 validation questions). Hidden test sets are stronger, but many repeat submissions can still overfit them.
  3. Harness and limits. Same scaffold (mini-swe-agent, bash-only) or a custom system? What were the step limits (100 steps on OSWorld-Verified, 500 on OSWorld 2.0) and timeouts (8 h on Terminal-Bench 4.0)?
  4. Verified or self-reported? τ-bench marks Verified runs. OSWorld keeps a separate self-reported tab. WebArena’s sheet has a “Result Source” column. Vendor launch posts are self-reported by definition.
  5. Error bars and trials. SWE-bench Pro and Terminal-Bench publish ±3-point intervals, so a 2-point lead is noise. Prefer pass^k over a single run.
  6. Human baseline and saturation. Once the top scores pass the human baseline (GAIA ~92%, OSWorld 72.36%), use the benchmark as a minimum bar, not a ranking.
  7. Validity audits. Has anyone audited the tasks? OpenAI found flawed tests in 59.4% of the audited SWE-bench Verified problems and an estimated ~30% broken tasks in SWE-bench Pro.
  8. Contamination exposure. Were the tasks public before the model’s training cutoff? Can answers be found on the open web during evaluation?
  9. Cost per success. A score without tokens or dollars is half the picture. Terminal-Bench reports both. HAL warns that its cost figures are overstated.

Why the same model scores so differently across leaderboards

Most agent numbers measure a system, not a model. GAIA shows this well. The same family of Claude models scores 74.55% on HAL’s fixed generalist scaffold (validation set). Purpose-built multi-model systems score 93–94% on the test set. What you are allowed to bring to the test makes a big difference:

  • Bare or minimal harness. The model runs in a simple loop with standard tools, as in SWE-bench bash-only or SWE-bench Pro’s mini-swe-agent. This tells you most about the model itself.
  • Vendor product harness. Claude Code, Codex and similar tools, as on Terminal-Bench 4.0. This tells you what a user of that product gets.
  • Full custom system. Routers, several models, retrieval indexes and retries, as at the top of GAIA and OSWorld-Verified. This tells you about an integrator’s engineering. It rarely transfers to your workload.

Four reasons agent benchmark scores overstate real-world ability

1. Training-data contamination

Most public agent tasks were online before the latest models’ training cutoffs. OpenAI found that every frontier model it tested could reproduce some SWE-bench Verified gold patches. The SWE-Bench+ study found that over 94% of SWE-bench issues were created before the models’ knowledge cutoffs. Newer suites fight this with held-out sets (SWE-bench Pro), private commercial repositories, and removing tasks once solutions leak (Terminal-Bench 4.0 dropped two for that reason).

2. Flawed graders

Graders can be wrong in both directions. Tests that are too strict reject correct fixes. Tests that are too weak accept broken ones. Some benchmarks fall back on LLM judges, which have documented position, verbosity and self-enhancement biases (Zheng et al., 2023). Deterministic end-state checks, as in OSWorld and WebArena-Verified, are the better default, but only if the checks themselves have been audited.

3. Single-run reporting

Agents are stochastic. In the original τ-bench paper, gpt-4o solved fewer than half the retail tasks, and its pass^8 was below 25%. A model that wins one trial may fail the same task the next time it runs. Only a few boards publish pass^k or multi-trial intervals.

4. Search-time leakage

Agents that browse during evaluation (GAIA, BrowseComp-style tasks) can land on pages that already discuss the answers, including blog posts about the benchmark. Good harnesses block known answer pages. Not every leaderboard says whether its harness did.

⚠ The single most useful habit

Never compare two numbers from different boards, splits or harnesses. And if a benchmark’s top scores already beat its human baseline, stop using it to pick between frontier agents.

Building your own evals: when to stop trusting public benchmarks

For a production agent, stop relying on public leaderboards once you have about 50 real tasks from your own workload and a deterministic grader. The grader is the hard part. The agent cannot reliably grade its own success, and an LLM judge brings back the noise described above. Our AI agent testing checklist covers the test design. AI agent observability metrics covers what to monitor once the agent is live.

The simplest pattern that holds up in practice:

Python · eval_harness.py
import statistics, time
from your_agent import run_agent                 # your agent entry point
from your_graders import deterministic_check     # NOT an LLM judge

def evaluate(tasks: list[dict], k: int = 4) -> dict:
    runs = []
    for task in tasks:
        outcomes = []
        for trial in range(k):
            t0 = time.time()
            result = run_agent(task["prompt"], env=task["env"])
            ok = deterministic_check(result, task["target_state"])
            outcomes.append({"ok": ok, "tokens": result.tokens,
                             "seconds": time.time() - t0})
        runs.append({
            "task": task["id"],
            "pass@1": outcomes[0]["ok"],
            "pass^k": all(o["ok"] for o in outcomes),   # tau-bench style reliability
            "median_tokens": statistics.median(o["tokens"] for o in outcomes),
        })
    n = len(runs)
    return {
        "pass@1": sum(r["pass@1"] for r in runs) / n,
        "pass^k": sum(r["pass^k"] for r in runs) / n,
        "median_tokens": statistics.median(r["median_tokens"] for r in runs),
        "tasks": runs,
    }

Three things make this more honest than most leaderboards. The grader is deterministic, with no LLM judge. It reports pass@1 and pass^k side by side, so you can see the variance. And it tracks tokens per task, so you can rank agents by cost per success. Use our LLM inference cost comparison to turn tokens into dollars.

Personal note: the deterministic verifier behind DTF

The agent that helps draft DTF articles is graded by a benchmark-of-one: scripts/check_seo.py. It is a small Python script of a little over 200 lines. It checks about a dozen specific properties of each article: title and meta-description length, slug format, AIO box markup, FAQ JSON-LD shape, SVG metadata, the bibliography wrapper and the internal-link count. The agent writes the article, the script grades it, and the agent retries on any error. This refresh went through the same loop before it was published.

That is the same pattern as a serious agent benchmark, scaled down to one workload. There is no LLM judge: the grader is a function that returns pass or fail and has no opinions. Before building that script I sometimes trusted a model’s own “looks good”, and that let broken markup through. The lesson generalises: if your agent does real work, build a deterministic eval before you add features.

What benchmarks still miss

  • Very long horizons. METR’s suite tops out around 16 hours, and OSWorld 2.0 is the first mainstream suite where tasks routinely take more than an hour. Multi-day sessions with persistent memory remain largely unmeasured.
  • Cost-aware ranking. Terminal-Bench publishes cost and tokens, and HAL reports cost next to accuracy. Most boards still rank on accuracy alone.
  • Adversarial safety at task level. Prompt injection and malicious web content are documented risks (OWASP Top 10 for LLM Applications). Most capability leaderboards do not score them, although OSWorld 2.0 adds separate safety reports. See also our explainer on AI agent traps.

For context on the models behind these numbers, see ChatGPT vs Claude vs Gemini 2026 and Claude Opus 4.8 explained. For the wider cluster, go back to the AI agents hub.

FAQ: AI agent benchmarks 2026

What are the most important AI agent benchmarks in 2026?

For coding agents: SWE-bench Pro and Terminal-Bench 4.0. For computer use: OSWorld-Verified and OSWorld 2.0. For browser agents: WebArena and WebArena-Verified. For research and agentic search: GAIA and BrowseComp. For customer-service tool use: τ²-bench and τ³-bench. For long autonomous tasks: METR’s time horizon. As of 13 October 2026, GAIA and OSWorld-Verified top scores are above their human baselines, so the newer suites do more to separate frontier agents.

What is the top score on the OSWorld leaderboard in 2026?

On OSWorld-Verified, the top entry as of 13 October 2026 is Intelligence-Indeed Agent at 90.19% (25 July 2026). The best general-model entry is claude-fable-5 at 85.96% (1 August 2026). Both are above the 72.36% human baseline. On the harder OSWorld 2.0, the project page lists Claude Opus 4.8 as the top scorer, with 20.6% binary completion and 54.8% partial score.

Who leads the GAIA benchmark leaderboard in 2026?

On the official GAIA test-set leaderboard on Hugging Face, Shadow Harness V8.01 leads with 94.0%, submitted 12 October 2026. Ops-Agentic-Search-2.0 has 93.7%. On Princeton HAL’s validation-set board, the HAL Generalist Agent with Claude Sonnet 4.5 leads at 74.55%. The two boards use different question sets and different rules, so do not compare their numbers directly.

Which benchmark should I use for a browser agent in 2026?

Use WebArena-Verified, or its 258-task hard subset, for reproducible runs with deterministic grading. Use Online-Mind2Web to test on live websites, and BrowseComp if the agent’s job is finding hard-to-locate information. The original WebArena leaderboard is led by WebTactix with DeepSeek V3.2 at 74.3%, close to the 78.24% human baseline.

What are the best agentic search benchmarks in 2026?

BrowseComp (1,266 hard-to-find-fact questions) and GAIA are the most cited. GAIA’s test set is saturated at about 94%. BrowseComp has no independent public leaderboard, so compare vendor numbers only when they used the same tools and blocked answer pages.

Is SWE-bench Verified still a reliable benchmark?

Not for ranking frontier models. On 23 February 2026 OpenAI stopped reporting it. Its audit found flawed tests in 59.4% of the problems it reviewed, plus evidence that all tested frontier models had seen some tasks during training. OpenAI recommended SWE-bench Pro instead, then withdrew that advice in July 2026 after estimating that about 30% of Pro tasks are broken.

What is METR’s time horizon and how fast is it growing?

It is the length of task, in skilled-human time, that an AI agent completes with 50% success. METR’s latest reliable estimate is about 12 hours, for Claude Opus 4.6. Its doubling time since 2023 is about 129 days. METR treats measurements above 16 hours as unreliable and stopped updating the page on 8 September 2026.

Bibliography & sources (24)
  1. OpenAI. (2026, February 23). Why we no longer evaluate SWE-bench Verified. openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
  2. OpenAI. (2026, July 8). Separating signal from noise in coding evaluations. openai.com/index/separating-signal-from-noise-coding-evaluations
  3. Jimenez, C. E. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770
  4. SWE-bench. Leaderboards (Verified, bash-only), data read 13 Oct 2026. swebench.com
  5. Aleithan, R. et al. (2024). SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992
  6. Deng, X. et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941
  7. Scale AI. SWE-Bench Pro (Public) leaderboard, checked 13 Oct 2026. labs.scale.com/leaderboard/swe_bench_pro_public
  8. Merrill, M. A. et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv:2601.11868
  9. Terminal-Bench. Terminal-Bench 4.0 leaderboard (updated 6 Oct 2026) and Benchmarks release list. tbench.ai/leaderboard, tbench.ai/news/terminal-bench-4-0
  10. Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. arXiv:2404.07972
  11. XLang Lab. OSWorld-Verified results (osworld_verified_results.xlsx), read 13 Oct 2026. os-world.github.io
  12. XLang Lab. (2026). OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. arXiv:2606.29537; osworld-v2.xlang.ai
  13. Zhou, S. et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854; leaderboard via webarena.dev/og
  14. ServiceNow. WebArena-Verified. github.com/ServiceNow/webarena-verified
  15. Xue, T. et al. (2025). An Illusion of Progress? Assessing the Current State of Web Agents (Online-Mind2Web). arXiv:2504.01382
  16. Mialon, G. et al. (2023). GAIA: a benchmark for General AI Assistants. arXiv:2311.12983
  17. GAIA. Official leaderboard (results_public dataset, read 13 Oct 2026). huggingface.co/spaces/gaia-benchmark/leaderboard
  18. Princeton HAL. GAIA leaderboard (validation set; cost caveat dated 5 Oct 2026). hal.cs.princeton.edu/gaia
  19. Wei, J. et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516
  20. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045
  21. Barres, V. et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982; τ-Knowledge arXiv:2603.04370; τ-Voice arXiv:2603.13686; leaderboard taubench.com
  22. METR. (2025). Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499
  23. METR. Task-Completion Time Horizons of Frontier AI Models (last updated 8 Sep 2026; data benchmark_results_1_1.yaml) and Time Horizon 1.1 (29 Jan 2026). metr.org/time-horizons, metr.org/blog/2026-1-29-time-horizon-1-1
  24. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena; OWASP (2025). Top 10 for LLM Applications. arXiv:2306.05685, genai.owasp.org/llm-top-10
RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

- Advertisment -

Most Popular

Recent Comments