The Best AI Agent Finished Just 12% of Real Expert Work, and in Finance and Legal Every Model Scored Zero
Most benchmarks give an exam. This one gives a job.
An exam hands the model everything it needs, asks one question, and grades one answer. A job does none of that. The files sit somewhere in a few thousand documents, half the context has to be gathered before the work can start, the tools are real, and a result that is 90 percent finished can still be worth nothing because one rule was broken or the deliverable is wrong.
On November 2, SemiAnalysis, the chip and infrastructure research firm, launched GDPx, a benchmark built on that second idea: 206 tasks across 11 domains, built with 14 specialist data partners, each running up to eight hours by default. The headline score is pass at first attempt, averaged over three attempts, with every domain weighted equally. The best score in the world right now is 12 percent.
Why these 11 domains, and not the whole economy
SemiAnalysis chose the domains on purpose. The 11 are chip design, cybersecurity, biology, finance, legal, quantitative trading, recursive self-improvement, hardware design, materials science, alignment, and software engineering. Government, real estate, and insurance were left out deliberately.
The reasoning, in the launch post by Max Kan, Andrew Megalaa, and Dylan Patel: the first trillion dollars of AI revenue will be concentrated in sectors that can deploy frontier intelligence at high return on investment, not spread evenly across the economy. A benchmark that mirrors the whole economy therefore measures the wrong thing. GDPx measures the work where frontier models are expected to earn first. It is one firm’s bet, but a more honest design than pretending a general score exists: 11 hard rooms, and a model can be strong in one and absent in another.
What a GDPx task actually looks like
The clearest contrast is with OpenAI’s GDPval, released on September 25, 2025, which GDPx is built to succeed. SemiAnalysis argues that GDPval tasks spoon-feed the model: the prompt carries the context and the model one-shots a document from a short description. A real finance manager would first spend hours gathering context, and that gathering is a large part of what makes the work hard.
A GDPx finance task, built by partner Living Context, starts from a real transaction. In the published example, a private equity fund buys about 62 percent of a snack company for $880 million plus an earnout, and the environment holds roughly 4,000 anonymized files: a virtual data room plus deal folders for the buyer, the banker, and the accounting advisor. The agent has to navigate a data room of about 2,000 files to find what an associate on the real deal actually needed.
The other domains follow the same pattern: cybersecurity tasks, from Arithmetic, are based on real zero-days with multiple live services running; materials science tasks use simulated laboratories with battery cyclers, a catalyst reactor, and an electron microscope; hardware design tasks include live Siemens NX sessions, with outputs tested on real AWS FPGA instances. That richness is why the scores are low, and why they are interesting.
The leaderboard: 12, 10.9, 9.6, and then a long fall
Thirteen models were ranked at launch. The top of the board:
- Claude Opus 5.5 (Anthropic): 12.0 percent, at a reported $51.41 per task.
- Gemini 4 Carbon (Google): 10.9 percent, at $16.29 per task. Carbon is not yet generally available.
- GPT-6 Astra (OpenAI): 9.6 percent, at $53.99 per task.
- Muse Wonder Full (Meta): 9.0 percent.
Below that, the field thins out fast: Muse Wonder at 6.8 percent, DeepSeek 4.1 Flash at 5.4 percent, GPT-6 Luna at 5.1 percent, Grok 4.7 at 3.9 percent, and GLM 5.3, the strongest open-weight entry, at 2.7 percent. On cost against performance, Carbon is the one to watch: near the top of the board at roughly a third of the leader’s cost per task.
The averages hide the real story, which is spikiness:
- Opus 5.5 wins quantitative trading, hardware design, and chip design, and is the best all-rounder.
- Gemini 4 Carbon wins biology and edges out software engineering.
- GPT-6 Astra wins recursive self-improvement and leads materials science.
- Muse Wonder Full is, in SemiAnalysis’s words, the best available cybersecurity model by a meaningful margin.
A headline number tells a buyer almost none of that. The domain row for your work is the leaderboard that matters.
Zero is a finding, not just a failure
The starkest result sits in two domains. On finance and legal, SemiAnalysis says its tasks are so challenging that basically every model scores exactly zero. Both are graded pass or fail, with no partial credit, which is how the work itself behaves: a deal analysis built on the wrong file, or a legal answer that misses the controlling clause, is not 80 percent useful. It is wrong.
Treat the zero carefully. It does not mean models cannot draft a contract summary or reconcile a spreadsheet, because GDPx does not test routine office work. It tests frontier expert work inside environments built to resist shortcuts. What it establishes is a ceiling: on long, high-stakes, context-heavy finance and legal tasks, no current agent can yet be left to finish alone. Anyone selling that outcome today is selling past the evidence.
Part of the score is willingness, not ability
SemiAnalysis is unusually open about a confound inside its own numbers. Opus 5.5 and GPT-6 Astra, it says, have sandbagged scores because their safety classifiers refuse or cut short work in cybersecurity, biology, and alignment. Exclude those three domains and the ranking reshuffles. Carbon’s biology win, the firm suggests, likely owes something to weaker safety classifiers.
The firm calls this underrated, and it is right: for security or biology work, the classifier is part of the product. Two models with similar raw ability can behave like different tools because one will attempt the task and the other will decline it. A blended score hides that decision. GDPx at least names it.
The hard part is a test a model can fail honestly
A benchmark this hard invites a suspicion: maybe the tasks are just broken. Labs, SemiAnalysis notes, will pay $5,000 or more for one high-quality task, because a task can fail a model unfairly, for example by grading against one of two valid solutions, and fixing every unfair task usually makes a benchmark easy enough to saturate. Its answer was process: every partner had to pass automated quality checks for consistency, completeness, cheating paths, and unfair verifiers on the first try, with only minor mistakes allowed. The same process, it says, flags nearly 50 percent of Terminal-Bench 4.0 tasks as broken. That claim is not independently tested, but the direction is correct: the scarce skill is now building a test a frontier model can fail honestly.
Read the 12 percent with care
Four limits belong in any honest account:
- One firm, one rubric. GDPx is a SemiAnalysis product, built with vendor partners, scored by its own methods, and not yet independently replicated.
- Harnesses differ. Models ran in their makers’ own harnesses where one exists, Claude Code for Opus, Codex for Astra, Antigravity for Carbon, OpenCode otherwise. The score measures a model plus its harness, not a model alone.
- The domains are a bet. Eleven high-value fields say little about the routine knowledge work most companies actually automate.
- Low is the design goal. A 12 percent top score measures how far agents are from expert autonomy, not whether they are useful on narrower work today.
SemiAnalysis says GDPx will refresh every three to six months. That refresh is the number to watch: a benchmark starting at 12 percent gives the industry something saturated leaderboards stopped giving it, a slope.
What to do with this, this week
- Build your own small GDPx. Take ten real jobs from your team, with real files and a pass or fail bar a manager would sign, and grade each model on those. It will predict your results better than any public board.
- Buy by domain, not by average. The winning model differs by row. The blended score is the least useful number on the page.
- Ask about refusals before you buy. For sensitive work, ask vendors how often their classifiers decline legitimate tasks, and test it on your own cases.
- Budget cost per passed task. Carbon at $16.29 per task against about $51 to $54 for the other two leaders shows why price per token is the wrong unit. Price per finished, accepted job is the unit your budget feels.
- Keep a human finisher on finance and legal work. Until a model scores above zero on independent expert tasks in those fields, agents should gather, draft, and check, with a person closing.
The bottom line
GDPx does not show that AI agents do not work. It shows where they do not work yet: long, expert, context-heavy jobs where the information is not in the prompt, the tools are real, and nearly right is wrong. The best agent finished about one task in eight, at roughly $51 a task, and in finance and legal the entire field finished none. That is a more useful truth than another saturated leaderboard at 95 percent, because it tells builders where the headroom is and buyers where to keep a person in charge. The next refresh will show how fast that 12 percent moves. That movement, not any single launch, is the progress chart worth following.