The AI industry has become exceptionally good at measuring language models.
We have benchmarks for reasoning, mathematics, coding, knowledge, multimodality, and almost every other dimension of model capability imaginable.
But an AI agent is not simply an LLM.
An agent is a system.
The language model is its reasoning engine, but surrounding that engine may be planning logic, memory, tools, retrieval systems, browsers, terminals, APIs, permission structures, retry mechanisms, evaluators, context-management systems, and rules governing when the agent can act.
That distinction is becoming increasingly important.
If we want to understand whether AI agents can actually perform useful work, asking “Which LLM is smartest?” is rapidly becoming the wrong question.
The better question is:
What happens when this particular intelligence is placed inside this particular agent architecture and asked to perform this particular kind of work?
That is a fundamentally different measurement problem.
The Model-Agent Equation
Research increasingly demonstrates how substantial the interaction can be.
A controlled 2026 study running five models through three different GAIA agent scaffolds found that changing the scaffold alone moved measured accuracy by as much as 28 percentage points for the same model. The researchers also found that stronger models were not necessarily less sensitive to agent architecture.
Earlier software-engineering evaluations produced similar signals. OpenAI tested GPT-4o with multiple agent scaffolds on SWE-bench Verified and reported substantially different outcomes depending on how the model was wrapped and allowed to interact with the development environment.
This suggests a simple relationship:
Agent Performance ≠ Model Performance
A more realistic approximation is:
Performance = Model × Agent × Tools × Environment × Task
And the multiplication sign matters.
A weakness in any component can degrade the entire system.
A Great Agent With the Wrong LLM
Imagine an exceptionally engineered agent.
It has excellent memory management. Its tools are reliable. Permissions are properly bounded. It knows when to retry. It verifies its work. Its software architecture is elegant.
Now connect it to an LLM that consistently makes weak decisions.
The agent may become an extraordinarily efficient mechanism for executing bad judgment.
Poor reasoning can cause the system to select the wrong tool, misunderstand intermediate results, prematurely conclude that a task succeeded, repeatedly pursue an incorrect strategy, or invent explanations for failures rather than recognizing them.
AgentBench researchers identified long-term reasoning, decision-making, and instruction following as major obstacles to reliable agent behavior.
The agent framework can compensate for some weaknesses through verification, constraints and recovery loops.
But scaffolding cannot manufacture intelligence indefinitely.
Eventually the model has to make a decision.
A Great LLM Inside a Bad Agent
The reverse is equally important.
Put an extremely capable frontier model inside an agent with poor context management, badly designed tools, weak permissions, no recovery logic and ambiguous objectives, and that powerful model can perform surprisingly badly.
The intelligence may be there.
The system simply fails to extract it.
Anthropic recently demonstrated another complication: even the infrastructure underneath an agent benchmark can materially affect results. On Terminal-Bench 2.0, changes in available computational resources alone produced a six-percentage-point performance difference, large enough to exceed many leaderboard gaps between leading models.
That means we are not merely benchmarking models.
We are benchmarking systems operating inside environments.
And that requires a different standard.
The Agent Performance Index
I believe agents eventually need something resembling a performance record.
Not another single benchmark.
A multidimensional Agent Performance Index — API — built from a standardized battery of tests performed repeatedly across multiple models.
A useful first version could score six dimensions:
Task Completion — 35%
Did the agent actually accomplish the objective?
Tests should include coding, web interaction, research, data manipulation, tool execution and computer-use tasks modeled after systems such as SWE-bench, WebArena, GAIA and OSWorld.
Reliability — 20%
Can it do it repeatedly?
An agent that succeeds eight times out of ten is fundamentally different from one that succeeds once and fails nine times.
τ-bench helped introduce this issue explicitly through consistency-oriented measurements rather than relying solely on best-case success.
Error Recovery — 15%
What happens after something goes wrong?
Deliberately introduce broken APIs, misleading information, unavailable tools and failed actions.
Does the agent recognize the failure, re-plan and recover—or confidently continue down the wrong path?
Policy and Boundary Compliance — 10%
Does the agent remain inside its authorized operating boundaries?
This matters enormously as agents gain permission to modify files, interact with databases, communicate externally and transact with other systems.
Efficiency — 10%
How much does success cost?
Measure tokens, tool calls, compute, elapsed time and unnecessary actions.
Two agents achieving identical outcomes may have radically different economic value.
Generalization — 10%
Can the agent handle unfamiliar variations of the problem rather than memorizing benchmark patterns?
This category should rely heavily on private and dynamically generated tests to reduce contamination.
The result becomes:
Agent Performance Index = Capability + Reliability + Recovery + Control + Efficiency + Generalization
But I would add one more mechanism:
A Failure Severity Multiplier
Not every failure is equal.
Failing to locate a webpage is inconvenient.
Deleting a production database is catastrophic.
An agent that succeeds 95% of the time but causes unacceptable damage during the other 5% should not receive a 95.
Its score should collapse.
The Experiment That Matters
The most revealing evaluation may actually be a matrix.
Take ten LLMs ranging from small open models to frontier systems.
Connect each one to exactly the same agent.
Run every combination through the identical test battery multiple times.
Then reverse the experiment.
Take one model and place it inside several different agent architectures.
Suddenly three numbers become visible:
Model Contribution
Agent Contribution
Model-Agent Interaction
That third number may eventually prove the most interesting.
Certain models may simply work better with certain agent structures.
The highest-performing LLM may not produce the highest-performing agent.
The cheapest model may become economically superior when coupled with an exceptionally disciplined scaffold.
This could fundamentally change how organizations select AI systems.
What We Still Don't Know
The positive potential is substantial.
Better measurement could produce safer agents, cheaper deployments, more efficient architectures and competition based on observable outcomes rather than marketing claims.
Specialized agents could even make smaller models economically competitive with giant frontier systems.
But there is another side we should be careful not to dismiss.
We genuinely do not know all of the negative consequences that emerge when increasingly capable models operate through increasingly autonomous software.
Long-horizon failures may compound in ways short benchmarks never expose.
Multiple individually reasonable decisions may combine into disastrous outcomes.
Agents may learn to exploit weaknesses in benchmarks rather than become genuinely better at accomplishing useful work.
Humans supervising high-performing agents may gradually develop automation bias and stop questioning outputs.
Organizations could unknowingly create correlated failure by deploying thousands of agents based on the same underlying models.
And completely new failure modes may emerge from interactions between models, agents, tools and other agents that researchers simply have not encountered yet.
METR makes an especially important observation here: benchmark tasks are generally cleaner and better specified than economically valuable human work, and agents perform worse as tasks become messier.
Meanwhile, Stanford's 2026 AI Index reports remarkable progress—OSWorld agent performance rose from roughly 12% to 66.3%—while still noting that agents fail about one-third of attempts even on structured evaluations.
That gap matters.
From Intelligence Claims to Performance Records
There is another warning hiding inside today's benchmark ecosystem.
OpenAI stopped recommending SWE-bench Verified for measuring frontier coding capability after finding serious test-quality and contamination problems, including material issues in 59.4% of a heavily audited subset of difficult tasks.
Benchmarks themselves must therefore evolve.
What the industry ultimately needs is not one permanent test.
It needs continuous competition against changing tests.
Unknown tasks.
Repeated trials.
Different environments.
Different models.
Different agents.
Observable outcomes.
Measured costs.
Recorded failures.
Agents should accumulate performance histories the same way athletes, financial strategies, machines and human professionals accumulate records.
Because as AI moves from answering questions to performing work, capability claims are no longer enough.
AI agents need performance records.
And the systems capable of measuring those records may become as important as the agents themselves.