Agent demos are seductive. You prompt an agent to research a competitor, draft a brief, update a spreadsheet, and it does something plausible enough that everyone in the room nods. Then you ship it, and your users slowly stop trusting it, because it keeps getting 80% of the way through tasks and failing at the end in ways that are genuinely hard to diagnose.
OSWorld 2.0, released in late June 2026 by the XLANG Research lab, is the benchmark that finally makes that failure mode measurable. The numbers are sobering: Claude Opus 4.8, the current leader, finishes just 20.6% of tasks end-to-end. GPT-5.5 plateaus at 13% regardless of whether you give it 150, 300, or 500 steps.
What Makes OSWorld 2.0 Different
The original OSWorld measured desktop computer-use on relatively short tasks. 2.0 extends it to 108 long-horizon workflows across seven professional domains: research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare. A skilled human takes a median of 1.6 hours per task. That is the baseline you are comparing against.
The benchmark runs against 31 self-hosted websites alongside real desktop applications, with authentic input files rather than contrived scenarios. Tasks include things like processing travel reimbursements end-to-end, completing visa application forms, formatting a presentation from a raw spec, and creating CAD models from a brief. Nothing here is a toy problem.
What is genuinely new is the scoring approach. Instead of binary pass/fail, OSWorld 2.0 grades each task against many weighted checkpoints distributed through the workflow. You get a partial score in addition to a binary completion rate. That distinction turns out to matter enormously.
The Gap That Should Worry You
Here is the number I keep coming back to: Claude Opus 4.8 scores 20.6% on binary completion but 54.8% on partial score. Partial scores across all evaluated systems cluster in the 20-55% range, even while binary completions sit near zero for most of them.
That gap is telling you something specific. Frontier agents make real progress on long-horizon tasks. They do not just stall immediately. They navigate complex multi-step workflows, maintain context across hundreds of tool calls, and complete meaningful chunks of work. But they fail to close the loop.
The finish line is where things fall apart.
I have not traced every failure in the OSWorld 2.0 dataset, but the benchmark authors point to two recurring failure modes: agents losing track of evolving context across many steps, and agents executing from stale plans, repeating work they have already done. Claude Opus 4.8 requires an average of 318 tool calls per task to reach that 20.6%. Compare that to roughly 30 tool calls per task in the original OSWorld. At ten times the call volume, small context-tracking failures have far more opportunity to compound.
The Step Budget Plateau
The GPT-5.5 finding is worth dwelling on. The model plateaus at 13% at 150 steps. At 300 steps, still 13%. At 500 steps, still 13%.
If you are building an agent and tuning your max-iterations parameter hoping it will push through stuck points, this is evidence that you might be optimizing the wrong variable. More steps do not fix context drift or stale plans. They run up the token bill and occasionally produce a different wrong outcome. The actual failure is usually in how the agent handles state across a long trajectory. Giving it more chances to try the same broken approach does not help.
What This Means for Your Own Evals
Most teams running production agents measure binary task completion: did the agent finish the objective or not? OSWorld 2.0 makes the case that this misses too much signal, especially at task lengths that real users actually care about.
A few things I would take from this:
Add checkpoint scoring. Define the meaningful milestones in your task and score against each one. If your agent consistently reaches step 8 of 10 and fails on the last two, that is a completely different problem than failing at step 3. Binary pass/fail hides the difference.
Test at realistic task length. Short demo tasks do not reproduce the failure modes that show up in long-horizon work. Context drift and stale-plan failures emerge at scale. If your eval suite only covers 10-15 step tasks, you do not know how your agent performs on the work your users actually care about.
Track where in the trajectory failures happen. Instrument your agent traces so you can see failure points distributed across a task timeline. If failures cluster late in long tasks, that points to context management. If they are uniformly distributed, that is a different problem entirely.
More steps is not a free fix. If your agent stalls, adding max iterations might not help. Diagnose whether the failure is a dead loop, a context problem, or a tool error before you touch the step budget. Budget the engineering effort on diagnostic infrastructure before you budget it on more compute.
OSWorld 2.0 is live at osworld-v2.xlang.ai with a full public leaderboard. If you are building serious computer-use or long-horizon automation agents, it is worth running your own system against it. The 20.6% ceiling and the gap to 54.8% partial are hard numbers that should shape what you prioritize next in your stack.
Comments
Post a Comment