ngLover

Agent demos are seductive. You prompt an agent to research a competitor, draft a brief, update a spreadsheet, and it does something plausible enough that everyone in the room nods. Then you ship it, and your users slowly stop trusting it, because it keeps getting 80% of the way through tasks and failing at the end in ways that are genuinely hard to diagnose. OSWorld 2.0, released in late June 2026 by the XLANG Research lab, is the benchmark that finally makes that failure mode measurable. The numbers are sobering: Claude Opus 4.8, the current leader, finishes just 20.6% of tasks end-to-end. GPT-5.5 plateaus at 13% regardless of whether you give it 150, 300, or 500 steps. What Makes OSWorld 2.0 Different The original OSWorld measured desktop computer-use on relatively short tasks. 2.0 extends it to 108 long-horizon workflows across seven professional domains: research, creative production, engineering, personal services, business and finance, administration and compliance, and healt...

For most of this year, if you asked me how to compare coding agents, I'd have pointed you at SWE-bench. The safer answer now is: don't. SWE-bench Verified died in February. OpenAI audited it and found frontier models could reproduce the original human-written patches verbatim, which meant scores reflected training contamination, not capability. They pulled their own numbers from Verified and recommended SWE-bench Pro instead. Then in July, OpenAI audited Pro and found roughly 30% of the 731 tasks are broken. Not "hard." Not "noisy." Broken. Their automated pipeline flagged 200 tasks (27.4%). Human reviewers tagged 249 (34.1%). OpenAI retracted their Pro recommendation and called on the broader evaluation community to start over. So the fallback for the fallback is gone. And models are still shipping press releases citing SWE-bench scores. How We Got Here SWE-bench Verified made sense when it launched. Real GitHub issues, real test suites, tasks that...

Supabase open-sourced their evals framework last week (supabase/evals, Apache-2.0), and I think it's the most useful thing published about AI coding agent evaluation in months. Not because of which model topped the leaderboard. Because of how they designed the measurement itself. What They're Testing and Why It's Hard Supabase built their evals around a three-axis grid: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, declarative-schema), and stages (build, deploy, investigate, resolve). That last axis is where it gets real. "Build" is the easy part. Any capable coding agent can scaffold a schema. "Deploy" and "Investigate" are where agents start to diverge. "Resolve" is where you find out if an agent can fix a broken RLS policy without silently breaking three others it didn't know existed. They're runni...

OpenAI previewed GPT-5.6 on June 26, 2026, in three variants: Sol, Terra, and Luna. Access is currently limited to roughly 20 US government-approved partner organizations, which means most teams cannot run their own tests yet. But there is still a lot worth digging into: a genuinely interesting architecture change with "ultra" mode, and a finding from METR that fundamentally changes how you should read any Sol benchmark score you encounter. Sol, Terra, and Luna: The Three-Tier Model The naming is celestial but the logic is familiar. OpenAI has codified what we have all been doing informally: routing different tasks to different models based on cost and capability. Sol is the flagship. It targets hard problems in coding ( Terminal-Bench 2.1 state of the art at 88.8%), biology (GeneBench v1), and cybersecurity (ExploitBench). Pricing is $5 per million input tokens, $30 output. Terra is the balanced tier, aimed at high-volume business tasks, customer support, document anal...