Skip to main content

Supabase Evals: What Task-Specific Benchmarks Teach You About AI Coding Agents

By Upendra

Precision probe tip touching a circuit board trace in a dimly lit electronics lab

Supabase open-sourced their evals framework last week (supabase/evals, Apache-2.0), and I think it's the most useful thing published about AI coding agent evaluation in months. Not because of which model topped the leaderboard. Because of how they designed the measurement itself.

What They're Testing and Why It's Hard

Supabase built their evals around a three-axis grid: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, declarative-schema), and stages (build, deploy, investigate, resolve).

That last axis is where it gets real. "Build" is the easy part. Any capable coding agent can scaffold a schema. "Deploy" and "Investigate" are where agents start to diverge. "Resolve" is where you find out if an agent can fix a broken RLS policy without silently breaking three others it didn't know existed.

They're running Claude Code, OpenAI Codex, and OpenCode through these scenarios against a real Supabase backend, not exam questions with pre-known answers. That distinction is the whole point.

The Numbers That Matter

Opus 5 and Kimi K3 both hit 100% pass rates unaided in the Build stage. That's expected. They're the biggest frontier models available right now.

Here's what I found more interesting. Sonnet 5 scored 78% unaided, then jumped to 100% after Supabase loaded their context files into the agent's session. They call these "skills" in the framework. GPT-5.6 Sol went from 89% to 100% with skills loaded. GPT-5.4 mini went from 78% to 89%.

Different model sizes, different baselines, but every single one improved with context injection. The floor lifted across the board. That's a meaningful signal. Even the smallest tested model can clear most build-stage tasks if you give it the right context. You're not necessarily locked into running the biggest model for every Supabase operation in your agent loop.

The Context Injection Effect Is the Real Finding

Loading relevant documentation before the agent runs isn't a new idea. But seeing a 22-point jump for Sonnet 5 on a domain-specific eval is not a rounding error. That's the difference between "this agent sometimes works on our stack" and "this agent passes every build scenario we throw at it."

The thing most teams are missing is measurement. You probably inject some context into your agents already: docs, API references, a README. But do you know how much it actually helps? Supabase's framework makes that testable. Run the eval with skills, then without. Get the delta.

One finding I haven't seen discussed much: Codex read around 8 pages of documentation per evaluation scenario, while Claude Code averaged roughly 2. Neither is obviously better. More doc pages might mean better coverage, or it might mean the agent is fishing for context it should already have internalized. That kind of behavioral difference would never surface on SWE-bench or HumanEval. It only shows up when you test on your actual domain with your actual tooling.

Two Tracks, Two Purposes

Supabase splits their scenarios into two buckets. Published benchmark scenarios are fixed. They move scores on the public leaderboard and stay stable so comparisons stay valid. Regression scenarios refresh daily, cover known failure modes, and don't feed the public numbers. Those are purely for internal signal.

That split matters more than it looks. A public benchmark needs stability or the comparison becomes meaningless. An internal regression needs freshness or it stops catching real failures. Most teams I've talked to conflate these, and end up with a stale benchmark they trust too much and a regression suite they check too rarely.

How to Borrow This Design

If you're shipping something that relies on AI agents doing domain-specific work, this is the eval structure worth copying:

  • Pick the axes that describe your problem. For Supabase it's products x topics x stages. For your team it might be services x intent types x error conditions.
  • Separate your stable benchmark from your regression suite. Don't conflate them.
  • Test agents with and without your context files. Measure the actual lift.
  • Add at least one behavioral metric beyond pass/fail. Doc pages read per scenario, tool calls made, retry count. These reveal things that pass rates hide.

I haven't built something this structured for my own agent work yet. But looking at that 78%-to-100% gain on a model I'm already running in production, I'm going to. The repo is at github.com/supabase/evals and the public leaderboard lives at supabase.com/evals. Worth studying even if you're not a Supabase shop.

Labels:

Comments

Post a Comment

Popular posts from this blog

AngularJs call one method of controller in another controller .

By Upendra
I have seen many question about calling one method of one controller in another controller or extending scope of one controller in another controller.so here are the ways. if you want to call one controller into another or extending scope of controllers there are four methods available $rootScope.$emit() and $rootScope.$broadcast() If Second controller is child ,you can use Parent child communication . Use Services Kind of hack - with the help of angular.element() 1. $rootScope.$emit() and $rootScope.$broadcast() Controller and its scope can get destroyed, but the $rootScope remains across the application, that's why we are taking $rootScope because $rootScope is parent of all scopes . If you are performing communication from parent to child and even child wants to communicate with its siblings, you can use $broadcast If you are performing communication from child to parent ,no siblings invovled then you can use $rootScope.$emit HTML <body ng-app = ...
7 comments
Read more

250,000 AI Agent Instances Exposed on the Internet — Is Yours One of Them?

By Upendra
If You're Running OpenClaw, You May Want to Read This A public watchboard has surfaced listing over 250,000 OpenClaw instances that are directly reachable from the internet. Some of these instances have leaked credentials. Many are running on infrastructure already flagged for known CVEs and threat actor activity. This isn't theoretical. It's happening right now. You can check the exposure list yourself at openclaw.allegro.earth . Why This Is a Big Deal OpenClaw is a powerful AI agent framework. That power comes with serious responsibility. A typical OpenClaw deployment runs with: Personal API keys — OpenAI, Anthropic, Google, cloud provider credentials Broad system permissions — file access, shell execution, network requests Autonomous execution capabilities — the agent can act without human approval Complex codebases — large attack surfaces that haven't been fully audited When one of these instances is publicly reachable without authentication...
Post a Comment
Read more

Closures in javascript and how do they work ?

By Upendra
JavaScript Closures for Dummies  Closures Are Not Magic This page explains closures so that a programmer can understand them — using working JavaScript code. It is not for gurus or functional programmers. Closures are  not hard  to understand once the core concept is grokked. However, they are impossible to understand by reading any academic papers or academically oriented information about them! This article is intended for programmers with some programming experience in a mainstream language, and who can read the following JavaScript function: function sayHello ( name ) { var text = 'Hello ' + name ; var sayAlert = function () { alert ( text ); } sayAlert (); } An Example of a Closure Two one sentence summaries: a closure is the local variables for a function — kept alive  after  the function has returned, or a closure is a stack-frame which is  not deallocated  when the function returns (as if a 'stack-fr...
Post a Comment
Read more