If you run more than a dozen A/B tests a year, test duration is probably the biggest drag on your experimentation velocity. Most teams try to fix it by bumping traffic allocation, cutting the number of variants, or calling the test early when it "looks significant." All three are the wrong answer. There's a better one that most experimentation teams outside big tech still aren't using: CUPED. Most of Your Test Variance Isn't From Your Test Here's the thing people miss. When you measure a metric like revenue per visitor or checkout conversion rate in an A/B test, a lot of the variation you see across users isn't from your treatment. It's pre-existing user heterogeneity. Power users who've been converting for months. Seasonal shoppers. First-timers who bounce regardless of what you show them. That variation was there before your test started. It has nothing to do with what you're testing. But it inflates your standard error and forces you to r...
Anthropic shipped Claude Sonnet 5 on June 30, positioning it as the go-to model for agentic work at a price that doesn't require a flagship budget. At $3 per million input tokens and $15 per million output tokens (after introductory pricing ends August 31), it's priced at roughly 60% of what Opus 4.8 costs. That sounds like an easy call. It's more complicated. Where Sonnet 5 Actually Earns the "Agentic" Label The benchmark numbers that matter for builders aren't the ones that get the most press. On SWE-bench Pro, Sonnet 5 scores 63.2% compared to Opus 4.8's 69.2%. That 6-point gap is real. For a coding agent doing open-ended software engineering, it matters. But Terminal-Bench 2.1 tells a different story. Sonnet 5 scores 80.4%. Opus 4.8 scores 74.6%. That's the first time a mid-tier Sonnet has beaten its flagship sibling on a major coding benchmark, and the margin isn't narrow. For agents that work in the terminal, run shell pipelines, orchest...