Skip to main content

Posts

Running Multiple A/B Tests at Once: Why Interaction Effects Are Mostly a Non-Problem

Most experimentation teams I talk to are more worried about concurrent tests than they should be. They treat isolation as the default, queue up experiments sequentially, and then wonder why velocity is low. The interaction effect problem is real in theory. In practice, it barely shows up. What Interaction Effects Are (and Why People Panic) When two A/B tests run simultaneously on the same user population, there's a risk that they interfere with each other. Test A changes the header. Test B changes the checkout button. A user sees both. The concern is that the combined effect is different from what you'd expect by adding the individual effects together. In statistics, this is called a qualitative interaction: the direction of one test's result reverses depending on the state of another. That's the scary version. What actually happens is almost never that dramatic. What the Data Actually Shows Microsoft's experimentation team ran a rigorous analysis across four...
Recent posts

Claude Mythos Broke a NIST Post-Quantum Candidate in 60 Hours

On July 28, Anthropic published research showing that Claude Mythos Preview, their unreleased frontier model, found two previously unknown cryptographic attacks in roughly 60 hours of autonomous work. One of them helped take down a NIST post-quantum signature candidate that had survived two rounds of expert human review spanning two years. That is not a prompt engineering trick. That is AI doing original mathematics. To be clear about what this is and is not: the production systems you depend on are not at immediate risk. Full AES-256 is fine. The attacks target weakened or candidate schemes. But the more important story is how fast this happened, and what it tells us about where AI fits in the research pipeline going forward. What Claude Mythos Actually Did Claude Mythos Preview ran for approximately 60 hours on cryptanalysis work targeting two algorithms: HAWK, a lattice-based post-quantum signature scheme that was in NIST's additional standardization process, and a weakened...

Google's Gemini Managed Agents Just Got Cron Triggers, Budget Caps, and Hooks

Google pushed an update to Gemini API Managed Agents on July 28 that I've been waiting for. It's not a new model. It's the production scaffolding that makes running agents as background workers actually viable: budget caps, cron triggers, environment hooks, and a proper API to manage the sandboxes. None of this is conceptually new, but it fills the gaps that made me nervous about committing to the platform for anything beyond a demo. What They Shipped Six things landed in this release. Gemini 3.6 Flash is now the default model for managed agents, with no code changes required. But the parts that matter operationally are the new production primitives. Cron triggers. You bind an agent, a prompt, an environment, and a cron expression into a single persistent resource. The trigger fires on schedule without you doing anything. Each run reuses the same sandbox, so files written in one execution are visible to the next. If a run fails five times in a row, the trigger pauses i...

Six Professional Creative Tools Got MCP Servers at SIGGRAPH 2026. Here's What That Actually Means.

Last week at SIGGRAPH 2026 in Los Angeles, six major professional creative applications announced Model Context Protocol server integrations at the same conference. Blender, Unreal Engine, Adobe Creative Cloud, SideFX Houdini 22, Boris FX Silhouette, and Foundry Griptape. Not demos or research previews. Announced and demonstrated, shipping or in active preview. That is the story. The protocol that started in AI coding assistants just crossed into the professional DCC (digital content creation) ecosystem all at once. If you have been building with MCP on the AI tooling side, this should get your attention. Not because Blender and Houdini suddenly matter to your agent pipeline, but because of what it means when a standard connection layer starts showing up in professional software categories at scale. What Each Integration Actually Does Let me be concrete, because the announcement headline does not tell you much on its own. Unreal Engine now exposes editor capabilities through an M...

Claude Opus 5 Lands, and It Outperforms Fable 5 Where It Counts

Anthropic shipped Claude Opus 5 on July 24. Four days later I'm still sitting with the benchmark numbers, because they tell a story I didn't expect: on the metrics that actually matter for builder work, Opus 5 outperforms Fable 5 at exactly half the token price. That's not "close enough." That's a genuine inversion. What the Numbers Actually Say Fable 5 has been Anthropic's flagship since June. It costs $10 per million input tokens, $50 per million output tokens. Opus 5 costs $5/$25 per million tokens, which is identical to Opus 4.8. Same price point, better performance on the key evals. On Frontier-Bench v0.1, the agentic coding benchmark I weight most heavily right now, Opus 5 scores 43.3%. Fable 5 scores 33.7%. On SWE-bench Pro it posts 79.2%. On ARC-AGI-3, which tests novel problem-solving rather than memorized patterns, Opus 5 scores 30.2%. The next-best model scored 7.8%. Fable 5 wasn't even tested on it. Artificial Analysis pegged the weig...

Gemini 3.6 Flash Is Out: What the Token Efficiency Gains Actually Mean for Agent Loops

Google dropped three new models into its Gemini Flash tier on July 21. Most coverage fixated on the naming confusion (what is a 3.6 Flash versus a 3.5 Flash-Lite anyway?) and on the Gemini 4 name-drop buried in the announcement. But the story that actually matters for teams running production agents is more concrete: the economics of calling Flash just got meaningfully better, and in a way that compounds harder than the headline numbers suggest. What actually changed with 3.6 Flash Gemini 3.6 Flash replaces 3.5 Flash as the default workhorse in the family. Same general job: coding, knowledge work, multimodal tasks at production scale. Two things changed: it scores higher on every major benchmark, and it uses fewer tokens to get the same work done. On DeepSWE , 3.6 Flash scores 49% versus 37% for 3.5 Flash. MLE-Bench: 63.9% versus 49.7%. OSWorld-Verified (computer use): 83.0% versus 78.4%. For a Flash-tier model to widen the gap that much while staying in the same cost bracket is a ...

Kimi K3: 2.8 Trillion Open Parameters and What the Weight Release Actually Changes

Moonshot AI released Kimi K3 on July 16. It's a 2.8-trillion-parameter mixture-of-experts model with native image input, a 1-million-token context window, and benchmark numbers that sit just below Claude Fable 5 and GPT-5.6 Sol on real-world agentic tasks. Open weights land July 27. That combination is worth paying attention to. What the Architecture Actually Is Kimi K3 is a sparse MoE. Of its 896 total experts, 16 are active per token, routed through what Moonshot calls the Stable LatentMoE framework. That sparsity is what makes 2.8T parameters economically viable at inference: you're not running all 2.8T on every forward pass. Two architectural changes distinguish it from K2. Kimi Delta Attention (KDA) replaces standard full attention with a hybrid linear attention mechanism designed for long-context at scale. Attention Residuals swap the standard residual connection for a depth-spanning retrieval mechanism that the company says delivers consistent scaling gains. Together...