Running Multiple A/B Tests at Once: Why Interaction Effects Are Mostly a Non-Problem

Most experimentation teams I talk to are more worried about concurrent tests than they should be. They treat isolation as the default, queue up experiments sequentially, and then wonder why velocity is low. The interaction effect problem is real in theory. In practice, it barely shows up.

What Interaction Effects Are (and Why People Panic)

When two A/B tests run simultaneously on the same user population, there's a risk that they interfere with each other. Test A changes the header. Test B changes the checkout button. A user sees both. The concern is that the combined effect is different from what you'd expect by adding the individual effects together. In statistics, this is called a qualitative interaction: the direction of one test's result reverses depending on the state of another.

That's the scary version. What actually happens is almost never that dramatic.

What the Data Actually Shows

Microsoft's experimentation team ran a rigorous analysis across four major product groups on Bing and other properties. Three of the four showed zero statistically significant interactions between concurrent experiments. The fourth detected an abnormally small p-value in just 0.002% of cases. One in fifty thousand. And even where interactions were detected, the treatment effects moved in the same direction; they weren't reversals.

The team's conclusion: "the scenario with small cross-A/B test treatment effects is interesting in an academic sense, but not typically a problem for decision-making."

That last phrase is doing a lot of work. Most A/B tests end in a binary decision: ship or don't ship. A small interaction that nudges your effect estimate by 2% doesn't change that call. You'd need a reversal to actually be misled, and reversals are rare.

What High-Volume Teams Actually Do

Booking.com runs thousands of concurrent experiments at any given time. Their philosophy is that interactions are more a feature than a bug. Seeing how two changes behave in combination tells you something real about the product.

Facebook operates on the same principle. The default is to let experiments run independently. Multivariate or mutually exclusive setups are reserved for specific situations, not the baseline.

The teams running the most tests per quarter tend to be the least worried about interaction effects. That's not negligence. It's calibrated risk.

When Isolation Actually Makes Sense

There are three cases where I'd recommend making experiments mutually exclusive or running them sequentially.

Technical conflicts. If two tests modify the same DOM element, or both control the same feature flag, they can't coexist cleanly. This is a correctness problem, not a statistics problem. Fix it at the platform level, not by serializing your roadmap.

Behavioral collision. If you're running a major navigation redesign at the same time as a completely different checkout flow, you're changing so much of the experience that separating them is just good sense. The question to ask is whether two tests are competing for the same behavioral trigger. A pricing test and a hero image test aren't. A pricing test and a "free shipping threshold" banner probably are.

Precise effect quantification. If you're building a forecasting model that depends on an accurate uplift estimate, or running a financial analysis where the absolute effect size matters, isolation helps you measure cleanly. Most product teams don't need this level of precision for ship/don't-ship calls.

Everything outside these three cases is usually fine to overlap.

The Real Cost of Over-Isolating

Here's what happens when teams default to isolation: experiment queues grow, tests run longer because traffic is split more thinly, and the program slows down. Slower programs generate fewer learnings. Fewer learnings mean the program loses credibility with stakeholders who want results.

I've seen teams go from running eight concurrent experiments to two because someone got spooked about interaction effects after reading a statistics paper. Their win rate didn't improve. Their throughput collapsed.

The fear is usually disproportionate to the actual risk.

A Simple Decision Rule Before You Isolate

Before isolating two tests, ask: do these experiments target the same user action or the same piece of UI? If no, overlap them by default. If yes, check whether they physically conflict first. Only then consider isolation, and even then, post-hoc segmentation by variant exposure can often answer the interaction question without you having to queue one test behind the other.

This isn't a call to throw caution out entirely. A quick design review before launching concurrent tests is worth it. But design review is cheap. Slowing your program down because of a problem that shows up in 0.002% of cases is expensive.

Run the tests. Check for obvious conflicts. Ship the winners. The interactions you're worried about are mostly not there.