A shared function in the site’s root layout turned an A/B-test exposure counter into a page-view counter, inflating the sample size and making the conversion rates meaningless. The test, a 50/50 split of the homepage, reported 178 hits for one variant and 57 for the other—far from the expected even split.
How the error slipped past the randomizer
The developer first verified the randomizer that assigns visitors to a variant. He read the middleware, inspected the cookie logic and ran a script that called the assignment function 10,000 times; it produced a perfect 50/50 split. The randomizer itself worked; the problem was how exposure was recorded.
A single function performed two jobs:
- Set the variant – runs on every page load to keep the visitor’s experience consistent.
- Record the exposure event – should fire only once per visitor, the moment the variant first appears.
The function lived in the root layout, a component rendered on every navigation. Because the exposure-recording code executed each time the layout rendered, every page view counted as a new exposure. The two homepage variants used slightly different layout trees, so their page-view rates diverged, creating the illusion of a broken randomizer.
Why the miscount mattered
Conversion numbers—click-throughs, sign-ups, purchases—were logged correctly. But the denominator (the number of exposures) was wrong. Calculated conversion rates appeared far lower than reality, and any decisions based on those rates were unreliable.
The test ran for two weeks before the discrepancy surfaced, forcing the team to discard the entire data set.
The fix
The solution was straightforward: split the responsibilities into separate functions. The exposure-recording code now checks whether the visitor has already been counted, firing only once per user. The variant-setting code stays where it is, continuing to run on every navigation.
Three take-aways for anyone running experiments
- Separate state-setting from one-off events. A function that both assigns a variant and logs an exposure will clash because the former repeats while the latter must not.
- Avoid one-shot logic in a root layout. Anything placed in a component that renders on every page load will execute repeatedly, turning “once per visitor” into “once per page view.”
- When the observed split contradicts the randomizer, audit the counter first. Developers often test the randomizer’s fairness but rarely verify that the counting mechanism is accurate.
What to watch for next
Any experiment that relies on a single counter for exposure should be audited for where that counter lives in the component tree. If the counter sits in a global layout, add a check that ties the event to a persistent identifier—such as a cookie or local-storage flag. Teams should also build a sanity check into their dashboards: if the observed variant distribution deviates beyond a small statistical margin, flag the test for a counter audit before assuming the randomizer is broken.
In short, a well-functioning randomizer is useless without a trustworthy exposure count. Splitting responsibilities and placing one-off events outside always-rendered components keeps A/B tests honest and saves weeks of wasted analysis.