The fix shipped, and half the customers never saw it

A pricing display fix went live on a product page. We checked it the way anyone would check it, by requesting the live page and reading what came back. It was correct every single time we asked. It was also missing for about half of the people actually visiting that page, and had been since the moment it shipped.

What we found

The store was running an A/B test. Most A/B tools on Shopify do not have a theme of their own, so a test needs two: the live published theme as the control, and a duplicate of it as the variant. That duplicate is unpublished, and it has no connection to the repository the theme deploys from.

Which means the duplicate is a frozen snapshot of the theme as it stood on the day the test started. Deploys only ever write to the published theme. Nothing you ship reaches the clone for the entire life of the test, however long that is, and traffic is split roughly evenly between the two.

The reason our verification passed is worth its own paragraph, because it will pass for you too. The traffic split is decided by JavaScript in the browser. A command-line request runs no JavaScript, so it lands on the control theme every time. We had a run of successful checks and a page that was fine in every one of them, while the clone was serving byte-for-byte pre-fix code to everybody bucketed into the variant. To reach the clone at all you have to pin its theme explicitly and carry cookies through the redirect, and the test configuration itself is sitting inlined in the control page, listing each clone by id.

What we did

Listed the live tests and their clone themes, then took it to the client rather than deciding it. Patching a clone mid-test is not a neutral act: the clone carries the variant’s own markup, so it cannot simply be replaced with a copy of the corrected live theme without destroying the test, and changing it partway through changes the thing being measured. The honest options are to hand-copy the changed files across and accept the effect on the result, or pause the test and lose the elapsed time. That is a business call about which matters more, the defect or the experiment, and it is the client’s to make.

What to take from it

If a store runs experiments, its deploy checklist needs one more line: list the active tests before promoting anything. Every test in flight is another copy of the theme that quietly misses every change you make, and no warning appears anywhere in the process. Verification that runs outside a browser will not catch it, because it cannot see the half of the traffic that is affected.