Infrastructure · Growth
Experimentation infrastructure
The problem
Client-side A/B tests flicker, which is a poor experience, and it also biases the result. The hosted tools that do assignment at the edge get expensive fairly quickly at our traffic, so we rationed tests.
What I built
Variant assignment runs on a Cloudflare Worker, at the edge, so there is no flicker. There is conversion logging and a view for results. It runs at about USD 5/month at 200K pageviews a day.
What changed
At that price nobody has to justify running a test. As a result, we run more of them, including ones that would not have been worth a quarterly planning conversation.
What I'd do differently
I would put an automatic stop for a clearly losing variant into the worker itself. Right now, killing a bad test requires someone to notice it and then ship a deploy.