So I tried to build an experiment service (here's every wrong turn first)

Say you run a website and have a hunch a new signup page converts better than the one you've got. You could just swap it and hope. Or — and I promise this is the more fun option — you could actually find out. Show half your visitors the old page, half the new one, wait for enough people to walk through, and compare who actually signed up.

That's A/B testing. And the second you try to do this "for real" instead of eyeballing a spreadsheet, a whole vocabulary falls out of it whether you like it or not. What you're testing — "does the new page convert better" — is an experiment. Each version on trial is a variant; the one you'd show if nobody were testing anything is the control. You don't have to split traffic 50/50 either — maybe I only trust the new page enough to show it to 10% of visitors while I watch nervously — so every variant carries a weight, its slice of the traffic. When someone actually sees a variant, that's an exposure. When they go do the thing I actually cared about, that's a conversion. Conversions over exposures, per variant, is the number that eventually tells me who won.

None of that vocabulary is hard. What's hard is building the service that makes all of it happen correctly, at real scale, without ever slowing down the page it's sitting on. That's what I actually sat down to build.

What does this thing even need to do?

Working backward from that vocabulary, a handful of jobs fall out almost on their own:

• Something to define an experiment — its variants, their weights.


• Something that, given a visitor and an experiment, decides which variant they see — and does it consistently, because a visitor flip-flopping between variants across visits makes the whole experiment meaningless.


• Something to record exposures and conversions as they happen.


• Something to turn all that raw event data into a number a human can actually read.


• And since more than one business would plausibly use this thing, some notion of "whose experiment is whose" — a tenant, or here, a "client."

Five jobs, five APIs: signup, experiment configuration, assignment, event tracking, reporting — plus a dashboard, because nobody wants to read raw JSON to figure out which button color won.

flowchart TD

    A[Signup API<br/>get an API key] --> E[(Database)]

    B[Config API<br/>define variants] --> E

    C[Assign API<br/>must be fast]:::hot --> E

    D[Event API<br/>track exposures] --> E

    E --> F[Dashboard<br/>per-tenant view]

    classDef hot stroke-width:3px

One thing worth tattooing on my forearm before going any further: the assignment API gets called on every single page view and has to answer in milliseconds, because a real visitor's browser is sitting there waiting on it. Everything else — configuring an experiment, logging an event, checking the dashboard — can afford to be slower, because nothing real-time is blocked on it. That one asymmetry, one latency-critical read path and everything else lax, ends up justifying almost every decision below.

The API, endpoint by endpoint (copy-paste these)

POST /clients — sign up

No auth here — you can't require a key to get your first key.

curl -X POST https://api.yourexperimentservice.com/clients \

  -H "Content-Type: application/json" \

  -d '{"name": "Acme Labs"}'

Response:

{ "id": "cl_9f2a", "name": "Acme Labs", "api_key": "sk_live_..." }

POST /experiments — configure an experiment

Requires the API key from signup.

curl -X POST https://api.yourexperimentservice.com/experiments \

  -H "Content-Type: application/json" \

  -H "X-API-Key: sk_live_..." \

  -d '{

    "key": "signup_test",

    "status": "live",

    "variants": [

      { "key": "control", "weight": 70, "static_copy": "Welcome!" },

      { "key": "variant_b", "weight": 30, "ai_prompt": "Write a short signup headline" }

    ]

  }'

Response: the persisted experiment, with every variant's static_copy filled in — generated ones included.

GET /assignment — the hot path

curl "https://api.yourexperimentservice.com/assignment?user_id=visitor_123&experiment_key=signup_test" \

  -H "X-API-Key: sk_live_..."

Response:

{ "experiment_key": "signup_test", "assigned_variant": "control", "static_copy": "Welcome!" }

Same user_id + experiment_key always returns the identical result, indefinitely.

POST /events — record an exposure or conversion

curl -X POST https://api.yourexperimentservice.com/events \

  -H "Content-Type: application/json" \

  -H "X-API-Key: sk_live_..." \

  -d '{

    "visitor_id": "visitor_123",

    "experiment_key": "signup_test",

    "variant_key": "control",

    "event_type": "exposure"

  }'

Response:

{ "status": "success", "event_id": "ev_88b1" }

A repeated identical exposure returns this same shape with the same event_id — not an error.

GET /results — reporting

curl "https://api.yourexperimentservice.com/results?experiment_key=signup_test" \

  -H "X-API-Key: sk_live_..."

Response:

{

  "experiment_key": "signup_test",

  "results": [

    { "variant_key": "control", "exposures": 812, "conversions": 94, "conversion_rate": 0.116 }

  ]

}

Dashboard — not curl-able, and that's fine

This one's a browser page, not another API call. A login page trades the same API key for a session, then shows the same data as /results — scoped to that one tenant. It needs its own auth mechanism because a plain page visit can't attach a custom header the way an API client can.

The signup API

Almost embarrassingly simple, and that's fine — not everything needs to be hard. A client signs up, gets an API key back, and attaches that key to every other call so the service knows whose experiments it's actually looking at.

The experiment API — and the first real decision

Configuring an experiment means naming it and defining variants and weights. Easy enough. But there was a genuine feature request buried in here: what if a variant's copy — a headline, a call to action — should be written by an AI model instead of typed by a human?

My first instinct was to make it dynamic: whenever we're about to show a variant, if it's AI-generated, call the model right then and get fresh copy. Sounds flexible, doesn't it? It's also a terrible idea the second I remember what I just said about the assignment path — it has to be fast, and it can't depend on anything that might be slow or might just... not respond. A model call can take seconds, time out, or vanish for reasons entirely outside my control. Putting that on the one path that has to answer in milliseconds, for every page view, is exactly the wrong place for it.

So instead: does this actually need to happen at decision time, or could it happen once, earlier, with the answer just remembered afterward? Nobody needs bespoke copy generated in the gap between a page loading and the server replying — they need copy that was already decided before that moment. So that's where it goes: the model gets called once, when the experiment is first configured — a moment that can comfortably afford a second or two, because nothing real-time is waiting on it — and whatever it returns gets written down permanently. The assignment path never talks to the model, ever. It just reads back what was already decided. And if the model call fails outright, the system falls back to plain, boring, serviceable text instead of blocking the one thing that's genuinely not allowed to fail.

The assignment API — where the actual hard problem lives

This is the one that's called constantly, has to be fast, and has to be consistent — same visitor, same experiment, same answer, every single time, forever.

My first instinct: when a new visitor shows up, pick a variant (weighted-random, respecting the split), and store that decision so I can look it up next time. Totally reasonable first move, and worth actually sitting with why it's not quite right instead of dismissing it out of hand. The problem is that "store it and look it up" turns every single assignment call — remember, the one on the critical path — into a stateful read-or-write. What happens under heavy concurrent load? What happens if two requests for a brand-new visitor land at the exact same instant, before either has finished writing? You can solve all of that with locking and careful transaction handling. But you're solving a problem you invented for yourself, by treating "which variant" as something that needs to be decided once and remembered, instead of something that could be recomputed identically, every time, from nothing.

That reframing is the actual unlock. If assignment is a pure function of (visitor, experiment) — nothing else, no external state — I never need to store the decision at all. I only need the two inputs, and I already have both on every request. Hash the visitor ID and the experiment key together, turn that into a number between 0 and 99, and slice the range according to the configured weights — bottom 70 numbers to variant A on a 70% split, the rest to variant B. Same visitor, same experiment, same hash, same number, every single time, computed fresh, remembered nowhere. It's basically a rigged coin that always lands the same way for the same person — which sounds like cheating until you remember that's exactly the point.

Here's a distinction worth sitting with, because it's genuinely easy to blur: this gives two different guarantees, and testing one doesn't automatically validate the other. For any one visitor, this is completely deterministic — no randomness anywhere in an individual decision. Across many different visitors, the outputs merely look randomly scattered, which is precisely the property that makes a configured 70/30 split actually land close to 70/30 across real traffic. That second property is worth actually checking empirically rather than assuming it, and it's worth being careful about how you check it: a small sample can look meaningfully skewed purely from noise, for a perfectly correct system, and a small sample can just as easily look fine when something's actually wrong. Only a genuinely large sample tells you anything trustworthy about whether the population-level split is real. It's a subtle enough trap that I'd double-check with a large sample before trusting a "yeah, looks about 70/30" glance at a dozen test visitors.

The event API — and the problem real visitors bring with them

Recording an exposure or a conversion sounds like the simplest operation in the whole system: someone saw a variant, write a row. The complication shows up the instant real visitors replace clean test data — people double-click, retry failed page loads, hit back and forward. The same "I saw this" signal can legitimately arrive more than once for the exact same visitor and variant, and if I just write a row every time, every conversion-rate number downstream is now quietly inflated.

Idempotency: making duplicates impossible instead of catching them

The natural first fix is application-level: before writing a new exposure, check whether I've already recorded this exact one, and skip it if so. Worth trying, and worth understanding why it doesn't fully hold up — "check" and "then write" are two separate steps with a gap between them, and two duplicate requests arriving close enough together can both sail through the check before either has actually written anything. I've reduced the problem, not eliminated it.

The concept that actually closes the gap is called idempotency — designing an operation so that doing it once and doing it five times leave the system in exactly the same state. The way to actually get there is to stop trying to detect duplicates in application code, and instead make them structurally impossible to write at all: a hard uniqueness constraint at the storage layer itself, on (visitor, experiment, variant, event type). A duplicate isn't something my code has to notice anymore — it's something the write itself refuses to allow, no matter how many requests land at the same instant. My application logic shrinks to something almost anticlimactic: if a write gets rejected as a duplicate, that's a success, not an error, because the thing the caller wanted recorded already is recorded. Think of it as a bouncer with a guest list instead of a bouncer trying to remember every face — the list itself won't let the same name in twice, no matter how fast people show up.

It's a pattern worth carrying well beyond this one system: wherever you can, push "detect and reject a duplicate" down into "make the duplicate physically impossible," as close to the actual data as the underlying store will let you.

The dashboard

Once exposures and conversions are flowing in, someone needs to actually look at the results — per experiment, per variant: exposures, conversions, conversion rate. The one design wrinkle worth mentioning is that this is a browser page, not another API call, and a tenant should only ever see their own experiments, never anyone else's — the same tenant-isolation principle from the very first requirements list, just showing up again on a different surface.

The data model

Four entities, and the relationships between them are doing real work, not just organizing storage:

erDiagram

  CLIENT ||--o{ EXPERIMENT : owns

  EXPERIMENT ||--o{ VARIANT : has

  EXPERIMENT ||--o{ TRACKING_EVENT : logs

  VARIANT ||--o{ TRACKING_EVENT : recorded_under

  CLIENT {

    uuid id PK

    string name

    string api_key

  }

  EXPERIMENT {

    uuid id PK

    uuid client_id FK

    string key

    string status

  }

  VARIANT {

    uuid id PK

    uuid experiment_id FK

    string key

    int weight

    string static_copy

  }

  TRACKING_EVENT {

    uuid id PK

    uuid experiment_id FK

    uuid variant_id FK

    string visitor_id

    string event_type

    timestamp created_at

  }

• 
  Client: the tenant. Everything else exists only in the context of one of these.


• 
  Experiment: belongs to exactly one client. Its name only has to be unique within that client, not globally — which is exactly what makes "scope every lookup to the tenant" a hard requirement, not a nice-to-have. Two clients can legitimately name an experiment the same thing; any lookup that isn't scoped by client risks serving one tenant's config to another tenant's request.


• 
  Variant: belongs to an experiment, carries a weight and either literal or AI-generated content.


• 
  TrackingEvent: one row per exposure or conversion, referencing both the experiment and the specific variant it happened under, carrying the idempotency-enforcing uniqueness constraint from the section above.

Worth naming explicitly: this ledger is append-only, not a set of counters incremented in place. I could track "exposures so far" as one mutable number per variant, but in-place counters have their own race conditions under concurrent writes, and they throw away the ability to answer questions I didn't think to ask up front — when exactly did this happen, in what order did events actually arrive. A new row per event, nothing ever mutated, trades some storage growth for meaningfully better correctness and flexibility. A diary, not a whiteboard I keep erasing.

Scaling: what breaks first, and what actually fixes it

Everything above works comfortably at modest volume. The interesting question is what gives out first as traffic grows — and matching each failure to the concept that actually addresses it, not reaching for every scaling technique that exists because it sounds thorough.

                        ┌───────────────────┐

   Configure   ───────► │  Experiment API   │───► background job queue ───► LLM call

                        └─────────┬─────────┘                                  │

                                  │                                            ▼

                                  ▼                                    variant.static_copy

                        ┌───────────────────┐        ┌─────────┐        (written once,

   Every page  ───────► │  Assignment API   │◄──────►│  Cache  │         read forever)

   view (hot)           └─────────┬─────────┘        └─────────┘

                                  │  (cache miss only)

                                  ▼

                        ┌───────────────────┐

                        │   Primary DB      │◄──── writes only

                        └─────────┬─────────┘

                                  │  replication

                        ┌───────────────────┐        ┌──────────┐

   Every event ───────► │  Event API        │───────►│  Queue   │──► workers ──► batched writes

                        └───────────────────┘        └──────────┘

                        ┌───────────────────┐

   Dashboard   ───────► │  Read replica     │  (reporting queries never touch the primary)

                        └───────────────────┘

Caching

The first thing to strain is reads on the assignment path. The query itself stays fast and indexed, but at high enough concurrency, requests start queuing for a free database connection, because configuration data is read constantly and almost never changes. Read far more than written is exactly the case a cache is built for. A cache-aside layer sitting in front of the database, invalidated whenever the underlying configuration actually changes, takes the database out of the hot path almost entirely except on a miss.

Background jobs, for the AI call

Once experiment configuration itself becomes frequent rather than occasional, the AI-generation step is worth revisiting: move it fully into a background job queue, so a slow model call can't tie up a web server thread at all, rather than merely being "synchronous but tolerable."

Message queues, for the write path

Further out, the write side starts to strain: every exposure and conversion is a synchronous insert, and the uniqueness constraint that gives us idempotency becomes, at high enough volume, itself a point of contention among many concurrent writers. The fix is to put a queue in front of the write — accept the event, acknowledge immediately, let a pool of workers drain the queue and perform the actual writes, often batched together, which is far cheaper per row at real volume than one insert at a time.

Read replicas

This is also the point where reporting queries start meaningfully competing with write traffic on the same database, which is the natural moment to introduce a read replica dedicated to reporting/dashboard queries, leaving the primary free to focus on writes.

Partitioning

Further still, the event ledger itself becomes the bottleneck — large enough that its indexes stop comfortably fitting in memory, and even indexed queries degrade. The fix here is partitioning, most naturally by time, since "how long do I actually need raw events" is a real, answerable retention question in a way that visitor or experiment distribution isn't. At true extremes, raw events stop being the right thing to query for reporting at all, and a separate streaming layer that maintains pre-aggregated counts becomes the actual read path instead.

Load balancing

The app tier scales horizontally with essentially no extra design work, provided it stays stateless — no server-side session storage anywhere, so any instance can serve any request and a load balancer can distribute purely by load, with no session-affinity rules to get wrong. The one thing that genuinely has to be right: the load balancer's health check needs to hit a dedicated, request-independent liveness endpoint, never a page whose response can legitimately vary by who's asking (a login-gated page returning a redirect for a signed-out visitor, for instance) — otherwise the load balancer can misread completely normal behavior as an unhealthy instance and start cycling servers that are actually fine.

Rate limiting, for multi-tenant fairness

Nothing so far stops one tenant's traffic spike from degrading latency for every other tenant sharing the same infrastructure. Per-tenant rate limiting, applied before it's ever actually needed, is far cheaper than retrofitting it after a real incident where one customer's bad day became everyone's bad day.

Observability

None of the fixes above should be applied before I can actually see the specific metric that motivates it — cache hit rate, queue depth, replica lag, per-tenant volume. Introducing infrastructure speculatively, because it sounds like the mature thing to do, is its own kind of mistake — solving a problem I haven't confirmed I actually have yet.

Closing thought

The organizing principle underneath almost everything above, worth stating plainly: on a system that sits directly on a page's critical path, the safe default when anything goes wrong — an experiment nobody's ever heard of, a failed external call, a cache that's unreachable — is to silently fall back to the plainest, safest experience, never to error or hang or block. A real visitor should never be able to tell that an experimentation system is having a bad day. Making the happy path fast turned out to be the easy part. Making every unhappy path invisible was the actual design problem.