A code deploy either works or throws. A model deploy usually does neither: the new version returns 200s at the same latency and is slightly worse at the thing you care about. That is why model rollouts need a quality gate as well as a health gate, and why the honest question about a canary is not “did it error?” but “have I seen enough requests to tell?”
Why model deploys are not code deploys
Three properties break the usual assumptions.
- Failure is silent and distributional. A 2% drop in extraction accuracy produces no errors, no latency change and no alert. It shows up as support tickets three weeks later.
- Output is non-deterministic. You cannot diff two responses and conclude anything from one pair. Comparison has to be statistical, over a sample.
- Capacity is the constraint on the strategy. Running blue and green simultaneously means holding two full sets of GPUs. For a large model that doubles the most expensive line in the budget for the duration of the rollout, which is a real reason to prefer a small canary over blue-green.
Add one more, which applies when you are switching between hosted models rather than your own weights: providers update models under a stable name. covers the duplication mechanics, including the trap of shadowing requests that have side effects.
Splitting traffic deterministically
The split must be sticky per user, not per request. A user whose conversation alternates between two model versions gets an inconsistent assistant, and any metric you compute over the session is contaminated. Hash a stable identifier into a bucket:
import hashlib
def variant(user_id: str, canary_percent: int, salt: str = "chat-model-2026-08") -> str:
"""Stable assignment: the same user always lands in the same bucket
for a given salt. Change the salt to re-randomise a later experiment."""
h = hashlib.sha256(f"{salt}:{user_id}".encode()).digest()
bucket = int.from_bytes(h[:4], "big") % 100 # 0..99
return "canary" if bucket < canary_percent else "stable"
Two details. The salt means a second rollout does not reuse the same unlucky users, which otherwise correlates your experiments. And sha256 rather than Python’s built-in hash, because the built-in is randomised per process and would reassign every user on restart.
At the infrastructure layer the same split can be expressed by two Deployments behind one Service with proportional replica counts, or by a weighted route in a service mesh or ingress. The replica-count method is the crudest — a 10% canary means one replica in ten, so you cannot express 1% without ten replicas — but it needs nothing installed. Whichever layer does the splitting, record the variant on the request so that every log line, trace and evaluation can be grouped by it. Without that label the canary produces no evidence at all.
The eval gate
The promotion criterion is a set of thresholds decided before the rollout, checked automatically, with no judgement call at 2am. Three tiers, evaluated in order:
- Offline eval, before any traffic. The frozen eval set from goes further into the multiple-comparison problem you create by watching six metrics at once.
Live traffic is not a randomised experiment unless you made it one. Sticky bucketing gives you random assignment; it does not stop the canary population drifting if the split correlates with geography, tenant or time of day. Check that the two variants saw comparable input distributions before believing a difference in outcomes.
Rollback criteria, written in advance
Write the abort conditions into the rollout plan before it starts, in the form “if X for Y minutes, revert”. The point of writing them down is that at 2am, watching a number wobble, nobody has to decide what counts as bad.
CODE# rollout.yaml — the plan, reviewed with the change
steps: [1, 5, 25, 50, 100] # percent of traffic
dwell: [1h, 4h, 12h, 12h, -] # minimum time at each step
abort_if:
- metric: error_rate_5xx op: ">" value: 0.5% for: 5m
- metric: p95_latency_ms op: ">" value: 1.25 relative_to: stable for: 10m
- metric: schema_valid_rate op: "<" value: 0.98 relative_to: stable for: 30m
- metric: cost_per_request op: ">" value: 1.15 relative_to: stable for: 1h
- metric: thumbs_down_rate op: ">" value: 1.30 relative_to: stable for: 4h
on_abort:
- set canary traffic to 0 # seconds; the stable pods never went away
- keep canary pods running # for diagnosis, not for traffic
- page the on-call, do not auto-promote again without a human
The last two lines are the ones people leave out. Keeping the canary pods running after an abort preserves the evidence — logs, traces, a live process to inspect — and setting traffic to zero rather than deleting the deployment makes the rollback take seconds rather than a scheduling cycle. What to do next is in
- On-Call Runbooks for AI Services
SOCIAL SHARE CARD GENERATOR