A healthy aggregate can hide a failing candidate
Suppose a release candidate handles 200 requests and fails 20. The stable version handles 19,800 requests and fails 79. Across the service, the error rate is only 0.495%. Inside the candidate, it is 10%. An aggregate dashboard can make a serious release defect difficult to see.
We built six assigned traffic windows and evaluated them with an explicit example policy. The inputs are fictional request counts, chosen to expose different decisions. They are neither production observations nor recommended thresholds for a SaaS product.
| Assigned window | Candidate errors | Control errors | Overall errors | Example decision |
|---|---|---|---|---|
| Healthy | 0.4% | 0.4% | 0.4% | Promote |
| Low traffic | 0% | 0.417% | 0.4% | Pause |
| Candidate regression | 6% | 0.4% | 0.68% | Roll back |
| Aggregate hides failure | 10% | 0.399% | 0.495% | Roll back |
| Critical segment | 1% | 0.4% | 0.46% | Roll back |
| Shared incident | 6% | 6% | 6% | Pause |
The critical-segment case rolls back despite a candidate-wide rate of 1% because one designated operation fails 8 of 50 candidate requests: 16%. The low-traffic case pauses because 40 candidate requests do not meet our assigned minimum. Zero observed errors is insufficient under that rule.
Rates in the table are rounded where needed; the source uses exact fractions for decisions. Each row represents one contemporaneous candidate/control window, not a time series or a confidence interval.
Define what each gate is allowed to conclude
Google's SRE Workbook describes canarying as a partial, time-limited deployment evaluated against a control population. It shows why per-version errors reveal defects that an overall service metric can obscure. Google SRE: canarying releases.
Our model requires at least 200 requests in both version populations. If a critical segment is supplied, each version needs at least 50 requests in that segment. An error rate above 2%, combined with more than a one-percentage-point increase over a healthy control, triggers the example rollback rule.
A candidate above 2% without that increase pauses for investigation. A control above 2% also pauses the release because the example cannot attribute the incident to the candidate. Only windows meeting the evidence requirements and remaining within the assigned error limits can promote.
These values define a transparent exercise. They do not establish statistical significance, a safe exposure duration or a universal service-level objective. A real release policy must derive its thresholds from the product's failure costs, traffic distribution and operational evidence.
Check 1: measure the candidate population directly
Tag observations with the actual serving version and retain the request denominator. A candidate handling a small share of traffic can fail frequently while contributing only a small number of errors to the service-wide rate.
In our fourth window, the candidate receives 1% of all requests but contributes 20 of 99 errors. The overall 0.495% is arithmetically correct. It simply answers a different question from whether the candidate is healthy enough to receive more traffic.
Use the same counting definition on both versions. Decide which requests are eligible, which response outcomes count as errors and how retries affect the denominator. A change in instrumentation can resemble a release regression or conceal one.
The gate should retain raw counts alongside the computed rate. During review, "20 failures out of 200 candidate requests" is easier to inspect than a percentage disconnected from its sample.
Check 2: compare with a contemporaneous control
The shared-incident window has a 6% error rate in both populations. Our policy pauses promotion and routes the situation to investigation. It does not identify the candidate as the cause simply because a deployment is in progress.
A contemporaneous control helps distinguish a version-specific change from a wider dependency failure, but comparable timestamps alone do not make populations equivalent. A canary receiving one region or one customer cohort may see a different workload from the control.
Record the traffic assignment strategy and compare relevant request types. If the candidate serves mostly a difficult operation while the control serves mostly cheap reads, the version-wide averages can mislead. A useful review asks both whether the candidate worsened and whether it received a comparable mix of work.
"Pause" means stop increasing exposure. It does not authorize leaving known harmful traffic running while waiting for a statistical conclusion. An incident response or emergency rollback policy can take precedence over this illustrative decision function.
Check 3: require enough relevant observations
The low-traffic candidate has zero failures in 40 requests. It still pauses because it does not meet the example minimum of 200. The count check prevents the demonstration from promoting on almost no evidence.
Google's guidance emphasizes a population and duration representative enough to exercise the change. Some metrics react quickly, while queue depth or resource accumulation may need longer observation. A fixed timer cannot guarantee that the important paths actually ran.
Request counts also do not make correlated events independent. Repeated traffic from one client or a single batch may offer less varied evidence than the count suggests. This model deliberately stops at transparent arithmetic and policy evaluation; it performs no statistical power calculation.
Missing observations need their own state. A failed metrics query, absent version label or zero denominator should not become a healthy zero-error rate. The download accepts complete positive-count fixtures, so test telemetry absence separately when integrating a real analysis provider.
Check 4: inspect a critical operation separately
The fifth window includes 1,000 candidate requests with 10 errors overall. Within them, a designated critical operation has 50 requests and 8 errors. Its stable counterpart has 450 requests and 2 errors, approximately 0.444%.
The critical segment fails the assigned absolute and relative error checks, producing rollback. Candidate-wide averaging would otherwise hide that outcome behind a 1% rate. The segment is a subset of the version population, so its requests must not be added again to the overall denominator.
Choose segments before evaluating the release. Business-critical writes, a new integration or an affected tenant cohort may need explicit coverage. Searching through many arbitrary segments after seeing a result introduces a different inference problem that this small policy does not address.
Error rate is only one signal. A release can return successful responses while increasing latency, corrupting a result or exhausting a shared resource. Our SaaS noisy-neighbor experiment examines resource competition; the appropriate canary signals depend on what the change can break.
Check 5: separate traffic intent from observed exposure
A declared canary percentage is a routing instruction. Verify which version actually served the requests used in the analysis. Session affinity, low request volume and uneven request costs can make observed exposure differ from a configured percentage.
Argo Rollouts provides setWeight and pause steps. Its documentation explains that without traffic management it approximates the desired split through whole replica counts. More precise routing requires the appropriate traffic-management integration. Argo Rollouts canary strategy.
Do not infer a request-level split from pod counts alone. The model evaluates counts already assigned to each population; it does not deploy a controller or validate a routing integration. Retain observed request distribution as evidence alongside the desired rollout step.
The sequence is to establish version identity, collect comparable windows, validate coverage, apply the explicit gates and then execute the selected action. Promotion remains a decision after evaluation, not an automatic consequence of waiting long enough.
Check 6: prove that rollback restores a usable state
Rollback is an operational action with preconditions. Confirm that stable capacity can receive the traffic, the routing change takes effect and the previous application can still interpret the current data. A healthy old container image is insufficient if an incompatible schema change has already occurred.
Our expand-contract migration example separates application compatibility from destructive schema cleanup. Use that distinction when deciding whether a release can be reversed during the canary window.
If implementation requires help across the delivery pipeline, Pharos Production's DevOps services for CI/CD pipelines, observability and rollback planning describe pipeline automation, infrastructure as code and operational handover with runbooks. Those service areas match the work needed to connect versioned deployments, release signals and a documented recovery procedure.
Treat the rollback procedure as something to execute and inspect. Record the selected version, routing result, application checks and remaining irreversible effects. The gate model's ROLLBACK output is only a decision label; it does not prove that infrastructure performed the action.
Run the policy before wiring it into delivery
Download the Python evaluator, window results as CSV and complete counts and reasons. Save the source as experiment.py and run it with Python 3.
python3 experiment.pyThe standard-library script evaluates all six windows twice, checks identical output and asserts the sequence: promote, pause, roll back, roll back, roll back, pause. It also checks the hidden 10% candidate error rate and the 16% critical-segment rate explicitly.
Before adapting the evaluator, write the consequences of each decision. State who can override it, how missing telemetry is handled and what evidence confirms a completed rollback. Then replay representative historical windows against the proposed rules. A readable policy and its retained inputs make release decisions reviewable when the next deployment behaves differently from the last one.
Sources
Documentation checked .