Twenty per second can admit forty almost together
Suppose an API allows 20 requests in each clock-aligned second. A client sends 20 just before the next second begins and another 20 just after it begins. The limiter can accept all 40 while obeying its configured rule. The application still receives a burst that is much larger than the phrase "20 per second" might suggest.
Our deterministic admission model places the first group at 990 milliseconds and the second at 1,010 milliseconds. A fixed one-second window admits all 40 requests across that 20-millisecond span. A token bucket with a capacity of 20 and a refill rate of 20 tokens per second admits the first 20 and rejects the second group because only 0.4 token has returned.
| Admission rule | Refill or window rate | Bucket capacity | Accepted requests | Accepted work units |
|---|---|---|---|---|
| Fixed one-second window | 20 requests per second | Not applicable | 40 | 120 |
| Request-count token bucket | 20 tokens per second | 5 | 5 | 5 |
| Request-count token bucket | 20 tokens per second | 10 | 10 | 10 |
| Request-count token bucket | 20 tokens per second | 20 | 20 | 60 |
| Work-charged token bucket | 20 work tokens per second | 20 | 12 | 20 |
The work units are assigned inputs, not measured CPU time. Each group contains 10 cheap requests costing one work unit and then 10 expensive requests costing five. Cheap requests arrive first within a timestamp. Changing that order can change which requests are admitted, especially with work-based charging.
A token bucket has two independent controls
The bucket capacity limits the reserve available for a burst. The refill rate determines how quickly that reserve returns. With capacity B, rate r and elapsed time measured in seconds, refill is min(B, current_tokens + r * elapsed). Admit a request only when enough tokens exist for its cost, then subtract that cost.
RFC 3290's token-bucket discussion provides the rate-and-burst model in a traffic-management setting. Here we adapt the accounting units to application requests and assigned work. The example is a strict admission decision: a rejected request does not wait inside a shaping queue.
An initially full bucket is an explicit assumption. A restarted limiter that forgets its state can unintentionally grant another full reserve. A deployment that starts several independent instances can do the same at a larger scale if each instance believes it owns the entire tenant allowance.
Lowering the bucket capacity from 20 to 10 changes the observed accepted count from 20 to 10 without changing the refill rate. Capacity 5 admits five requests. These values are sensitivity examples, not recommended defaults. Choose a burst allowance from the amount of simultaneous work the downstream system can absorb and the client behavior you intend to support.
Count the resource you mean to protect
In the capacity-20 request-count run, the first group consumes all 20 tokens. Those requests represent 60 assigned work units: ten cheap operations contribute 10 and ten expensive operations contribute 50. The limiter treats both classes as equal because its unit is a request.
The work-charged run spends one token for a cheap request and five for an expensive one. It admits the first ten cheap requests and two expensive requests, consuming 20 work tokens across 12 accepted calls. That is less accepted work under this particular ordering. It does not establish that the policy is fair to expensive operations or that the assigned cost predicts actual execution.
Google SRE's discussion of overload explains why queries per second can be a poor proxy when requests have different resource requirements. Use measurements from the resource you want to protect to calibrate a cost model. A request with a tiny payload can still trigger an expensive query or a large export.
Estimated costs will sometimes be wrong. Decide what happens when actual work exceeds the estimate: enforce a runtime budget, charge the difference, restrict the operation size or require asynchronous admission. A token decision at the edge cannot cap resources it neither measures nor controls.
Keep rate, concurrency and fairness separate
A rate limiter controls admission over time. A concurrency limit controls how much admitted work remains in progress. If requests become slower, the same admitted rate can occupy more worker slots and database connections. Waiting for a token does not reserve a connection safely, and receiving a token does not prove a connection is available.
| Resource or promise | Control to consider | What it does not establish |
|---|---|---|
| Sustained arrival volume | Tenant and endpoint rate budgets | A bound on long-running work already admitted |
| Short burst absorption | Bucket capacity | Fair progress for every workload class |
| Occupied workers or connections | In-flight limit with release handling | A long-term usage entitlement |
| Unequal operation cost | Cost-aware admission and runtime budgets | Accurate cost without measurement |
| Progress among tenants | Scheduling and queue admission policy | Security isolation of tenant data |
The noisy-neighbor queue experiment examines the scheduling side of this boundary. A tenant can obey a rate limit while still monopolizing a shared queue with long-running work. Queue policy and rate policy should be reviewed together, using separate metrics for accepted volume and in-flight occupancy.
The proposed request path in the diagram is:
- Identify the authenticated tenant and the requested work class.
- Evaluate the applicable tenant, endpoint and shared budgets.
- Charge the selected rate cost through an atomic admission decision.
- Acquire execution capacity or place work into a bounded queue.
- Release execution reservations on completion or failure and record actual usage.
If capacity cannot be acquired after tokens were charged, choose and document a refund policy. Partial refunds, no refunds and reservation-based accounting produce different client behavior. With multiple budgets, avoid consuming one allowance permanently when another rejects the same request unless that is the intended contract.
A distributed limiter needs a failure policy
The reference model has one admission authority and an ordered input list. A real fleet needs agreement about shared counters. Separate per-instance buckets can be useful as local protection, but their aggregate capacity is the sum of the independent allowances unless a coordination mechanism assigns smaller shares.
For a shared store, the refill, eligibility check and debit must behave as one decision. A client-side read followed by a later write can allow concurrent requests to spend the same balance. Define how the clock is sourced, how state expires and what happens when the store cannot be reached.
Failing open favors availability and can admit excess work. Failing closed protects the budget and can reject legitimate customers during a limiter outage. Neither choice is universally correct. Some operations can fall back to conservative local limits while expensive mutations require a stricter boundary.
Managed gateway behavior also has limits. Amazon API Gateway's REST API throttling guidance describes rate and burst settings and calls throttling best effort. Treat the documented service contract as the source for provider guarantees. The strict reference bucket in this article is not a simulation of API Gateway's implementation.
Tell clients what a rejection means
RFC 6585 defines HTTP 429 for too many requests and permits a Retry-After header. It does not prescribe how a service identifies a caller or counts requests. Your response should explain the relevant limit in a way the client can act on, without exposing another tenant's state.
For a simple token bucket, the time until a request can afford its cost is (cost - available_tokens) / refill_rate when the balance is insufficient. This is an estimate that assumes no competing request spends the replenished tokens first. If the operation costs more than the bucket's maximum capacity, waiting cannot make it eligible. The operation needs a different path or a revised contract.
A retry after a rate rejection can itself add load. Bound retries and use the provider's documented waiting guidance. For a mutation with an uncertain outcome, preserve the original operation identity as described in the API idempotency article. A fresh key is not a general way to repair a failed attempt.
Measure admitted requests, rejected requests, charged work and actual resource occupancy separately. Observe the oldest queued work and the distribution across tenants. A low rejection rate can coexist with an overloaded database if the accepted operations are expensive enough.
Replay the boundary before choosing settings
The standard-library Python model can be saved as experiment.py and run with Python 3. It writes results.json beside the script. The saved trace records each request's timestamp, assigned work, admission decision and remaining balance for every policy.
The implementation uses integer milliseconds and milli-tokens, avoiding floating-point accumulation in its admission decision. It also checks that a stream of 100 unit-cost requests spaced 50 milliseconds apart remains admitted at the refill rate, and that a request costing six tokens cannot enter a bucket with capacity five. A second complete run must match the first output.
No request actually executes. The model contains no distributed store, network delay, clock skew, concurrency reservations or queue. Use the provided cases to establish the meaning of your settings, then test the missing failure boundaries in your service. The number printed beside a rate limit becomes useful only when the burst, cost unit and coordination scope are equally explicit.
Sources
Documentation checked .