Rate Limiting

Rate limiting is a traffic-control method that restricts how many requests, messages, packets or operations a source or class may perform in a defined time interval. It protects shared capacity, enforces fairness, bounds abusive or faulty clients, and prevents one source from turning a local problem into system-wide overload.

The method appears in APIs, distributed services, packet networks, telemetry pipelines, embedded diagnostics, control gateways and shared infrastructure. A rate limit should state what is counted, who or what is keyed, how bursts are handled, which response is returned, and how retry behavior is controlled.

Rate Cap

For a source with incoming rate:

and configured limit:

the allowed rate satisfies:

The rejected, delayed or degraded rate is:

This screen is simple, but the engineering details matter: measurement window, burst allowance, clock consistency, key selection, priority class and retry contract determine whether the limiter protects the system or only moves the overload somewhere else.

Fixed and Sliding Windows

For a window length:

and rate limit:

the maximum count in the window is:

A fixed window is simple but can allow two bursts near a boundary. A sliding window or rolling counter reduces boundary artifacts by estimating the count over the most recent interval. The validation case should include boundary traffic, clock skew and concurrent limiter instances.

Token Bucket

A token bucket allows bounded bursts while enforcing a long-term rate. Let bucket capacity be:

and token refill rate be:

tokens per second. If a request costs:

tokens, it is allowed only when:

After a time step:

the token state is:

for an accepted request. If there are not enough tokens, the system can reject, delay, queue, downgrade or route the request according to the response contract.

Burst Duration

If a source sends a burst at:

and the bucket starts full, the approximate time before burst tokens are exhausted is:

After the bucket is empty, sustained allowed rate returns to approximately:

unless a higher-level admission rule, priority class or load-shedding policy applies.

Fairness Scope

The rate-limit key determines fairness. A limit can be per user, device, tenant, API key, route, IP address, queue, topic, diagnostic tool or control class. The wrong key can make the system unfair. A per-IP limit may punish many users behind a gateway. A global limit may allow one tenant to starve others. A per-route limit may protect an endpoint but leave a shared dependency overloaded.

For:

sources each limited to:

the aggregate admitted rate is:

This aggregate still needs to fit the protected capacity:

Worked Example

An API gives each diagnostic client a token bucket with:

and burst capacity:

A faulty client sends:

with one token per request. The bucket drains at:

The burst allowance lasts:

After that, the client is limited to:

and the excess rate is:

If there are:

diagnostic clients at the same limit, aggregate admitted traffic is:

For service capacity:

the remaining capacity margin is:

This does not prove the system is safe, but it shows that the configured per-client limit can fit the shared capacity before retries and priority traffic are added.

Retry Interaction

Rate limiting should be coordinated with retry budgets. If a limiter returns an immediately retryable response, clients can convert a controlled limit into a retry storm. A response contract should state whether the client should stop, wait until a retry-after time, use jittered backoff, reduce quality, drop noncritical telemetry or enter degraded mode.

If retry multiplier is:

then attempt load after retries is:

The limit should be set so:

or the protected resource may still overload even though the original request rate appears limited.

Relationship To Neighbor Terms

Rate limiting is narrower than admission control. Admission control decides whether work may enter based on capacity, priority, deadline and state. Rate limiting is a specific method that bounds frequency over time. Load shedding is a broader overload response that may reject, drop or degrade work after an overload trigger. Queue backpressure asks cooperative producers to slow down based on downstream queue state. A token bucket rate limiter can support these mechanisms, but it does not replace their system-level decisions.

Rate limiting is also different from actuator rate limits and electronic slew-rate limits. Those constrain physical or analog rate of change. Software and network rate limiting constrains operation frequency or traffic volume.

Validation Evidence

Validation should test nominal traffic, burst traffic, boundary windows, clock skew, distributed limiter consistency, per-key fairness, priority classes, retry behavior, response contracts, telemetry counters, failure of the limiter store and recovery after limit release.

Useful metrics include allowed count, limited count, delayed count, rejected count, retry-after compliance, queue depth, p95 and p99 latency, dependency attempts, tenant fairness and error-budget impact. A limiter that silently drops work, hides the rejection reason, counts the wrong key or triggers synchronized retries can make reliability worse while appearing to reduce traffic.

Common Mistakes

The most common mistake is setting a rate limit without linking it to protected capacity. Another is limiting original requests while ignoring retries, batch replays or queue consumers. A third is using a global limit where fairness requires per-tenant or per-class limits. A fourth is treating rate limiting as security by itself; abuse control may need authentication, authorization, anomaly detection and incident response.

A strong rate-limit design states the limit, the burst allowance, the counted unit, the key, the response, the retry rule, the protected resource and the evidence that the limit holds under realistic concurrency.