invokationA rate limit belongs to a scope
Handling API rate limits starts by identifying what is limited. A provider may apply quotas by account, credential, customer, endpoint, operation, or a combination. A single global “requests per second” setting rarely models all those boundaries. Record the documented scope before choosing a queue or retry policy.
A fictional application uses transactional email, a record API, and a messaging API. The email account hits a dispatch quota. A shared worker pool that repeatedly waits on email retries can delay record reads that have no relationship to that quota. Fair scheduling matters as much as calculating a backoff interval.
The examples here are deterministic illustrations, not throughput benchmarks or claims about a particular provider’s limits.
Separate permission to retry from permission to send now
A retryable transport or quota error means another attempt might be useful. It does not mean the attempt should happen immediately, or that repeating the business action is safe. Check three conditions: the operation remains relevant, the side-effect policy permits another attempt, and the quota scheduler allows it.
For an interactive read, waiting beyond the user’s deadline may be pointless. For a queued write, an ambiguous prior outcome may require reconciliation before another attempt. A mapping-validation error should not enter a backoff loop at all. Authentication renewal has its own bounded budget and must not multiply retries of the business operation.
Respect the destination’s timing signal
HTTP Retry-After semantics allow a delay in seconds or an HTTP date. Parse the supported form, account conservatively for clock differences, and avoid sending earlier than the applicable delay. Malformed or absent headers need a documented fallback rather than an immediate retry storm.
A delay longer than the operation’s remaining budget should move the operation to a later scheduled state or end it with an actionable outcome. Do not silently clamp a provider’s 10-minute delay to your local 30-second cap and then claim to honor the header.
For example, imagine attempt one at simulated time 0 receives a minimum wait of 30 seconds. A configured jitter of 4 seconds makes the next eligible time 34. If that attempt receives another 30-second wait, it cannot run again before time 64; an additional 2 seconds of jitter would make it 66. These are chosen fixture values, not sampled performance results.
Use bounded backoff and jitter
A common local fallback is exponential backoff with random jitter, capped by a maximum delay and total elapsed-time budget. Randomness spreads independent callers across time, but it does not replace coordination for a genuinely shared quota.
// Policy pseudocode; not a complete HTTP client.
function nextAttempt({ attempt, now, deadline, retryAfterMs, jitterMs }) {
const localBackoff = Math.min(30_000, 500 * 2 ** (attempt - 1));
const minimum = Math.max(localBackoff, retryAfterMs ?? 0);
const eligibleAt = now + minimum + jitterMs;
if (attempt >= 4 || eligibleAt >= deadline) return { stop: true };
return { eligibleAt }; // Scheduler still checks quota and side-effect safety.
}
Validate the parsed delay and bound arithmetic before using it. The jitter source, permitted range, and deadline behavior are policy choices. The example does not perform the request, verify idempotency, or implement a distributed quota counter. Those responsibilities should remain visible in a real client.
Partition capacity without inventing quota
| Scheduling boundary | Why it helps | What it does not establish |
|---|---|---|
| Provider account | Coordinates a documented shared quota | That each credential has independent allowance |
| Connection / credential | Contains one profile’s backlog | That another profile cannot share its provider limit |
| Operation class | Separates interactive reads from bulk work | That the provider recognizes your local categories |
| Customer or tenant | Prevents one workload monopolizing capacity | That per-customer fairness increases total throughput |
Use a hierarchy when limits overlap. An operation may need permission from both an account-wide budget and a connection-specific concurrency slot. Release worker capacity while waiting for a future attempt rather than sleeping inside every worker. Keep the scheduled job durable so process restarts do not lose its due time.
Concurrency limits and circuit breakers solve different problems
A concurrency limit caps simultaneous in-flight work. A rate limiter controls activity over time. A circuit breaker can stop repeated calls to a failing dependency and allow a bounded probe later. None of these mechanisms creates destination capacity or proves that a prior write failed.
Consider a slow custom API that takes several seconds to respond. A low request rate can still create many simultaneous calls if each remains in flight. Conversely, a very fast API can exhaust a per-minute quota without using much concurrency. Model the observed failure rather than applying the same knob everywhere.
Use circuit state per relevant failure domain. One invalid credential should not open a provider-wide circuit. A broad outage might justify shared suppression, but keep recovery probes bounded so every connection does not probe simultaneously.
Observe waiting as well as failing
A successful-request graph can look healthy while queued work waits beyond useful deadlines. Track queue age, next eligible attempt time, quota scope, attempt count, and final disposition. Preserve redacted destination headers only when they are useful and safe. Avoid putting account IDs or arbitrary response text in high-cardinality metrics.
Alert when a backlog will miss an application deadline, not every time a documented quota returns a retryable response. Distinguish provider rate limits from your own admission control so the owner knows which policy to change. The monitoring guide explains the connection and operation views that make this actionable.
A review checklist
- Identify documented provider, account, credential, customer, and operation scopes.
- Decide read/write retry eligibility before scheduling attempts.
- Honor valid destination timing without violating total deadlines.
- Bound attempts, elapsed time, concurrency, and queue age.
- Isolate unrelated dependencies while coordinating genuinely shared quotas.
- Test two connections sharing a limit and two that should remain independent.
A central operating model can make these policies consistent across connections. It still requires explicit destination semantics. Read the idempotency and replay guide before treating a well-timed retry as a safe write, or explore the developer design preview for bounded requests and queued runs.