Most LLM API error handling starts out looking like normal HTTP error handling.
If it is a 429, retry later.
If it is a 500, retry with backoff.
If it is a timeout, maybe retry once.
That works fine right up until your first real production incident, where the question is no longer:
"Did the API return an error?"
The question becomes:
"What should the system do now?"
That is where normal HTTP error handling starts to feel too shallow.
The problem is not the error code
A 429 from an LLM provider can mean very different things depending on the provider, account, model, and workload.
You might be hitting:
- requests per minute
- tokens per minute
- daily quota
- concurrency limits
- temporary provider-side throttling
- a model-specific capacity limit
Those are not the same operational problem.
If you hit requests per minute, slowing request frequency may help.
If you hit tokens per minute, retrying the same large prompt immediately is probably useless.
If the provider is overloaded, retrying aggressively can make the incident worse.
If your own queue is backing up, switching models may hide the symptom while the real issue keeps growing.
The HTTP status code tells you the broad class of failure. It does not tell you the correct response.
SDK error handling is not an incident policy
Most SDKs are designed to help you make a request and catch an exception.
That is useful, but it is not the same as a production policy.
An SDK can tell you:
- the request failed
- the provider returned
429
- the provider returned
529
- the request timed out
- the response body had a certain error shape
But your application still has to decide:
- should this be retried?
- should the request be queued?
- should we switch models?
- should we degrade the feature?
- should we show the user a specific message?
- should this page someone?
- should we stop retrying because this action has side effects?
That decision tree usually does not live in the SDK.
In a lot of teams, it lives in a runbook someone wrote after the first painful incident.
That is better than nothing, but it creates a gap: the code sees the error first, while the runbook explains what humans should do later.
The failure policy is the layer in between.
What I mean by a failure policy
A failure policy is a small layer near your LLM client that maps provider-specific errors into internal failure categories and actions.
Instead of spreading logic like this across your app:
if (error.status === 429) {
retry();
}
I prefer something closer to this:
const decision = classifyLlmFailure(error, requestContext);
if (decision.action === "retry") {
await retryWithBackoff();
}
if (decision.action === "queue") {
await enqueueForLater();
}
if (decision.action === "fail_closed") {
throw new UserVisibleError(decision.userMessage);
}
The important part is that the policy returns a decision, not just a label.
A useful failure policy usually answers four questions:
- What happened?
- How confident are we?
- What should the system do first?
- What should humans look at if this continues?
A small example
Here is a simplified JavaScript example.
This is not meant to cover every provider or SDK. The point is the shape of the layer.
function classifyLlmFailure(error, context = {}) {
const status = error.status || error.response?.status;
const headers = normalizeHeaders(error.headers || error.response?.headers || {});
const provider = context.provider || "unknown";
if (status === 429) {
const resetRequests = headers["x-ratelimit-reset-requests"];
const resetTokens = headers["x-ratelimit-reset-tokens"];
const retryAfter = headers["retry-after"];
if (resetTokens && !resetRequests) {
return {
category: "rate_limit_tokens",
action: "queue",
retryable: true,
confidence: "medium",
logLevel: "warn",
runbook: "llm-rate-limit-tokens",
reason: "Provider returned token reset metadata."
};
}
if (resetRequests && !resetTokens) {
return {
category: "rate_limit_requests",
action: "retry",
retryable: true,
confidence: "medium",
retryAfterMs: parseRetryAfter(retryAfter),
logLevel: "warn",
runbook: "llm-rate-limit-requests",
reason: "Provider returned request reset metadata."
};
}
return {
category: "rate_limit_unknown",
action: "slow_down",
retryable: true,
confidence: "low",
retryAfterMs: parseRetryAfter(retryAfter),
logLevel: "warn",
runbook: "llm-rate-limit-unknown",
reason: "429 without enough metadata to distinguish request vs token limits."
};
}
if (status === 529 || status === 503) {
return {
category: "provider_overloaded",
action: "retry_with_jitter",
retryable: true,
confidence: "medium",
logLevel: "warn",
runbook: "llm-provider-overload",
reason: "Provider appears temporarily overloaded."
};
}
if (error.name === "AbortError" || error.code === "ETIMEDOUT") {
return {
category: "timeout",
action: context.hasSideEffects ? "fail_closed" : "retry",
retryable: !context.hasSideEffects,
confidence: "medium",
logLevel: "warn",
runbook: "llm-timeouts",
reason: "Request timed out before a reliable result was received."
};
}
if (status >= 500) {
return {
category: "provider_5xx",
action: "retry_with_jitter",
retryable: true,
confidence: "medium",
logLevel: "error",
runbook: "llm-provider-5xx",
reason: "Provider returned a server-side error."
};
}
if (status >= 400) {
return {
category: "bad_request_or_auth",
action: "fail_fast",
retryable: false,
confidence: "medium",
logLevel: "error",
runbook: "llm-client-errors",
reason: "Client-side request, auth, or configuration error."
};
}
return {
category: "unknown",
action: "alert_human",
retryable: false,
confidence: "low",
logLevel: "error",
runbook: "llm-unknown-failures",
reason: "Could not classify LLM failure."
};
}
function normalizeHeaders(headers) {
const result = {};
for (const [key, value] of Object.entries(headers)) {
result[key.toLowerCase()] = Array.isArray(value) ? value.join(",") : String(value);
}
return result;
}
function parseRetryAfter(value) {
if (!value) return undefined;
const seconds = Number(value);
if (Number.isFinite(seconds)) {
return seconds * 1000;
}
const date = Date.parse(value);
if (Number.isFinite(date)) {
return Math.max(0, date - Date.now());
}
return undefined;
}
The exact categories will vary by app.
The useful part is not the specific code. It is the habit of turning provider errors into internal operational decisions.
Log the policy decision, not just the error
During an incident, a raw stack trace is rarely enough.
I want one log event that answers:
- which provider failed?
- which model was used?
- what status code came back?
- what raw response headers mattered?
- how many input and output tokens were estimated?
- what failure category did we assign?
- what action did the system take?
- which runbook should a human open?
A log line like this is much more useful than "LLM request failed":
logger.warn("llm_request_failed", {
provider: "example-provider",
model: "example-model",
status: 429,
category: decision.category,
action: decision.action,
confidence: decision.confidence,
retryable: decision.retryable,
retryAfterMs: decision.retryAfterMs,
runbook: decision.runbook,
estimatedInputTokens: context.estimatedInputTokens,
estimatedOutputTokens: context.estimatedOutputTokens,
requestId: context.requestId
});
The goal is simple: if someone is on call, they should not need 20 minutes to figure out whether this was a request-rate issue, token-rate issue, provider overload, timeout, or bad config.
If the first useful question during an incident is "what kind of failure is this?", your logs should answer that directly.
Do not retry every retryable error the same way
This is where a lot of LLM API code gets too optimistic.
"Retryable" does not mean "retry immediately."
A token-per-minute limit, request-per-minute limit, timeout, and provider overload may all be retryable, but they should not share the same retry behavior.
For example:
rate_limit_requests: retry after the reset window, possibly with queueing
rate_limit_tokens: reduce token usage, queue, or switch to a smaller request
provider_overloaded: retry with jitter, but avoid synchronized retry storms
timeout: retry only if the operation is safe to repeat
stream_interrupted: decide whether partial output is usable or must be discarded
tool_call_uncertain: do not blindly execute side effects twice
That last one matters a lot for agentic workflows.
If the LLM call was only generating text, a retry is usually fine.
If the LLM call was planning or triggering an external action, retrying can accidentally duplicate real work. Sending two emails, creating two tickets, charging twice, or calling the same webhook twice is a very different failure mode from "the answer took too long."
Keep the runbook close to the code
I do not think the whole incident runbook belongs inside the codebase.
Human context still matters.
But the first decision should be encoded near the client.
A practical compromise is:
- code classifies the failure
- code chooses the first action
- logs include the runbook ID
- the runbook explains investigation steps and business context
For example:
{
category: "rate_limit_tokens",
action: "queue",
runbook: "llm-rate-limit-tokens"
}
Then the runbook can explain:
- where to check token usage
- which dashboards matter
- whether to reduce max output tokens
- whether fallback models are approved
- who owns quota increases
- what user-facing degradation is acceptable
That keeps institutional knowledge closer to the place where the incident starts, without turning your application code into a giant wiki page.
The checklist I use now
Before shipping a new LLM provider or model path, I try to answer these:
- Do we log raw provider error metadata?
- Can we distinguish request limits from token limits?
- Do we classify overload separately from rate limits?
- Are timeout retries safe for this workflow?
- Do streaming failures preserve partial state intentionally?
- Do tool calls have idempotency keys or operation IDs?
- Does every failure category have a first action?
- Does every common category point to a short runbook?
- Can someone on call understand the failure from one log event?
If the answer is no, the integration may still work in development, but it is probably not ready for production traffic.
Final thought
LLM APIs look like normal APIs until they fail under real workload.
Then the hard part is not catching the exception. The hard part is knowing whether to retry, slow down, queue, fallback, fail closed, or wake someone up.
That decision should not live only in a forgotten runbook.
Put a small failure policy next to your LLM client.
Your future on-call self will be a lot less annoyed.
I work on TokenBay, so I spend a lot of time thinking about multi-model API behavior, routing, and failure handling. This post is less about any one provider and more about the production layer I wish more LLM apps had from day one.