LLM API Retry vs Fallback: Five Gates Before Another Model Call

Retry the same LLM API route only when the failure is probably transient, the operation is safe to replay, and another attempt still fits the workflow's time, cost, and quota budget. Switch to a fallback model only when that route has already passed the same schema, tool, safety, data, latency, cost, and quality tests. Otherwise, queue the work, return a controlled degraded response, or fail closed.

The important distinction is not “same model versus another model.” A retry assumes the same operation can succeed after a short change in time or load. A fallback changes the execution contract. It may change the provider, model behavior, context window, tool calling, output shape, safety filter, data region, price, or latency.

Start with the failure owner, not the status code

HTTP status is evidence, not a complete policy. OpenAI's current error guide separates two different 429 outcomes: request-rate pressure should be paced, while exhausted quota or spend needs a billing or limit change. Repeating the latter cannot create budget.

The same principle applies across providers:

Anthropic's current API error reference is a useful warning about hidden attempts: its official SDKs retry transient connection, rate-limit, and 5xx failures twice by default and honor retry-after when present. Gemini's troubleshooting guide likewise documents automatic transient retries in official SDKs. If your application adds “three retries” without inspecting the SDK and gateway, the real attempt count can be much larger.

Count one workflow budget across every layer

Do not give the SDK, gateway, application service, queue consumer, and job scheduler independent retry counters. Use one budget:

text
total_attempts =
  initial_request
  + SDK_internal_attempts
  + gateway_attempts
  + application_retries
  + fallback_attempts
  + queue_redeliveries

The budget also needs ceilings for elapsed time, tokens/cost, and acceptable quality change. An interactive chat might have enough time for one short retry but not a 40-second fallback chain. A nightly enrichment job can wait, but it still needs a cost ceiling and a next-attempt timestamp.

AWS's timeouts, retries, and backoff guidance explains why this matters: retries at multiple layers multiply load, and retries can make an overloaded dependency recover more slowly. Backoff and jitter spread attempts; a cap and a single retry point prevent recovery logic from becoming the incident.

Use a policy card per workflow:

yaml
workflow: support-answer
max_total_attempts: 3
max_elapsed_ms: 9000
max_incremental_cost_usd: 0.02
replay_safe_until: first_visible_token
allowed_fallbacks:
  - route: approved-support-backup
    reasons: [provider_overload, pre-stream_timeout]
queue_after_ms: null
fail_closed_on: [auth, policy, unapproved_route, uncertain_side_effect]

These numbers are examples, not universal defaults. Derive them from the user-facing SLO and test them under injected faults.

A timeout does not prove that nothing happened

Before replaying, locate the commitment boundary.

• Before the provider accepts the request, a replay may be safe.
• After the provider accepts it but before a response arrives, the outcome may be unknown.
• After the first user-visible token, a transparent model switch can duplicate or contradict the response.
• After a tool call, database write, webhook, email, or purchase-related action, a timeout does not prove the side effect failed.

For state-changing workflows, separate generation from execution. Give external actions an idempotency key or application-owned operation ID. Record whether an action was requested, started, committed, and acknowledged. If the state is uncertain, query or reconcile it; do not ask a second model to “try again.”

Streaming needs a product decision. Before the first visible event, you can reset state and use an approved route. After visible output, either surface the interruption, let the user explicitly restart, or implement a real resume protocol. Appending a fresh model's answer to a partial stream is not seamless failover.

A fallback needs an equivalence contract

“Supports chat completions” is not enough. Approve an alternate route for one workflow only after it passes all of these:

1. Input contract: context size, images/files, system instructions, locale, and request limits.
2. Output contract: JSON Schema, refusal shape, finish reasons, citations, and truncation behavior.
3. Tool contract: tool names, argument schema, parallel calls, tool-result return path, and idempotency.
4. Safety contract: moderation behavior, disallowed tasks, escalation path, and high-impact stop rules.
5. Data contract: retention, region, tenant isolation, and allowed data class.
6. Operational contract: latency distribution, streaming behavior, status visibility, and request IDs.
7. Economic contract: price units, retries, cache behavior, quota owner, and maximum spend.
8. Quality contract: the same evaluation fixtures and a workflow-specific acceptance threshold.

A smaller model may be an excellent fallback for classification but unacceptable for a customer-facing policy answer. A different provider may improve availability but fail your tool schema or data boundary. When no generative fallback passes, a cached result, deterministic rules, or a clear “try later” state is safer than an unverified answer.

Implement the policy as a state machine

The following TypeScript is deliberately route-neutral. Provider adapters normalize error evidence; the workflow policy makes the decision.

ts
type Action = "retry" | "fallback" | "queue" | "degrade" | "fail_closed";

type Failure = {
  owner: "transient" | "rate" | "quota" | "request" | "auth" | "policy" | "unknown";
  commitState: "none" | "committed" | "unknown";
  retryAfterMs?: number;
};

type Budget = {
  attemptsLeft: number;
  elapsedMsLeft: number;
  incrementalCostLeft: number;
};

function decide(
  f: Failure,
  budget: Budget,
  estimate: { retryMs: number; retryCost: number; fallbackMs: number; fallbackCost: number },
  routes: { primary: "healthy" | "degraded" | "open"; fallback: "healthy" | "unhealthy" },
  fallbackApproved: boolean,
  queueAllowed: boolean,
  degradeApproved: boolean
): Action {
  if (["request", "auth", "policy"].includes(f.owner)) return "fail_closed";
  if (f.commitState !== "none") return "fail_closed";
  if (f.owner === "unknown") return queueAllowed ? "queue" : "fail_closed";

  const canRetry =
    budget.attemptsLeft > 0 &&
    budget.elapsedMsLeft >= (f.retryAfterMs ?? 0) + estimate.retryMs &&
    budget.incrementalCostLeft >= estimate.retryCost;
  const canFallback =
    budget.attemptsLeft > 0 &&
    budget.elapsedMsLeft >= estimate.fallbackMs &&
    budget.incrementalCostLeft >= estimate.fallbackCost;

  if (canRetry && ["transient", "rate"].includes(f.owner) && routes.primary !== "open") {
    return "retry";
  }
  if (
    canFallback &&
    ["transient", "rate", "quota"].includes(f.owner) &&
    fallbackApproved &&
    routes.fallback === "healthy"
  ) return "fallback";

  if (queueAllowed) return "queue";
  return degradeApproved ? "degrade" : "fail_closed";
}

Here retryAfterMs applies only to the primary-route retry. The fallback branch uses its own expected latency and cost plus an independent quota check inside fallbackApproved; a long primary retry-after therefore does not veto a healthy fallback that still fits the workflow SLO. A committed or unknown result requires reconciliation or an explicit resume/idempotency protocol outside this automatic dispatcher. Persist an attempt record before every new call and apply capped exponential backoff with jitter only after the provider-specific error evidence identifies the failure as transient. A 500 can also be request-related—for example, current Gemini guidance lists overlong input context as one possible cause—so the HTTP code alone never sets owner: "transient".

Prove every branch in staging

Do not accept “the gateway returned 200” as resilience proof. Inject each condition and verify the observable result:

• 429 with a short retry-after: one controlled wait, no concurrency spike.
• primary retry-after longer than the remaining SLO, with an approved healthy fallback that fits its own latency and cost budget: fallback, not queue.
• 429 with insufficient_quota: no unchanged retry and no hidden budget switch.
• 503 before streaming: bounded retry, then only an approved alternate route.
• malformed tool schema: immediate application error, not failover.
• timeout after a mocked tool commit: reconciliation, not duplicate execution.
• stream failure after visible tokens: explicit interrupted state, not appended output.
• fallback response with a missing required field: output validator rejects it.
• circuit open: normal user traffic avoids the route; a bounded probe decides recovery.

Keep a row for every attempt: workflow ID, attempt index, requested route, selected route, provider request ID, error class, delay, tokens, cost, commitment state, fallback reason, and final disposition. Alert separately on primary success, retry recovery, fallback recovery, degraded responses, and fail-closed outcomes. Otherwise, a “successful” dashboard can hide a failing primary.

Choose the next path

If you already have a provider-specific 429, use the narrower guides for OpenAI API rate limits, Claude API rate limits, or Gemini API rate limits. They help prove the owner before this policy acts.

If you need to test several approved models behind one compatible endpoint, review the current LaoZhang AI API documentation in staging. A unified endpoint can simplify adapters, but it does not remove the need for workflow-specific equivalence tests, budgets, logs, and stop rules.

The acceptance condition is straightforward: every failure class produces one explainable action, every retry fits one shared budget, every fallback has passed the same workflow contract, and no final success erases the attempts that came before it.