How I Built a 5-Model LLM Fallback Router, Archived 2026

Archive note, August 2026: This page records a May 2026 AutoKaam routing experiment. The named providers, fallback order, prices, quotas, and observed availability below are historical records, not the active AutoKaam configuration or a recommendation to use those exact models. Keep the engineering lessons, then evaluate a current provider contract for your own workload.

In May 2026, every automated job in my empire imported one router instead of calling a model directly. Content critics, nightly fanout, and JSON extractors called chat(system=..., user=..., profile="json"), then the router selected a provider. I built that experiment after several low-cost providers retired models or started returning 429s during bursts. The configuration is no longer an active AutoKaam deployment, but the failure modes are still worth documenting for anyone who is considering unattended LLM automation.

Why a router, not a single API

A single API key is a single point of failure. That matters most when a background job cannot wait for a human retry. Two events in the May 2026 experiment made the case more clearly than a design diagram could.

On 16 May 2026, I removed two providers from the chain in one sitting. Ring 2.6 retired its OpenRouter free tier, and the smoke check returned no longer available as a free model. The same day, I removed the Nous Qwen 3.6 family after it drained a subscription bucket in eight hours. Had either been the only model, the automation would have stopped until I noticed and rewrote the caller. Because they were isolated tiers, the historical router fell through and downstream work continued.

The durable lesson is narrower than "always use five models": decide what failure a caller may tolerate before routing production work. A low-cost model can be a candidate when a retry, a second provider, or a human review keeps the failure from silently becoming user-facing output. A task that needs one exact model, tokenizer, or tool behavior needs a different contract.

The May 2026 fallback chain

The archived router used several profiles because a low-latency tool call and a long-form draft had different requirements. This table is a historical snapshot, not an active provider menu.

profile	archived order	intended use case
judgment	DeepSeek-V4-Flash, Azure gpt-5.4-mini, Cerebras	critique and contradiction finding
json	DeepSeek, Azure gpt-5.4-mini, Cerebras	structured extraction
literary	DeepSeek, Owl Alpha, Azure Kimi K2.6, Cerebras	long-form prose
research	Owl Alpha, DeepSeek, Azure Kimi	long-context work
bulk_prose	DeepSeek, Owl Alpha, Cerebras, Azure gpt-5.4-mini	overnight fanout
fast	Laguna XS.2, DeepSeek, Cerebras, Azure gpt-5.4-mini	low-latency calls
code	DeepSeek, Owl Alpha, Azure	repository work

The order was a cost-and-reliability ladder, not a quality ranking. Before the experiment admitted a low-cost tier, I used a fixed battery of checkable tasks to see whether it met the lane's acceptance rule. That test-first habit remains more useful than the historical model names.

The specific provider details aged quickly. The old article described DeepSeek as the first cheap tier, two OpenRouter free models as backups, Azure as a credit-covered safety net, and Cerebras as a last resort. Those statements describe the May snapshot only. They do not establish current availability, quota, billing, or a current AutoKaam routing order.

The routing pattern that survived the archive

The useful core is a list of candidate callables per profile, a strict definition of a usable response, and an explicit terminal error. A 200 OK with an empty body should not be treated as success.

def chat(system, user, profile="judgment", max_tokens=4096, json_mode=False):
    for candidate in PROFILE_CHAIN[profile]:
        try:
            text = candidate(system, user, max_tokens, json_mode)
            if text and text.strip():
                return postprocess(text)
            logger.warning("empty response from %s", candidate.name)
        except (RateLimited, ServerError, Timeout, JSONDecodeError) as error:
            logger.warning("candidate failed: %s", error)
    raise AllCandidatesDown(profile)

The historical router treated these conditions as candidate failures:

• HTTP 402, 429, or 500 through 504
• An empty content field
• A request timeout
• A JSON-decode failure on the response body

For a new implementation, make the retry policy deliberate. Some errors should fail closed because changing the model changes the result; others can move to a separate candidate because the output is independently validated. Record the candidate identity, latency, status, retry count, and validation result so that an apparent quality regression can be traced to a real request rather than guessed from a dashboard.

A smoke command was also valuable, but its job was limited: it checked that each archived candidate could return a non-empty response. A green smoke result did not prove quality, current quota, privacy terms, or capacity. Those are separate gates and should be tested with the full prompt and the current vendor contract.

A historical empty-response failure

The nastiest May 2026 failure was not an outage. A reasoning model returned 200 OK with no final content. In that archived case, a large reasoning_content block consumed the token budget before the final prose was emitted. During a prose bakeoff on 5 May 2026, a request with 1,130 prompt tokens and a 16,000-token completion limit returned zero characters in content and 66,020 characters in the reasoning field. It took about ten minutes to establish that the output was budget-starved rather than missing.

I removed the old model-specific workaround from this guide because it would now bind a reader to a retired configuration. The durable safeguards are simpler:

1. Treat an empty final response as a failed request even when the HTTP status is successful.
2. Test long generation with the real prompt, target length, and output budget, not a toy prompt.
3. Prefer a clean retry or a clear failure over attempting to salvage hidden reasoning text into user-facing output.

This is why response validation belongs next to transport validation. A client that watches only status codes can pass an empty string downstream and make the resulting incident much harder to diagnose.

Cost planning without an expired price card

The May price table has been removed because model prices, free credits, rate limits, and currency assumptions change too quickly to serve as current buying advice. The cost model behind it is still useful:

input to measure	why it matters
Input and output tokens per accepted result	Price cards charge different rates and long outputs can dominate cost.
Retry rate and invalid-output rate	A nominally cheap model stops being cheap when validation repeatedly fails.
Cache hit rate	Stable prompts can make a larger difference than switching providers.
Fallback frequency	A chain may quietly move work to a more expensive candidate during a provider incident.
Human review cost	A low API bill is not a saving if it creates reader-facing rework.

Run that worksheet against representative traffic before changing a default. For a batch task with an objective validator, a lower-cost candidate may earn a lane. For a high-stakes or reader-facing task, reproducibility and review can outweigh a token-price difference. The right decision comes from accepted work, not headline price per million tokens.

When not to build a fallback router

Skip this pattern when one application makes a small number of calls and a human can safely retry an occasional failure. Profiles, smoke checks, validation, logging, and incident playbooks are operational overhead.

Skip it too when the caller requires one model's exact behavior, a fixed tokenizer, a specific tool contract, or reproducible output. A router trades model-identity guarantees for availability. If that trade would corrupt the result, use one pinned model with clear health checks instead. The same principle applies to choosing a backend: match the machinery to the actual failure mode and load.

Related

• Where a cheap DeepSeek API fit in an earlier routing stack
• The GPT-5.5, Claude 4.7, Gemini 3 router playbook for May 2026
• Running edge inference on Cloudflare Workers AI
• PocketBase vs Supabase in 2026 for bootstrapped founders