AI model router selection for developers: compare fallback, logs, cost controls, and self-hosting

Last checked: 2026-07-15. LiteLLM is a self-hostable gateway and Python SDK that advertises a unified OpenAI-format interface for 100+ LLMs. Portkey combines an MIT-licensed open-source gateway with a managed cloud: Developer is Free Forever with 10K recorded logs/month, while Production is $49/month with 100K logs/month and $9 overage per additional 100K requests. LiteLLM is the stronger fit when a Python team wants routing in code or owns a self-hosted proxy; Portkey is the clearer managed starting point when hosted logs and conditional routing justify $49/month.

LiteLLM vs Portkey current official snapshot

The LiteLLM repository and routing docs support the LiteLLM rows. The Portkey Gateway repository, documentation, and pricing page support the Portkey rows. Provider/model counts are vendor scopes, not an independent compatibility benchmark.

Direct choice by team shape

• Python application team: LiteLLM SDK/Router keeps fallback and cost tracking close to application code.
• Central self-hosted gateway team: compare LiteLLM Proxy and Portkey's open-source gateway on authentication, state, upgrades, high availability and logs; LiteLLM exposes virtual-key and spend-management surfaces directly in its official project.
• Managed production team: Portkey's verified $49/month Production tier provides a concrete hosted entry point with 100K recorded logs/month.
• One-provider startup: use neither until centralized policy, a second provider, fallback or shared observability creates enough value to justify another dependency.

Routing log schema

Require one record per application request:

{
  "request_id": "app-20260715-0042",
  "router_version": "recorded-version",
  "route_rule": "support-summary-primary",
  "attempts": [
    {
      "provider": "provider-a",
      "model": "model-a",
      "status": 503,
      "latency_ms": 840,
      "input_tokens": 0,
      "output_tokens": 0,
      "cost": 0
    },
    {
      "provider": "provider-b",
      "model": "model-b",
      "status": 200,
      "latency_ms": 1420,
      "input_tokens": 620,
      "output_tokens": 110,
      "cost": "provider-priced amount"
    }
  ],
  "final_provider": "provider-b",
  "fallback_reason": "primary_503",
  "schema_valid": true,
  "budget_action": "allow"
}

Do not log prompts containing secrets or customer data by default. The schema is for routing evidence, not a reason to duplicate sensitive content.

Forced-provider-failure experiment

A four-person SaaS team summarizes support tickets with two providers. Prepare 60 sanitized requests with a fixed output schema. Set a $3 test cap. Requests 1-20 use a healthy primary. Requests 21-40 force the primary deployment to return a controlled 503 or point to a test deployment that fails. Requests 41-60 restore the primary.

For each request, record route rule, every attempt, error, latency, provider/model, token usage, estimated cost, final schema validity and whether the response was accepted by the application.

Acceptance:

• at least 19 of 20 forced failures reach the intended fallback;
• the application accepts no duplicate final response;
• all 60 requests have complete route and provider attribution;
• retries remain within the configured maximum;
• fallback output passes the same JSON schema;
• p95 latency during recovery stays below the team's predeclared threshold;
• total estimated/router-recorded spend stays below $3 and reconciles with provider usage.

The exact p95 threshold belongs to the product. Declare it before the run. Changing it after seeing results turns evaluation into storytelling.

Important failure samples

Fallback loop: primary fails, fallback has a configuration error, and the router cycles back to primary. A request consumes multiple attempts until a global timeout. Stop the test; fallback must be acyclic and bounded.

Schema drift: provider A returns the expected JSON, provider B returns prose. The router reports success because HTTP status is 200. The application must treat schema validation as part of recovery, not as a separate cosmetic check.

Missing attribution: the final response works, but the log records only the logical model alias. The team cannot reconcile provider invoice or incident. That fails observability.

Budget bypass: retries or fallback usage are not counted toward the spend cap. The router continues after the expected $3. That fails control even if every user gets an answer.

LiteLLM or Portkey: three team profiles

Python team that wants router logic close to application code

Start by evaluating LiteLLM's Python SDK and Router. Its official project describes direct library integration, retries/fallbacks, load balancing, cost tracking and observability callbacks. The tradeoff is that routing behavior becomes part of the application runtime and upgrade surface. Test whether the team can centralize configuration enough to avoid different services inventing different rules.

Platform team that wants a central gateway control plane

Evaluate both gateways as services. LiteLLM documents a proxy with virtual keys, spend management and gateway controls. Portkey emphasizes routing configs, fallbacks, conditional routing, analytics and gateway operation. Compare authentication, config review, audit fields, high availability, state stores, secret handling and deployment ownership rather than dashboard appearance.

Small product team with only one provider today

Do not add a router just because multi-model systems are popular. First use a thin provider adapter and complete request logging. Add a gateway when a second provider, spend policy, centralized keys, consistent observability or real fallback requirement creates measurable value. A router is another production dependency.

Self-hosting checklist

If self-hosting, identify who owns upgrades, database/state, encryption, provider secrets, config review, high availability, metrics, backup and incident response. Pin and record the tested version. Run the forced-failure suite after upgrades. Keep a bypass or rollback path so a gateway incident does not make every model provider unreachable.

Pause conditions

Pause when fallback loops, provider/model attribution is missing, retry counts are unbounded, schema changes across routes, sensitive prompts leak into logs, budget controls cannot stop traffic, or the gateway adds more operational risk than the current adapter. Also pause if official documentation cannot confirm the exact provider feature you need.

FAQ

Do I need an AI model router for one provider?

Usually not. A provider adapter plus good logs is simpler. Add routing when multiple deployments, centralized policy, fallback, spend controls or shared observability justify it.

Is fallback successful when the HTTP response is 200?

Not necessarily. Validate the output schema, safety/policy requirements, provider attribution, latency and cost before accepting the response.

Should I select by supported model count?

No. Test your exact providers, parameters, streaming/tool behavior, error classes and usage fields. Vendor counts are not an application compatibility test.

What is the first router test?

Force the primary to fail for a bounded request set, then verify fallback rate, loops, schema, logs, latency, cost and spend-cap behavior.

Related reading

• Qwen API pricing and access for builders: verify the model, region, and real test cost
• MiniMax long-context API evaluation: test evidence retrieval before trusting the window size
• GLM coding API evaluation for builders: test one repository patch with diff, tests, and rollback
• Cloudflare Pay Per Crawl test guide: what publishers and AI crawlers can verify in private beta

RadarAI helps builders track AI updates, compare source-backed signals, and decide which changes are worth acting on.