GitHub - mohamedjaha/ai-router · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
mohamedjaha
ai-router
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>6 Commits<br>6 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>ai_router
ai_router
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
pyproject.toml
pyproject.toml
View all files
Repository files navigation
AI Router CLI
A BYOK, CLI-first, local-only AI router that pools the free daily quotas of<br>multiple AI providers (Groq, Cerebras, Mistral, Gemini, Cloudflare Workers AI)<br>behind a single command-line tool, with health/quota tracking, circuit-breaking,<br>and automatic failover.
Status: V1 MVP.
Install
pip install -e .
Commands
Run ai-router --help to list all commands. The CLI exposes the following<br>top-level commands (plus the config group):
Command<br>Description
ai-router config<br>Manage provider credentials and models (group).
ai-router chat<br>Start an interactive chat session routed across configured providers.
ai-router providers<br>Show which providers are supported and which are configured.
ai-router status<br>Combined health + quota snapshot for every provider/model.
ai-router health<br>Detailed health view: status, consecutive failures, latency, breaker.
ai-router quota<br>Detailed quota/rate-limit view per provider/model.
ai-router metrics<br>Read-only totals and per-provider request/fallback metrics.
ai-router serve<br>Start a local-only OpenAI-compatible HTTP server (post-V1 web layer).
ai-router config
Manage provider credentials and models. Supported providers:<br>groq, cerebras, mistral, gemini, cloudflare.
# Interactively add/overwrite a provider (API key is hidden on input).<br># Cloudflare additionally prompts for an Account ID.<br>ai-router config add groq<br>ai-router config add cloudflare
# List configured providers with masked API keys.<br>ai-router config list
# Remove a provider's configuration.<br>ai-router config remove groq
ai-router chat
Start an interactive chat session. Requests are routed across the configured,<br>eligible providers with automatic failover. Type your message and press Enter;<br>send an empty line or Ctrl+C to exit.
ai-router chat
ai-router providers
Show all supported providers and whether each is configured locally, with the<br>configured model name.
ai-router providers
Example output:
Provider Supported Configured Model<br>groq yes yes openai/gpt-oss-120b<br>cerebras yes no<br>mistral yes yes mistral-small-latest<br>gemini yes yes gemini-3.6-flash<br>cloudflare yes no
ai-router status
Combined health + quota snapshot for every configured provider/model.
ai-router status
ai-router health
Detailed health view: health score, health status, consecutive failures,<br>average latency, and circuit-breaker state.
ai-router health
ai-router quota
Detailed quota/rate-limit view per provider/model: quota status, whether quota<br>is available, requests/tokens used today, and any retry/reset times.
ai-router quota
ai-router metrics
Read-only request metrics: total requests, successes, failures, and fallback<br>events, both as totals and broken down per provider. This command never mutates<br>provider state.
ai-router metrics
ai-router serve
Start a local-only OpenAI-compatible HTTP server. It exposes the router's<br>route() core over HTTP so any OpenAI-compatible client (e.g. LangChain's<br>ChatOpenAI) can use the pooled providers as a drop-in backend.
# Bind to 127.0.0.1:8080 (default). Keep it local-only.<br>ai-router serve
# Custom port (e.g. to match a client's base_url).<br>ai-router serve --port 8081
Endpoints (all bind to 127.0.0.1 only — no remote access, no auth):
POST /v1/chat/completions (alias: POST /chat/completions)
GET /v1/models (alias: GET /models)
GET /healthz
Example — drive the router from LangChain:
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(base_url="http://127.0.0.1:8080", api_key="not-needed")<br>print(llm.invoke("Hello! Explain what an API is in simple words."))
The server is local-only by design (AGENTS.md §8): it must never be exposed<br>beyond 127.0.0.1.
How it works
Provider selection filters eligible (configured + healthy + available)<br>providers, then picks one uniformly at random.
Failover retries the next eligible provider on recoverable errors<br>(e.g. 429, 503, timeouts, connection errors), respecting the circuit<br>breaker and quota state.
State (health, quota, breaker, metrics) is persisted locally to<br>state.json (git-ignored). Conversation history is in-memory only.
Disclaimer
You are responsible for complying with each provider's...