Mantis · GitHub
" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
README.md
Mantis LLM Gateway
Mantis is an open-source, self-hosted LLM gateway for teams building applications across multiple model targets. It gives client applications one stable chat-completions API while centralizing routing policy, failover behavior, response caching, guardrails, observability, and AWS deployment configuration.
The project is designed for small teams that want the benefits of an LLM gateway without giving up control of their infrastructure or data.
What Mantis Provides
One API for LLM calls: send chat-completion requests through a single gateway endpoint instead of integrating directly with each provider.
Configurable routing: route by metadata, model aliases, weighted targets, fallback chains, retries, timeouts, and cooldowns.
Response caching: reduce repeated LLM calls with exact prompt caching and optional semantic caching.
Guardrails: use AWS Bedrock guardrails to mask sensitive data and block policy-violating prompts or responses.
Observability: capture request IDs, latency, token usage, cache behavior, errors, and request outcomes through CloudWatch.
AWS-native deployment: provision and run Mantis with Terraform, ECS Fargate, ALB, ElastiCache, Parameter Store, S3, IAM, and CloudWatch.
Repositories
llm-gateway: the FastAPI gateway service, React configuration dashboard, Terraform infrastructure, and deployment scripts.
mantis-sdk: a Python SDK for calling the Mantis /v1/chat/completions endpoint from application code.
mantis-llm-gateway.github.io: the public documentation site and case study.
Start Here
Read the documentation for the project overview, guides, API reference, and architecture case study.
Follow the quick start to run or deploy the gateway.
Review the routing configuration guide to understand how model selection, fallback, caching, and cooldown behavior are controlled.
Project Goals
Mantis exists to make multi-LLM application development more reliable, observable, and operationally manageable. Instead of spreading provider-specific logic across application code, teams can put model routing, cache policy, failover behavior, guardrails, and deployment concerns behind one gateway layer.
The result is a system where application code stays simple, model choices remain configurable, and teams keep control over how requests move through their own AWS environment.
Popular repositories
Loading
llm-gateway<br>llm-gateway<br>Public
An LLM Gateway
Python
12
mantis-sdk<br>mantis-sdk<br>Public
SDK
Python
mantis-llm-gateway.github.io<br>mantis-llm-gateway.github.io<br>Public
HTML
.github<br>.github<br>Public
Organization description
Repositories
-->
Loading
Type
All
Public
Sources
Forks
Archived
Mirrors
Templates
Language
All
HTML
Python
Sort
Last updated
Name
Stars
Showing 4 of 4 repositories
mantis-llm-gateway.github.io
Public
Uh oh!
There was an error while loading. Please reload this page.
mantis-llm-gateway/mantis-llm-gateway.github.io’s past year of commit activity
HTML
Updated Jun 26, 2026
llm-gateway
Public
An LLM Gateway
Uh oh!
There was an error while loading. Please reload this page.
mantis-llm-gateway/llm-gateway’s past year of commit activity
Python
12
Apache-2.0
Updated Jun 25, 2026
.github
Public
Organization description
Uh oh!
There was an error while loading. Please reload this page.
mantis-llm-gateway/.github’s past year of commit activity
Updated Jun 23, 2026
mantis-sdk
Public
SDK
Uh oh!
There was an error while loading. Please reload this page.
mantis-llm-gateway/mantis-sdk’s past year of commit activity
Python
Updated Jun 19, 2026
People
This organization has no public members. You must be a member to see who’s a part of this organization.
Top languages
Loading…
Uh oh!
There was an error while loading. Please reload this page.
Most used topics
Loading…
Uh oh!
There was an error while loading. Please reload this page.
You can’t perform that action at this time.