GitHub - Liana64/eek: Tiny LLM proxy · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Liana64
eek
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>29 Commits<br>29 Commits
.github/workflows
.github/workflows
src
src
.gitignore
.gitignore
Cargo.lock
Cargo.lock
Cargo.toml
Cargo.toml
LICENSE-APACHE
LICENSE-APACHE
LICENSE-MIT
LICENSE-MIT
README.md
README.md
config.example.toml
config.example.toml
flake.lock
flake.lock
flake.nix
flake.nix
View all files
Repository files navigation
Tiny, like a mouse, key-brokering proxy for LLM providers
I wrote this while looking for a lightweight version of litellm, and after much<br>frustration at all of the npm projects with huge blankets (quilts, even) of<br>dependencies. Eek! isn't the most comprehensive solution, but it will get you<br>quick access to open models without the need to worry about auditing thousands<br>and thousands of lines of code.
Eek! is configurable using TOML/ENV and limited to chat/completions, messages,<br>and responses with upstream requests over TLS. Inbound requests to the proxy are<br>cleartext, so deploy on kubernetes with TLS-terminating ingress!
There are no retries, rate limits, budgets, load balancing, or metrics. Bodies<br>cap at 32MB, connections at 4096, and stalled upstream connections die on idle<br>timeout. SIGTERM drains for 30s. Gateway keys compare in constant time and<br>proxy-* headers never touch the provider.
Dependencies: hyper, rustls, tokio, and serde.
Demo
Deployment
eek config.toml # or EEK_CONFIG=config.toml
nix run github:Liana64/eek \<br>-- config.toml
docker run -p 8551:8551 \<br>-v ./config.toml:/config.toml \<br>-e EEK_CONFIG=/config.toml \<br>ghcr.io/liana64/eek
Config
See config.example.toml
[providers.anthropic]<br>base_url = "https://api.anthropic.com" # https only<br>auth_header = "x-api-key" # optional; default "authorization"<br>api_key = "${ANTHROPIC_API_KEY}"<br>protocol = "anthropic" # optional; enables translation (see below)">listen = "127.0.0.1:8551"<br>gateway_keys = ["${GATEWAY_KEY}"] # clients send Authorization: Bearer
[providers.anthropic]<br>base_url = "https://api.anthropic.com" # https only<br>auth_header = "x-api-key" # optional; default "authorization"<br>api_key = "${ANTHROPIC_API_KEY}"<br>protocol = "anthropic" # optional; enables translation (see below)
Gateway key(s) can be 16 to 256 characters. We recommend openssl rand -hex 32.
Routes
POST //v1/{chat/completions,messages,responses} and eek! swaps<br>your gateway key for the provider's key and forwards the request.
GET /healthz returns ok.
Translation
Easily translate between anthropic (/v1/messages) and openai<br>(/v1/chat/completions) protocols.
License
MIT or Apache-2.0
About<br>Tiny LLM proxy<br>Topics<br>ai-gatewayrust<br>Resources<br>Readme<br>Apache-2.0, MIT licenses found<br>Activity<br>Stars<br>0 stars<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository
Packages
Contributors
Languages
You can’t perform that action at this time.