GitHub - Oaklight/llm-comply: Multi-format LLM API compliance testing tool — validates endpoints against OpenAI Chat, Open Responses, Anthropic, and Google GenAI specs · GitHub
/" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Oaklight
llm-comply
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
master
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>36 Commits<br>36 Commits
.github/workflows
.github/workflows
src/llm_comply
src/llm_comply
.gitignore
.gitignore
.pre-commit-config.yaml
.pre-commit-config.yaml
CHANGELOG.md
CHANGELOG.md
Dockerfile
Dockerfile
LICENSE
LICENSE
Makefile
Makefile
README.md
README.md
pyproject.toml
pyproject.toml
View all files
Repository files navigation
llm-comply
Multi-format LLM API compliance testing tool — validates endpoints against official specs for OpenAI Chat Completions , Open Responses , Anthropic Messages , and Google GenAI .
Try it online →
Install
pip install llm-comply<br># For colored terminal output:<br>pip install llm-comply[rich]
Quick Start
# Open Responses (default)<br>llm-comply -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini
# OpenAI Chat Completions<br>llm-comply --format openai-chat -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini
# Anthropic Messages<br>llm-comply --format anthropic -u https://api.anthropic.com/v1 -k $KEY -m claude-haiku-4-5 \<br>--auth-header x-api-key --no-bearer -H anthropic-version:2023-06-01
# Google GenAI<br>llm-comply --format google-genai -u https://generativelanguage.googleapis.com -k $KEY \<br>-m gemini-2.5-flash --auth-header x-goog-api-key --no-bearer
Options
-u, --base-url URL API base URL (required)<br>-k, --api-key KEY API key (or set OPENRESPONSES_API_KEY env)<br>-m, --model MODEL Model name (default: gpt-4o-mini)<br>--format FORMAT API format: open-responses, openai-chat, anthropic, google-genai<br>-f, --filter IDS Comma-separated test IDs to run<br>-i, --ignore PATTERNS Ignore errors matching substrings (e.g. refusal,verbosity)<br>-H, --header K:V Extra headers (e.g. anthropic-version:2023-06-01)<br>--auth-header NAME Auth header name (default: Authorization)<br>--no-bearer Don't prepend "Bearer " to API key<br>-v, --verbose Show request/response on failure<br>--json JSON output for CI<br>--list List available test IDs
Test Coverage
Format<br>Tests<br>What's Validated
Open Responses<br>12<br>Schema + semantic (lifecycle, phases, compaction)
OpenAI Chat<br>Schema + semantic (choices, finish_reason, delta)
Anthropic Messages<br>Schema + semantic (content blocks, stop_reason)
Google GenAI<br>Semantic only (candidates, parts, finishReason)
License
MIT
About<br>Multi-format LLM API compliance testing tool — validates endpoints against OpenAI Chat, Open Responses, Anthropic, and Google GenAI specs<br>llm-comply.service.oaklight.top<br>Resources<br>Readme<br>MIT license<br>Activity<br>Stars<br>0 stars<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository
Releases
Packages
Used by
Contributors
Languages
You can’t perform that action at this time.