GitHub - dburnett11155-rgb/Tokencompress · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
dburnett11155-rgb
Tokencompress
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>2 Commits<br>2 Commits
Makefile
Makefile
README.md
README.md
compressed.json
compressed.json
go.mod
go.mod
mcp_client.py
mcp_client.py
tc
tc
tokencompress
tokencompress
tokencompress.conf
tokencompress.conf
tokencompress.go
tokencompress.go
tokencompress_test.go
tokencompress_test.go
View all files
Repository files navigation
⚡ tokencompress (v1.1.0)
A zero-dependency, sub-millisecond Go CLI and MCP sidecar that prunes raw tool outputs (JSON, terminal logs, HTML) before they enter your AI agent's context window.
Cut LLM context token consumption by 60% to 80% without using a second LLM summarization turn.
💡 The Problem
When AI agents execute tools (calling APIs, running terminal commands, or scraping web pages), they receive thousands of lines of raw, unparsed data:
Massive JSON Payloads: A 1,000-item array floods the prompt with 15,000+ tokens.
Verbose Stack Traces: Framework noise and node_modules paths drown out the root exception.
Raw HTML: CSS, scripts, and navigation menus bloat the context window.
This leads to high API costs, slower response times, and context rot —where agents hallucinate or repeat tool calls mid-task.
⚡ The Solution
acts as a high-speed, deterministic filter between your tools and your model:
JSON Truncation: Keeps representative schema examples and replaces redundant array items with metadata (: N).
Log Pruning: Strips internal framework paths and returns only the root error message, user file paths, and execution lines.
HTML Cleaning: Strips scripts, styles, and navigation elements, converting content to readable text/markdown.
Duplicate Loop Detection: Hashes tool outputs per session and prepends a warning header if an agent receives duplicate tool results twice in a row.
🚀 Quickstart
1. Installation
make build<br>sudo mv tokencompress /usr/local/bin/
2. CLI Pipe Mode
Compress a large JSON response
cat large_response.json | tokencompress --mode json
Prune a verbose stack trace log
cat app.log | tokencompress --mode log --log-internal-marker "mycompany/internal"
Clean raw HTML
curl -s https://example.com | tokencompress --mode html
3. MCP (Model Context Protocol) Integration
Add directly to your Claude Desktop config ():
"mcpServers": {<br>"tokencompress": {<br>"command": "/usr/local/bin/tokencompress",<br>"args": ["--mode", "mcp"]
📊 Benchmarks
Input Payload<br>Raw Token Count<br>Compressed Token Count<br>Token Reduction<br>Execution Time
JSON Array (500 items)<br>~12,500 tokens<br>~850 tokens<br>93.2%
Python Stack Trace<br>~3,200 tokens<br>~410 tokens<br>87.1%
HTML Web Scrape<br>~18,000 tokens<br>~2,100 tokens<br>88.3%
💼 Custom Integrations & Consulting
Need a custom high-performance MCP proxy, specialized parsers for enterprise tools, or custom token-optimization setups for your agent infrastructure?
Custom Setup Gigs: 50 - 00 per custom integration setup.
Custom Log/AST Parsers: 50 per custom domain parser.
Contact: Open an issue on this repository or contact directly via GitHub profile.
📜 License
MIT License. Free to use in open-source and commercial agent setups.
About<br>No description, website, or topics provided.<br>Resources<br>Readme<br>Activity<br>Stars<br>0 stars<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.