Add TY25 Kimi K3 support via OpenRouter by michaelrbock · Pull Request #93 · column-tax/tax-calc-bench · GitHub
//voltron/pull_requests_fragments/pull_request_layout" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//voltron/pull_requests_fragments/pull_request_layout;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
column-tax
tax-calc-bench
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>23
Star<br>102
Merged<br>Add TY25 Kimi K3 support via OpenRouter#93michaelrbock merged 1 commit intomaincolumn-tax/tax-calc-bench:mainfrom codex/ty25-kimi-k3-supportcolumn-tax/tax-calc-bench:codex/ty25-kimi-k3-supportCopy head branch name to clipboard
Conversation
michaelrbock
commented
Jul 20, 2026
Copy link
Copy Markdown
Contributor
Summary
Adds TY25 support for moonshotai/kimi-k3 through OpenRouter using OPENROUTER_API_KEY.
Includes Kimi K3 in default TY25 runs and restricts it to ultrathink, mapped to its required reasoning_effort="max".
Sends the original PDFs as raw base64 file blocks for OpenRouter’s default PDF processing, without local text extraction or a parser override.
Preserves slash-containing model IDs and calls LiteLLM with the full openrouter/moonshotai/kimi-k3 model name.
Keeps Kimi out of TY25 web-search runs and rejects unsupported lower thinking levels.
Adds saved model outputs and evaluation reports for all 50 TY25 cases.
Updates the README leaderboard, detailed results, documentation, and regenerated TY25 charts.
Results
Model<br>Thinking<br>Tests<br>Strict<br>Lenient<br>By line<br>By line, lenient
moonshotai/kimi-k3<br>ultrathink<br>50×1/50<br>6.00%<br>12.00%<br>65.04%<br>68.43%
The saved artifacts contain one model output and one evaluation report for every TY25 case.
Implementation notes
Uses reasoning_effort="max", max_tokens=131072, streaming, and the existing four-hour timeout and finish-reason checks.
Omits sampling arguments, parser plugins, and web-search configuration.
Retains LiteLLM 1.89.3. A regression test verifies that its generic OpenRouter adapter preserves the nested model slug, PDF file block, and explicitly allowed reasoning parameter without requiring static Kimi model metadata.
Documents OpenRouter PDF processing, possible OCR charges, and the current upstream capacity warning.
Validation
uv run pytest — 142 passed
uv run ruff check tax_calc_bench tests
uv run scripts/update_charts.py
git diff --check
Verified 50 model outputs and 50 evaluation reports
Visually verified the regenerated TY25 overall and detailed charts
-->
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
-->
All reactions
Add TY25 Kimi K3 support via OpenRouter
5926c4a
michaelrbock
marked this pull request as ready for review
July 20, 2026 19:19
michaelrbock
requested a review<br>from KaraMolisee
July 20, 2026 19:19
KaraMolisee
approved these changes
Jul 20, 2026
View reviewed changes
Hide details<br>View details
michaelrbock
merged commit c1e99a4<br>into
main
Jul 21, 2026
1 check passed
Uh oh!
There was an error while loading. Please reload this page.
michaelrbock
deleted the
codex/ty25-kimi-k3-support
branch
July 21, 2026 02:00
-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.<br>Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free<br>to join this conversation on GitHub .<br>Already have an account?<br>Sign in to comment
-->
Reviewers
KaraMolisee
KaraMolisee approved these changes
-->
Assignees
No one assigned
Labels
None yet
-->
Milestone
No milestone
-->
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one...