AI Agent Coding Comparison: The Rust Leap Year Challenge Let's talk Bitcoin (and sometimes Rust & AI)
Subscribe
Let's talk Bitcoin (and sometimes Rust & AI)
Tip me via Bitcoin LN: mariusb@blink.sv
AI Agent Comparison
By Marius on July 13th, 2026
I ran a little exercise to give the following 5 AI Agents the same prompt and see how they would fare, in my opinion, in completing the task:
Claude: Sonnet 5 Medium from Antropic
ChatGPT: GPT-5.5 from OpenAI
Gemini: Gemini 3.5 Flash from Google
Qwen: Qwen 3.7-plus from Alibaba Cloud via Auto Efficient model from Kilo Code
Hy3 from Tencent via Hy3 Free model from Kilo Code
Except for the Qwen model, which used some Kilo credits (Gave each model a short exercise to produce some Rust code and see how each performs.
In the prompt, I asked the models to ask clarifying questions if there is any doubt; do not make any assumptions. Did any model ask, yes 2 did:
ChatGPT asked whether the Gregorian calendar leap year should be implemented.
Gregorian calendar
Kilo Code Auto Efficient Qwen 3.7-plus asked whether modulo (%) can be used and also whether it should just provide the code or provide the code and execute.
Yes, % can be used; just provide the code.
Note: There was no need to ask whether modulo can be used; this is an operator just like + , - , / or *
All the models produced decent code, but only 3 used the #[cfg(test)] test harness that Rust provides:
CharGPT
Claude
Kilo Code Hy3
The other 2, Gemini and Kilo Code Auto Efficient Qwen 3.7-plus, include some test code as part of the main() function with println!() output.
Even though only Kilo Code Efficient Qwen 3.7-plus asked whether % can be used, all except Kilo Code Hy3 used it. Kilo Code Hy3 implemented a remainder() function to determine a remainder of 0 after division.
If I have to pick a model based on the prompt, then I would go with either of the Kilo Code models since its output is very detailed and it explains it’s thinking and what it was going to do.
So what was the goal here?
It was to determine:
How well does the model “listen” to the prompt, did they get the task right?
Were any follow-up questions asked to clarify something the model did not understand?
Did the model make any assumptions?
How accurate was the model in solving the task?
Note, I was not interested this time at all in the cost and/or time that it took the model to solve the task.
Before we look at the result, let’s see how they. stack up on Kilo Bench:
Model<br>Code Rank<br>Coding Completion<br>Cost/Attempt
Sonnet 5 Medium<br>10<br>59.6%<br>$36.19
GPT-5.5<br>54<br>74.2%<br>$72.63
Gemini 3.5 Flash<br>57<br>64.7%<br>$104.49
Qwen 3.7-plus*<br>56<br>54.6%<br>$20.65
Hy3 (free)<br>34<br>47.6%<br>$0.00
Used the data from Qwen 3.7-max (slightly more expensive)
Prompt
Here is the prompt I gave each of the models.
Create a Rust function from first principles that returns TRUE or FALSE for a given number whether it is a leap year. Use no crates, dependencies or built-in standard traits. Do not read, write or reference any local files; just provide the code as text in a code block.
Also, create tests and test the function for the following set of numbers: 1, 4, 2023, 2024, 2025, 2048, 1900, 2100, 1600, 2000
Do not make any assumptions, and if in doubt, then ask me.
Detail
ChatGPT
What did it get right?
Asked clarifying question(s)
Made use of Rust test harness
Implement the ask, leap year function correctly.
What did it get wrong?
Gave no explanation or plan on what it is going to do.
Test was created as 1 big test instead of 10 individual ones.
Personal Rating: 7/10
Claude
What did it get right?
Test cases was split so that one can run each individualy.
Gave an explanation of what it did not use.
Implement the ask, leap year function correctly.
What did it get wrong?
Made assumption on what calendar to use
Was very brief with no plan
Personal Rating: 6/10
Gemini
What did it get right?
Implement the ask, leap year function correctly.
What did it get wrong?
No test case, test was in the main() function
Assume the calendar to use.
Again very brief with no plan.
Personal rating: 5/10
Kilo Code Auto Efficient: Qwen 3.7-plus
What did it get right?
Asked clarifying questions
Implement the ask, leap year function correctly.
Was very detailed on what it was going to do, almost too detailed but I prefer it.
What did it get wrong?
Did not use the Rust test harness, tests. was just some comparisons and println!() output.
Personal rating: 8/10
Kilo Code Hy3
What did it get right?
Use test harness, although only 1 test case.
As is common for Kilo Code, the output was very detailed and explained exactly what it was going to do.
What did it get wrong?
Would have preferred individual tests for the 10 tests.
Should have used modulo (%) operator instead of creating a remainder() function.
Assumed the calendar to use.
Personal rating: 8/10
See the References below for the raw output from each model.
Conclusion
All the models...