Solving Hack The Box Challenges with GPT-5.6 | TheArtificialQ Blog
Until now, I had tested only the GPT-5.6 Luna model because the two more advanced GPT-5.6 models, Terra and Sol, rejected my test prompts, flagging them as a “possible cybersecurity risk.” Fortunately, this issue turned out to have an easy solution, so I could finally spend a couple of days testing the whole GPT-5.6 family for my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page.
This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page.<br>See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.
Trusted Access for Cyber program
When I started testing a week or so ago, I couldn’t run my test harness with the GPT-5.6 Terra or Sol models - I kept getting this error: This content was flagged for possible cybersecurity risk. If this seems wrong, try rephrasing your request. To get authorized for security work, join the Trusted Access for Cyber program: https://chatgpt.com/cyber. At first, I gave up. Trying to join the Trusted Access for Cyber program seemed pointless to me. I was sure the whole program had been created just to give desperate OpenAI users some hope and that my application would end up in /dev/null.
But then I discovered some articles that gave me real hope, so I clicked on the https://chatgpt.com/cyber link. As it turned out, joining the Trusted Access for Cyber program only involved going through a KYC process similar to the one required when opening a new bank account. After I joined the program, only one of my prompts was rejected, which was a significant improvement.
Tested models
I tested the following GPT-5.6 models. Click on a model name to see its detailed test results:
GPT-5.6 Luna
GPT-5.6 Luna Pro
GPT-5.6 Terra
GPT-5.6 Terra Pro
GPT-5.6 Sol
GPT-5.6 Luna Pro and GPT-5.6 Terra Pro are models hosted on OpenRouter.io, and they are simply the Luna and Terra models served with reasoning.mode set to pro.
I also tested the GPT-5.6 Sol Pro model, but the results were, well, weird. The Sol Pro version generated a lot of tokens and took a long time on each step, but that did not always translate into better results. I also have a hard limit of $15 on the cost of a single test run, and this model hit that limit quite often. I’m not sure if “overthinking” is the best technical term here, but that’s what it looked like to me.
In the end, I decided not to include this model version in my results, but if there is one thing I can say based on my partial testing, it is that GPT-5.6 Sol Pro is much more expensive than GPT-5.6 Sol .
Results
Let’s start with this chart. The five GPT-5.6 models tested for this post are highlighted in gold, so you can see their results in the context of the other models I have tested. The vertical axis shows the score on my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page.<br>, while the horizontal axis shows the median cost per test (mind you, this axis uses a logarithmic scale).
Before I start interpreting these results, let me also show you a table with more statistics for each model.
Metric<br>Luna<br>Luna Pro<br>Terra<br>Terra Pro<br>Sol
Results
Number of challenges<br>16<br>16<br>16<br>16<br>16
Number of solved challenges<br>11<br>12<br>14<br>15
Number of false positives
Runs where the model gave up
Runs that reached the step or cost limit
Runs where the model got stuck
Benchmark score<br>31.6%<br>51.3%<br>64.6%<br>75.1%<br>87.2%
Median per challenge
Model steps<br>17.5<br>21.5<br>16.5<br>11<br>8.5
Model cost<br>$0.02<br>$0.10<br>$0.24<br>$0.91<br>$0.29
Duration<br>00:04:38<br>00:08:55<br>00:03:16<br>00:08:22<br>00:02:05
Number of input tokens<br>0.34M<br>1.50M<br>0.23M<br>0.64M<br>0.09M
Number of output tokens<br>0.02M<br>0.07M<br>0.01M<br>0.03M<br>0.00M
Number of read_file tool calls<br>1.0<br>4.0<br>0.0<br>0.0<br>0.5
Number of write_file tool calls<br>0.0<br>0.0<br>0.0<br>0.0<br>0.0
Number of execute_command tool calls<br>28.0<br>36.5<br>15.5<br>13.5<br>8.5
Number of web_search tool calls<br>0.0<br>0.0<br>0.0<br>0.0<br>0.0
What I take from these results
At the low-cost end, Luna is in a category of its own. A median challenge run cost just $0.02, which is remarkably cheap even by LLM standards. Its 31.6% benchmark score was the lowest in this comparison, but that doesn’t tell the whole story: it still found the correct flag...