The Changing Landscape of CTF-Based Learning — exploiting.systems

A day before I was planning on posting this, a blog post from kabir.au hit the front page of HN. I've removed a lot from the original version to avoid being redundant, but kept the pieces I believe are additive to the conversation. Read the other post here.

Background

About a decade ago, I participated in my first Capture the Flag (CTF) competition and was immediately hooked. The rush of solving a challenge, getting the flag, and inching up the scoreboard scratched some kind of itch I didn't know I had.

CTFs became a cornerstone of my career. Each one would teach me something new, and as I progressed there was a lot of headroom to grow and learn. I wouldn't say I truly entered the competitive scene, but I eventually co-ran a hobby team with mild success before life pulled the members in different directions.

As I took on more leadership-oriented positions, it became not just a great way to learn, but also an excellent way to mentor. I often extended an invitation to new team members, hoping they would discover the same passion and competitive drive I had previously found, which many did.

Things have changed. Here are some screenshots are from recent CTFs which still had the old scoreboard graphs available.

Before: Dice CTF 2023

After: Dice CTF 2025

(Note: I included the 2025 scoreboard instead of 2026 because DiceCTF changed their scoreboard design)

Before: srdnlen CTF 2025

After: srdnlen CTF 2026

Before: UMass CTF 2025

After: UMass CTF 2026

I'm pretty confident you can guess what's going on here.

The Modern Landscape

Agentic workflows have taken over. Whether it's Claude Code, Codex, or even a harness with a local model, teams and individuals are being forced to incorporate agents in order to remain competitive. On top of this, GPT-5.5 and Opus 4.6 and 4.7 have all proven to be a large step above the models from even a year ago.

This reflects the state of the industry. Anthropic's Mythos marketing push has seen a surge in the use of LLMs for discovering real-world vulnerabilities in significant projects such as Firefox, the Linux kernel, and even cURL. There are a lot of caveats there, and I encourage you to read more on the specific bugs if you haven't. But regardless of how much is hype versus a truly significant capabilty (it's probably somewhere in the middle), the increase in the capabilities of models are affecting CTFs significantly.

In some aspects, top CTF competitions have always represented the cutting edge, showcasing newly discovered vulnerabilities with a twist or demonstrating new technologies (AIxCC in 2025 or the Cyber Grand Challenge in 2016). So it's no surprise that the trend is continuing. The winning team being decided by who has the best agentic harness is just a (depressing) reflection of the times, right?

But what about academic settings where CTFs are often used as a learning tool?

Only two years prior, studies identified LLM assistance as a potential issue in academic settings. But due to the lack of capability in models at the time the study concluded that the assistance provided was limited and there was no need for structural changes. Yet even then, separate studies in 2024 were showing that LLMs were able to achieve a "higher success rate than an average human participant".

In 2026, leveraging a fully automated or hybrid (human-in-the-loop) workflow has become the most effective strategy for winning competitions. On certain platforms, specifically Hack the Box, a significant decrease in average time-to-solve has been thoroughly documented, suggesting participants in large, competitive learning platforms are already using LLMs for automation. Hack the Box has historically been a very competitive platform, so this also isn't too much of a surprise.

However, there are significant implications for CTFs intended explicitly for academic and learning purposes. Examples include PicoCTF (which has now been rebranded CyLab Security Academy) or those hosted as part of college courses.

The reason CTFs were such an effective learning tool was because they leveraged the psychology of competitiveness to motivate new participants to achieve new heights. That leverage evaporates if someone else is foregoing learning and using LLMs to simply win.

The Information Search Process

Some may be familiar with Kuhlthau's Information Search Process. This is the process most CTF participants went through prior to agentic workflows and today's models.

Using agentic workflows completely removes every part of the ISP. A quick pull of ctf-agent, and the flags start pouring in. Build a good framework (or just clone or vibecode one) once and you can effectively point it at a CTF server and rocket up the leaderboard.

This compromises components of CTFs as a learning tool: the incentives and the repetition necessary to form strong pathways in your brain.

Incentive Structure

CTFs incentivize quick time-to-solve using...