LLMs: code is just untrusted text, until you validate it. – H[ack]-∞S

Skip to content

People often ask me what the best programming language is to use with LLMs. One of the strongest options, in my opinion, is Rust. However, before choosing it, you first need to ask yourself whether it’s worth picking a language with manual memory management instead of one with a garbage collector.

And regardless of that decision, the first thing is to understand that LLMs are not deterministic systems.

They should be used for what they actually are: smart suggesters.

You just don’t have to trust suggesters by default.

They read input text.

They responds with output text.

Disclaimer: it’s not code. Not yet.

Even if it looks like code, it compiles as if it was actual code, trust me.

It’s not.

The rule is:

code is in un untrusted state, until you verify it. untrusted state

No matter of the language.

Rust is not different. Its compiler strictness, doesn’t allow you to break the rule.

But let’s take a step back, in order to understan why have to evaluate other languages, not only Rust.

Manual memory management forces you to obsess over ownership & lifetimes at every step.

The bugs:

use-after-free

double-free

leaks

dangling pointers

overflows

… are brutal to debug, especially in large or concurrent codebases. Code tends to be complex, and sometimes application logic is more related to the way memory is allocated, not only at the way data is processed. It means there are tons of elements you have to consider in the I/O design, which tends to be exponentially complicated in complex infrastructures, where you can have async processes, threads, locks, concurrent accesses.

In this scenario, people often choose GC languages, because they remove an entire class of problems. You allocate and move on. Cleaner code, simpler APIs, safer concurrency.

But GC has costs. Latency and memory.

For systems, games, kernels, databases, manual control still wins.

On the other side, modern GCs are so highly optimized that, for most software, you can choose them unless your main target is the extreme performance, over productivity and simplicity.

And Rust is a modern language that tries to thread the needle with the borrow checker. It gives you extreme performances at the cost of massive complexity.

Rust teams fight the compiler all day long. Consider for instance, a typical Rust+Tokio project: Rust’s strictness + Tokio’s zero-cost async creates a combination that is extremely safe & fast when done right. But mentally expensive, even with apparently small details like lifetime management.

This is exactly why some people prefer simpler GC languages.

But there’s a lot of people that choose Rust because of its pattern-heavy nature, and as a consequence, for its reputation of being LLM-friendly.

What they often miss, however, is that this same strictness brings its own inner complexities: the ownership model, borrow checker, and lifetime management can add significant cognitive overhead and development friction that goes beyond what most teams anticipate. If you only consider how easy it is to generate Rust code, but you don’t evaluate properly how much time you’ve to spend to review and validate it, you’re introducing a new generation of technical debt.

This is one of the most underestimated issues: LLMs are excellent at generating code, but they don’t truly understand simplicity or the deeper abstractions in software. They’re just very capable text generators, helping you to get sophisticated scaffolding without any inherent logic, intuition, or understanding of consequences.

It’s entirely up to you to steer the process: to keep things simple, eliminate unnecessary code, and avoid hidden side effects. And in Rust, believe me, it takes time and effort.

What worries me is how many developers underestimate this aspect, treating this generated "text" as if it were the reliable output of a deterministic system, so there is no need to double check it.

They speak of LLMs as just another clean layer in the stack, comfortably sitting between the human and the machine.

It’s not.

It’s probabilistic text, the result of statistical patterns learned from a vast, uncontrolled corpus of human writing.

This shift has quietly made humans the bottleneck. We no longer need to type code to create software. Or at least, typing is not the only way.

The act of writing it used to force us to think, design, and truly understand what we were building, though. And this cannot be skipped, being delegated to text-generators machines.

They just don’t do it.

Now we write a vague prompt with some requirements and expect the machine to figure out what we actually need. This works surprisingly well for simple, common tasks, which is why it’s fair to call LLMs a powerful autocomplete.

"Given the id parameter, write the SQL statement that updates the user's email" SQL prompt

Easy. Isn’t it?

Real world development is rarely that linear, though....