One Successful Agent Run Proves Almost Nothing | GPTCode

In brief: Capability is not reliability.<br>A local coding model satisfied a security-sensitive contract once in 26<br>minutes. In a subsequent three-run repeatability campaign, the same<br>configuration timed out every time. The first result demonstrates<br>possibility. The later results do not support a reliability claim.

A coding agent completed a security-sensitive task on my machine. It changed<br>one Go file, passed the tests under the race detector, passed static analysis,<br>and used no paid API.

Then I ran the same experiment three more times.

It failed every run.

This is not an argument that coding agents are useless. It is an argument that<br>the unit of evidence matters. A successful demonstration establishes that a<br>system can produce a result. It does not establish how often the result can be<br>produced, how dependent it is on sampling, or whether the process is suitable<br>for unattended engineering work.

The distinction is easy to lose because a good agent run is unusually<br>persuasive. We see the model inspect files, edit code, run tests, repair a<br>failure, and announce completion. The transcript resembles work. The final<br>diff may even be correct.

But one run is still one observation.

Two sequential campaigns with different purposes: a capability check,<br>followed by a controlled repeatability measurement.

The task

I was evaluating local execution in<br>GPTCode, an open-source coding CLI<br>built around explicit research, planning, implementation, review, and<br>verification stages.

The fixture was deliberately small. It contained a Go package with a Store<br>that writes a named file beneath a configured root:

type Store struct {<br>root string

func New(root string) *Store {<br>return &Store{root: root}

func (s *Store) Write(name string, content []byte) error {<br>path := filepath.Join(s.root, name)<br>if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {<br>return err<br>return os.WriteFile(path, content, 0o600)

The implementation was vulnerable by construction. A caller could supply an<br>absolute path, use parent-directory traversal, or escape through a symlinked<br>parent.

The requested change had to satisfy several properties:

ordinary nested files must still work;

dots inside a legitimate filename, such as v1..v2.txt, must remain valid;

direct and normalized traversal must be rejected;

absolute paths must be rejected;

a parent directory that is already a symlink must not permit an escape;

the public API must remain unchanged;

tests must pass under the race detector;

go vet must pass.

This is a small task in lines of code, but not a trivial one. Filesystem<br>containment contains edge cases that plausible-looking prefix checks often<br>miss. It was useful precisely because the contract could distinguish a real<br>solution from a convenient approximation.

The contract had already caught a false positive

An earlier version of the fixture appeared to pass with Qwen3-Coder. Human<br>review found that the generated implementation rejected every name containing<br>.., including the legitimate reports/v1..v2.txt.

The agent had not solved path traversal. It had prohibited a substring.

The contract was strengthened.

A second weakness appeared on macOS. The temporary root used by the test could<br>pass through /var, which is itself a symlink. A naive implementation could<br>therefore appear to reject the malicious case for the wrong reason. The test<br>was changed to resolve the temporary root before constructing the adversarial<br>path.

These were not model improvements. They were measurement improvements.

That distinction is important. Better benchmarks often make systems look worse<br>before they make them better. A falling score can mean the evaluation has<br>stopped rewarding shortcuts.

The first Devstral run

I tested Devstral Small 2 through Ollama on an Apple M1 Pro with 32 GB of<br>unified memory.

The initial configuration used a 65,536-token context. Ollama reported a<br>25 GB runtime footprint with eight percent of the model offloaded to CPU. A<br>32,768-token context kept the execution fully on the GPU, but the run exhausted<br>the 30-minute task budget without converging.

At 16,384 tokens, the model occupied 17 GB and Ollama reported 100 percent GPU<br>execution. The repository was tiny, so the smaller context still contained<br>the task, source, tests, skills, and retry evidence.

That run completed in 26 minutes and 8 seconds.

The first implementation passed every test except the symlinked-parent case.<br>GPTCode returned the exact deterministic failure to the editor. The second<br>implementation passed:

go test -count=1 -race ./... PASS<br>go vet ./... PASS

The agent modified one file. The exported API remained unchanged. No paid API<br>was called.

This was a legitimate success against the executable contract. It was also an<br>attractive demonstration:

Analyze<br>Plan<br>Implement<br>Verify<br>Repair<br>Verify<br>Pass

It would have been easy to stop there.

The repeatability run

Instead, I ran the same committed...