Thomas Gazagnaire :: Reusing Buffers in Multicore OCamlThomas Gazagnaire<br>Building Functional Systems from Cloud to Orbit. thomas@gazagnaire.org

Reusing Buffers in Multicore OCaml<br>2026-07-24<br>#ocaml<br>OCaml is a pragmatic functional language. You<br>can do very advanced things with its type system (like an entire<br>roguelike), and you can also<br>hide the traditional buffer management techniques of C behind a nice<br>abstract interface. One of those techniques stopped being safe when<br>OCaml became multicore; I've hit it in<br>ocaml-wire last week. ocaml-wire is<br>the codec layer of Parsimoni's space protocol<br>stack; its decode path runs for every packet, so it should be safe and<br>fast (and ideally, work with multiple domains!).

Validators

ocaml-wire implements<br>EverParse's model of<br>binary formats. EverParse is Project Everest's verified parser<br>generator: a format description carries its own validity constraints,<br>and the generated validator rejects malformed input before any<br>application code sees it. The library keeps that shape in OCaml, and<br>can emit .3d files when formally verified C parsers are needed.

For instance, take a record of three 16-bit fields x, y and z<br>whose last field must be the sum of the first two. EverParse calls<br>this kind of constraint a where clause:

let f_x = Field.v "X" uint16be<br>let f_y = Field.v "Y" uint16be<br>let f_z = Field.v "Z" uint16be

let codec =<br>Codec.v "Record"<br>~where:Expr.(Field.ref f_x + Field.ref f_y = Field.ref f_z)<br>(fun x y z -> { x; y; z })<br>(f_x $ fun r -> r.x);<br>(f_y $ fun r -> r.y);<br>(f_z $ fun r -> r.z);

A hand-written version of the same validator shows what runs<br>underneath: the decoder pulls each field into a slot, then evaluates<br>the constraint over the slots:

let valid_naive buf off =<br>let slots = Array.make 3 0 in<br>for i = 0 to 2 do<br>slots.(i)<br>It allocates its slots on every call. A good way to see it is by using<br>memtrace, OCaml's allocation<br>profiler. The program opts in with one call at<br>startup, Memtrace.trace_if_requested (), and setting<br>MEMTRACE=naive.ctf in the environment turns sampling on, at a<br>default rate of one sample per million allocated words. Running the<br>validator ten million times under the profiler gives:

$ memtrace_hotspots naive.ctf<br>36 samples of 0.3 GB allocations

0.3 GB (100.0%) at Dune__exe__Records.valid_naive (records.ml:8:14-28)

Four words per call (the three slots plus the array header) over ten<br>million calls, at eight bytes per word, is the 0.3 GB; the sampling<br>rate reduces it to a few dozen samples, each with a source location.

The usual fix is to hoist the scratch to the toplevel and reuse it:

let slots = Array.make 3 0

let valid_scratch buf off =<br>for i = 0 to 2 do<br>slots.(i)<br>The same trace now reports 0 samples of 0 B allocations.<br>ocaml-wire's validators use this layout: decoded fields land in slots owned<br>by the codec value, allocated once when the codec is built.

Domains

OCaml 5 gave the language real<br>parallelism: spawn a<br>domain and two pieces of OCaml run on two cores at once. It also<br>turned every persistent scratch into shared mutable state, because a<br>codec value shared between domains shares its slots too. Run<br>valid_scratch from two domains, one validating a well-formed record<br>and one a malformed record, a million times each, and each domain<br>overwrites the other's slots mid-validation:

115726 wrong decodes out of 2000000

No exception is raised; the calls return wrong verdicts. For<br>ocaml-wire it means that valid packets can be silently dropped (in a<br>TCP stack that runs one connection per core, the dropped segment<br>stalls the connection), so adding cores decreases throughput.<br>Fortunately, OCaml ships a<br>ThreadSanitizer build,<br>available as an opam switch with ocaml-option-tsan (read more on the<br>Tarides<br>blog).<br>It instruments every memory access, and the first run reports the<br>racing pair, both stacks pointing at the validator:

WARNING: ThreadSanitizer: data race (pid=90385)<br>Write of size 8 at 0x00010a5ff720 by thread T4 (mutexes: write M0):<br>#0 camlRecords$valid_scratch_432 (race_tsan:arm64+0x10000465c)<br>Previous write of size 8 at 0x00010a5ff720 by thread T2 (mutexes: write M1):<br>#0 camlRecords$valid_scratch_432 (race_tsan:arm64+0x10000465c)

To fix this, the OCaml standard library has several tools. Here I<br>picked Domain.DLS, domain-local storage: a key is created once, and<br>each domain gets its own value, initialised on first use. With it the<br>scratch moves from one per program to one per domain:

let slots_key = Domain.DLS.new_key (fun () -> Array.make 3 0)

let valid_dls buf off =<br>let slots = Domain.DLS.get slots_key in<br>for i = 0 to 2 do<br>slots.(i)<br>The race test now reports zero wrong decodes and ThreadSanitizer is<br>silent. The per-call allocation count stays at zero, since the scratch<br>is created once per domain and Domain.DLS.get is a lookup.<br>Domain-local is not thread-local: two preemptive systhreads sharing a<br>domain must still not decode one codec value at once. OCaml 5 also<br>ships green-thread machinery through effect handlers, with...