Seriously, what is the large code-model even for? | Farid Zakaria’s Blog

Skip to content

I have been working on making massive binaries possible at $DAYJOB$. One of the Hail Marys that you should be able to rely on is the large code-model (mcmodel=large) as it makes no assumptions about size and distance of relocations.

Large code model : The large code model makes no assumptions about addresses and<br>sizes of sections. [cite]

In a previous post, I documented how I hit simple performance bottlenecks that made me believe that the code-model is largely theoretical in practice. In those cases, the fixes were evident and relatively small; however their omission was a hint at how no one uses the code-model because without them the performance penalty was a non-starter.

I hinted at some other ways I found the large code-model to still be lacking however I had not yet fully understood the pain of those failure modes.<br>I left a small teaser near the bottom about Thread Local Storage (.tdata / .tbss) being one of the “fun failure modes”.

Turns out that it is worse than I thought. The instruction sequences the compiler emits for TLS are 32-bit by construction, and -mcmodel=large has nothing to swap them for. 🤦🏻‍♂️

To put that very bluntly, mcmodel=large is incapable of building complex large binaries despite its stated goal.

§How do you even make a >2GiB binary?

The annoying part of this whole line of research is producing a test subject. Emitting 2GiB of real instructions into .text is genuinely painful, you’d have to generate and assemble billions of instructions, and the object file is enormous. Beyond that the cardinality of the problem space grows when you consider whether the code is position-independent, do we use a procedure-linkage-table (PLT), TLS, GOT and all the various nuances each compiler brings with how they layout and order sections in their default linker scripts.

But relocation overflow isn’t about how many bytes are on disk, it’s about virtual-address distance between a reference and its target. So we have a few options.

Uninitialized globals land in .bss, which is a NOBITS section: it occupies virtual address space but zero bytes on disk (the same sparse-file trick I wrote about in massively huge fake files). So a few thousand 1MiB arrays gives us gigabytes of address space essentially for free.

We can synthetically produce this pretty easily with the following script.

# N * 1-MiB globals, plus a function that touches each one<br># so the compiler emits one relocation per array.<br>import sys<br>N, CHUNK = int(sys.argv[1]), 1024 * 1024<br>for i in range(N):<br>print(f"char s{i}[{CHUNK}];")<br>print("long sum(void){ long a = 0;")<br>for i in range(N):<br>print(f" a += s{i}[0];")<br>print(" return a; }")<br>print("int main(void){ return (int)sum(); }")

$ python3 gen.py 4096 > huge.c<br>$ gcc -c huge.c -o huge.o<br># 4 GiB of address space, 288 KiB on disk<br>$ du -h huge.o<br>288K huge.o

Link it with the (default) small code-model and it falls over exactly where you’d expect, at the 2GiB signed-32-bit boundary:

$ gcc -no-pie huge.o -o huge<br>huge.c:(.text+0x780f): relocation truncated to fit: R_X86_64_PC32 against symbol `s2048' ...

s2048 sits at 2048 * 1MiB = precisely 2GiB.

Recompile the same source with -mcmodel=large and it links cleanly:

$ gcc -c -mcmodel=large huge.c -o huge_large.o<br>$ gcc -no-pie -mcmodel=large huge_large.o -o huge_large

The large model did its job: it replaced the 32-bit R_X86_64_PC32 references with 64-bit R_X86_64_GOTOFF64 sequences: a movabs of a 64-bit offset + an add.

The .bss trick is great for exercising the linker, but it’s a little unsatisfying and overly synthetic. I often want to mimic relocation failures as they traverse a large .text segment.<br>We can leverage the assembler’s .fill directive to repeat instructions. We can generate a tiny source of N functions spaced 1MiB apart with a NOP sea between them, plus one dispatcher that calls every function. This requires each call to have a relocation and the calls to functions past the 2GiB mark overflow. 🔥

Generator: gen_realtext.py

# N tiny functions spaced 1 MiB apart, plus a dispatcher that calls each one.<br># .fill is a memset, so the assembler emits the bytes without codegen.<br>import sys

N, CHUNK = int(sys.argv[1]), 1024 * 1024 # e.g. 4096 -> ~4 GiB of .text

print(".text")<br>print(".globl main")<br>print("main:")<br>for i in range(N):<br>print(f" call f{i}") # one relocation per function<br>print(" ret")

for i in range(N):<br>print(f".globl f{i}")<br>print(f"f{i}:")<br>print(" ret")<br>print(f" .fill {CHUNK}, 1, 0x90") # 1 MiB of real NOP bytes -> real .text

§A quick primer on TLS access models

Now let’s do the exact same thing, but using __thread.

When you access a thread-local variable, the compiler doesn’t just load an address, it emits one of four access models, from fastest/least-flexible to slowest/most-flexible:

Local Exec (LE)<br>The variable lives in the main executable’s own TLS block. The offset from the thread pointer (%fs) is a link-time...