Actually Portable Executable

24 aug 2020 @ justine's web page

αcτµαlly pδrταblε εxεcµταblε

One day, while studying old code, I found out that it's possible to<br>encode Windows Portable Executable files as a UNIX Sixth Edition shell<br>script, due to the fact that the Thompson Shell didn't use a shebang<br>line. Once I realized it's possible to create a synthesis of the binary<br>formats being used by Unix, Windows, and MacOS, I couldn't resist the<br>temptation of making it a reality, since it means that high-performance<br>native code can be almost as pain-free as web apps. Here's how it works:

MZqFpD='<br>BIOS BOOT SECTOR'<br>exec 7<> $(command -v $0)<br>printf '\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7<br>exec "$0" "$@"<br>exec qemu-x86_64 "$0" "$@"<br>exit 1<br>REAL MODE...<br>ELF SEGMENTS...<br>OPENBSD NOTE...<br>NETBSD NOTE...<br>MACHO HEADERS...<br>CODE AND DATA...<br>ZIP DIRECTORY...

I started a project called<br>Cosmopolitan which<br>implements<br>the αcτµαlly<br>pδrταblε εxεcµταblε format. I chose the name because I like the idea<br>of having the freedom to write software without restrictions that<br>transcends traditional boundaries. My goal has been helping C become a<br>build-once run-anywhere language, suitable for greenfield development,<br>while avoiding any assumptions that would prevent software from being<br>shared between tech communities. Here's how simple it is to get started:

gcc -g -O -static -fno-pie -no-pie -mno-red-zone -nostdlib -nostdinc -o hello.com hello.c \<br>-Wl,--oformat=binary -Wl,--gc-sections -Wl,-z,max-page-size=0x1000 -fuse-ld=bfd -gdwarf-4 \<br>-Wl,-T,ape.lds -include cosmopolitan.h crt.o ape.o cosmopolitan.a

In the above one-liner, we've basically reconfigured the stock compiler<br>on Linux so it outputs binaries that'll run on MacOS, Windows, FreeBSD,<br>OpenBSD, and NetBSD too. They also boot from the BIOS. Please note this<br>is intended for people who don't care about desktop GUIs, and just want<br>stdio and sockets without devops toil.

Platform Agnostic C / C++ / FORTRAN Tooling

Who could have predicted that cross-platform native builds would be this<br>easy? As it turns out, they're surprisingly cheap too. Even with all the<br>magic numbers, win32 utf-8 polyfills, and bios bootloader code, exes<br>still end up being roughly 100x smaller than Go Hello World:

life.com is 12kb (symbols,<br>source)

hello.com is 16kb (symbols,<br>source)

Please note that zsh has a minor backwards compatibility glitch with<br>Thompson Shell [update<br>2021-02-15: zsh<br>has now been patched] so try sh hello.com rather<br>than ./hello.com. That one thing aside, if it's this easy,<br>why has no one done this before? The best answer I can tell is it<br>requires a minor ABI change, where C preprocessor macros relating to<br>system interfaces need to be symbolic. This is barely an issue, except<br>in cases like switch(errno){case EINVAL:...}. If we feel<br>comfortable bending the rules, then the GNU Linker can easily be<br>configured to generate at linktime all the PE/Darwin data structures we<br>need, without any special toolchains.

PKZIP Executables Make Pretty Good Containers

Single-file executables are nice to have. There are a few cases where<br>static executables depending on system files makes sense, e.g. zoneinfo.<br>However we can't make that assumption if we're building binaries<br>intended to run on multiple distros with Windows support too.

As it turns out, PKZIP was designed to place its magic marker at the end<br>of file, rather than the beginning, so we can synthesize ELF/PE/MachO<br>binaries with ZIP too! I was able to implement this efficiently in the<br>Cosmopolitan codebase using a few lines of linker script, along with a<br>program for incrementally compressing sections.

It's possible to run unzip -vl executable.com to view its<br>contents. It's also possible on Windows 10 to change the file extension<br>to .zip and then open it in Microsoft's bundled ZIP GUI. Having that<br>flexibility of being able to easily edit assets post-compilation means<br>we can also do things like create an easily distributable JavaScript<br>interpreter that reflectively loads interpreted sources via zip.

hellojs.com is 300kb (symbols,<br>source)

Cosmopolitan also uses the ZIP format to automate compliance with the<br>GPLv2 [update 2020-12-28: APE is now licensed ISC]. The non-commercial<br>libre build is configured, by default, to embed any source file linked<br>from within the hermetic make mono-repo. That makes binaries roughly 10x<br>larger. For example:

life2.com is 216kb (symbols,<br>source)

hello2.com is 256kb (symbols,<br>source)

Rock musicians have a love-hate relationship with dynamic range<br>compression, since it removes a dimension of complexity from their<br>music, but is necessary in order to sound professional. Bloat might work<br>by the same principles, in which case, zip source file embedding could<br>be a more socially conscious way of wasting resources in order to gain<br>appeal with the non-classical software consumer.

x86-64 Linux ABI Makes a Pretty Good Lingua Franca

It wasn't until very recently in computing history that a clear shakeout<br>occurred with...