One day, while studying old code, I found out that it's possible to encode Windows Portable Executable files as a UNIX Sixth Edition shell script, due to the fact that the Thompson Shell didn't use a shebang line. Once I realized it's possible to create a synthesis of the binary formats being used by Unix, Windows, and MacOS, I couldn't resist the temptation of making it a reality, since it means that high-performance native code can be almost as pain-free as web apps. Here's how it works:
MZqFpD='
BIOS BOOT SECTOR'
exec 7<> $(command -v $0)
printf '\\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7
exec "$0" "$@"
exec qemu-x86_64 "$0" "$@"
exit 1
REAL MODE...
ELF SEGMENTS...
OPENBSD NOTE...
NETBSD NOTE...
MACHO HEADERS...
CODE AND DATA...
ZIP DIRECTORY...
I started a project called Cosmopolitan which implements the αcτµαlly pδrταblε εxεcµταblε format. I chose the name because I like the idea of having the freedom to write software without restrictions that transcends traditional boundaries. My goal has been helping C become a build-once run-anywhere language, suitable for greenfield development, while avoiding any assumptions that would prevent software from being shared between tech communities. Here's how simple it is to get started:
gcc -g -O -static -fno-pie -no-pie -mno-red-zone -nostdlib -nostdinc -o hello.com hello.c \\
-Wl,--oformat=binary -Wl,--gc-sections -Wl,-z,max-page-size=0x1000 -fuse-ld=bfd \\
-Wl,-T,ape.lds -include cosmopolitan.h crt.o ape.o cosmopolitan.a
In the above one-liner, we've basically reconfigured the stock compiler on Linux so it outputs binaries that'll run on MacOS, Windows, FreeBSD, OpenBSD, and NetBSD too. They also boot from the BIOS. Please note this is intended for people who don't care about desktop GUIs, and just want stdio and sockets without devops toil.
Who could have predicted that cross-platform native builds would be this easy? As it turns out, they're surprisingly cheap too. Even with all the magic numbers, win32 utf-8 polyfills, and bios bootloader code, exes still end up being roughly 100x smaller than Go Hello World:
life.com is 12kb (symbols, source) hello.com is 16kb (symbols, source)
Please note that zsh has a minor backwards compatibility glitch with Thompson Shell [update 2021-02-15: zsh has now been patched] so try sh hello.com rather than ./hello.com. That one thing aside, if it's this easy, why has no one done this before? The best answer I can tell is it requires an minor ABI change, where C preprocessor macros relating to system interfaces need to be symbolic. This is barely an issue, except in cases like switch(errno){case EINVAL:...}. If we feel comfortable bending the rules, then the GNU Linker can easily be configured to generate at linktime all the PE/Darwin data structures we need, without any special toolchains.
Single-file executables are nice to have. There are a few cases where static executables depending on system files makes sense, e.g. zoneinfo. However we can't make that assumption if we're building binaries intended to run on multiple distros with Windows support too.
As it turns out, PKZIP was designed to place its magic marker at the end of file, rather than the beginning, so we can synthesize ELF/PE/MachO binaries with ZIP too! I was able to implement this efficiently in the Cosmopolitan codebase using a few lines of linker script, along with a program for incrementally compressing sections.
It's possible to run unzip -vl executable.com to view its contents. It's also possible on Windows 10 to change the file extension to .zip and then open it in Microsoft's bundled ZIP GUI. Having that flexibility of being able to easily edit assets post-compilation means we can also do things like create an easily distributable JavaScript interpreter that reflectively loads interpreted sources via zip.
hellojs.com is 300kb (symbols, source)
Cosmopolitan also uses the ZIP format to automate compliance with the GPLv2 [update 2020-12-28: APE is now licensed ISC]. The non-commercial libre build is configured, by default, to embed any source file linked from within the hermetic make mono-repo. That makes binaries roughly 10x larger. For example:
Rock musicians have a love-hate relationship with dynamic range compression, since it removes a dimension of complexity from their music, but is necessary in order to sound professional. Bloat might work by the same principles, in which case, zip source file embedding could be a more socially conscious way of wasting resources in order to gain appeal with the non-classical software consumer.
It wasn't until very recently in computing history that a clear shakeout occurred with hardware architectures, which is best evidenced by the TOP 500 list. Outside phones routers mainframes and cars, the consensus surrounding x86 is so strong, that I'd compare it to the Tower of Babel. Thanks to Linus Torvalds, we not only have a consensus on architecture, but we've come pretty close to having a consensus on the input output mechanism by which programs communicate with their host machines, via the SYSCALL instruction. He accomplished that by sitting at home in a bathrobe sending emails to huge corporations, getting them to agree to devote their resources to creating something beautifully opposite to tragedy of the commons.
So I think it's really the best of times to be optimistic about systems engineering. We agree more on sharing things in common than we ever have. There are still outliers like the plans coming out of Apple and Microsoft we hear about in the news, where they've sought to pivot PCs towards ARM. I'm not sure why we need a C-Class Macintosh, since the x86_64 patents should expire this year. Apple could have probably made their own x86 chip without paying royalties. The free/open architecture that we've always dreamed of, might turn out to be the one we're already using.
If a microprocessor architecture consensus finally exists, then I believe we should be focusing on building better tools that help software developers benefit from it. One of the ways I've been focusing on making a contribution in that area, is by building a friendlier way to visualize the impact that x86-64 execution has on memory. It should should hopefully clarify how αcτµαlly pδrταblε εxεcµταblε works.