This is a big forwards-compatibility risk. Suppose glibc adds a new symbol, and then a GPU driver adds a dependency on that symbol. The user wants to run an old executable with the updated GPU driver (maybe the old GPU driver doesn’t support their GPU). Normally, this would work fine: the user has to use a new copy of glibc, which will be compatible with both the new GPU driver and the old executable. But with your approach, the GPU driver is forced to use the glibc reimplementation which has been statically linked into the executable. Which, since the executable is old, can’t possibly implement the new symbol.
The same issue would occur if glibc adds a new version of an existing symbol and then the GPU driver is recompiled. (Or, for that matter, if a GPU driver adds a dependency on a symbol which glibc has always supported but which isn’t in the subset that you reimplemented, though in theory that could be solved if you reimplemented 100% of the symbols.)
Yes this is just asking for trouble - and all it does is solve a problem that doesn't really exist. Just dynamically link against the oldest glibc you want to support. Its annoying that Linux toolchains don't have built in easy mode support for that but its much easier to deal with than this thing will be when it breaks.
It's also not just new symbols, the loader semantics also aren't static and new enough libraries may not support older semantics - e.g. the loader used to use DT_HASH entries for symbol resolution but now they are no longer present on all distributions.
This is not a desirable solution on musl based systems. Whatever you do in that situation ends up horrible, so it is about finding the least bad solution. Which this seems like a workable variant of.
Yes but only the very last version and only paid support. Don't mind to pay, it's just that no amount of payment would be a proper solution to ancient environment with everything.
(I'm certain you know this already but I'm posting this for people who might want more context)
Part of the problem that people don't really realize is that it's not practical to just 'link against old glibc'.
In order to ensure compatibility with RHEL7 we need to link against RHEL7 libraries, which means we need to build on RHEL7. That means old, unpatched versions of glibc, libpcap, libcurl, openssl, or who knows what else, in case there's some backwards-incompatible change in newer RHEL versions.
Alternately, we can build/patch the libraries we use and compile them statically into the binary, which is vastly more maintenance work for us for very little benefit.
Meanwhile, compiling against older versions of libraries like glibc means we don't get the benefit of updates; not just new features in glibc, but things like more/better SIMD support in glibc algorithms, more/better optimizations in GCC, and so on.
Alternately, we can stop supporting RHEL7, like Redhat did, and build against RHEL8... Or we can build a separate version against each version of RHEL we want to support.
Funnily enough, we've been shipping RHEL7 RPMs of our product for years, and only recently realized that they won't actually run on RHEL7 because at some point the toolchain updated and now we're compiling against too new of a glibc version and then packaging it into a RHEL7 RPM. It wouldn't have worked on RHEL <8 for the past... few years? But no one uses RHEL7 so it went completely untested for ages and we didn't get any customer complaints.
Now we're re-labelling our RPMs as EL8, but it's just a cosmetic change so that we're claiming the version that we actually require.
Yeah, to make things spicy, what we ship is a plugin, so there's limit of how static we can link. glibc stays dynamic and that's a huge limiting factor.
I've got plans to try to build on RockyLinux9 and package its glibc along with the app. Simple helloworld works, so I have a glimpse of hope that it would be possible to ship like that.
> Just dynamically link against the oldest glibc you want to support.
Just the other day I tried running an older binary and it failed with a glibc error, despite it being linked to a glibc version that's barely 5 releases behind the one on my system. So maybe glibc isn't backwards compatible after all...
Exactly. I think you're doing fine. It's weird that you're using C++ here, since that brings in a lot of crap, but then your programs are probably C++ anyway, so it's not a problem.
Honestly I'm impressed by this work. It's a thankless thing to have built, it's definitely feasible to build but more work than almost anyone would have been willing to do. What would be even neater is if Musl had this builtin.
So we are re-inventing patched a.out files, back when UNIX systems started to introduce dynamic loading, before ELF was invented?
Advocates of static linking keep forgetting once upon a time UNIX only had static linking, then we had overlays, and eventually dynamic linking came to be.
And I also remember very well that dynamic linking appeared ONLY because we were catastrophically short on memory; everything else was added much later.
Now we have plenty of memory, and we can very well return to our blessed roots!
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?
On the one hand, this is technically true, but on the other, what serious issues do you know that will cause problems in practice? I run tests on 1,000 of the most popular Debian packages.
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ ABI all depend into each other. You cannot change any of them independently. All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library and make calls like dlopen.
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
In what sense do binary interpreter / loader, GCC compiler, C library and system C/C++ ABI dependent on each other? I have certainly mixed different versions of all these components without problems so far.
When you compile GCC you need to provide a full glibc installation as your target. It is also a dependency of libstdc++.
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
I can't comment on the C++, I can imagine there plenty of issues, but for C I don't see this. You need some libc if you compile with gcc, but this generally does not introduce a hard version dependency on the specific version (there may be a minimum requirement if you compile against a new version that a symbol with a different ABI).
The interpreter/loader is glibc and a key part of bootstrapping an executable built against glibc is loading libc itself before continuing on to load the program. Versioning is a problem when distributing binaries linked against a newer glibc to distros that ship an older one. The C compiler doesn't really care as much.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well.
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
There is no libc independent ABI. ABI doesn't purely mean just calling conventions.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
Sorry, again this too vague for me. What is the exact problem with atomics and thread_local in C that would make the ABI dependent on a specific version of glibc? I know the ABI is not just calling convention and e.g. for atomics may involve calling a function from libatomic. But from my understanding, this is all part of a standardized ABI that does not change and can be provided by different implementations.
Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
> Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
You still have functionality like `dlopen` with C or `pthreads` with C. ld-linux.so is the thing that prepares the stack frame, or the segment pointers (FS_BASE on x86_64).
When you have your main loaded __libc_start_main_impl is called from glibc and if you disassemble it you'll see this line:
call 0x7ffff7c28790 <_dl_audit_preinit@plt>
Guess where _dl_audit_preinit lives?
(gdb) info symbol _dl_audit_preinit
_dl_audit_preinit in section .text of /lib64/ld-linux-x86-64.so.2
> I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
Thanks, I need to dig into this more. But I still not convinced there is a real issue here. Of course different infrastructure needs to exist for different feature. What exactly is the underlying issue with thread_local in your specific scenario? It seems Musl does not setup the the infrastructure needed for ABI-compliant thread-locaL data access?
THERE IS NO STANDARD, AGREED ON, RFC'd or EVEN BEHAVIORALLY DOCUMENTED ABI FOR MUSL TO FOLLOW!
It's not Musl's fault that Glibc has an undocumented, badly designed, tightly coupled implementation. It's not Musl's fault when they decide to not follow Glibc's design which they have to reverse engineer and rewrite from scratch. Then Glibc can break it in a future version anyway. They give 0 guarantees and have a track record of breaking things.
Sorry but I feel like you're trying to excuse Glibc out of their terrible design. Glibc didn't ask anybody for standardization. Glibc didn't document their behavior. Musl doesn't need to follow any undocumented behavior. Musl has its own implementation of thread_local variable and it works only for Musl dynamic binaries (which you cannot load with Glibc's ld-linux.so either). You need to tell your compiler to generate code that Musl prefers not Glibc. So effectively we have two ABIs.
There is no standard and any design that tightly couples the system binary loader to the C standard library is just extremely terrible design. Both Musl and Glibc do it. Both of them are terrible. Glibc was first and it set the horrible, binary hostile behavior "have no standards, trust Glibc, and recompile your programs again" as the "standard".
The entire Linux userspace depends on Glibc's terrible design. Due to Hyrum's Law, the exact needs of the complex and emergent behavior can only be surfaced via making a better designed, completely independent system loader and a completely independent libc and then fixing the entire tens of thousands of userspace libraries in the upcoming decades. It is just Sisyphean work to fix Linux userspace.
There is no simple "let libc-X obey the standard S" solution. There is no standard. Linux Standard Base project tried to set a standard. It failed.
I'll won't further discuss this with you. Maybe you have good intentions, maybe you're playing dumb. I'm not sure anymore. If you're the former, I'm not going to deliver every single bit of information. I provided enough resources and you can learn.
Unless you use musl-gcc which is a proper GCC frontend with musl libc sysroot. The compiler is indeed (too) tightly coupled to its sysroot but linux is not special here. If anything interacts with the CRT it needs to know which CRT to use.
I'm not arguing that there aren't problems here, just that I think people have different expectations that are a little extreme. The libc implementation is a core part of the operating system, if you want to target that OS you should target that libc. Some distros (alpine) prefer musl libc, most prefer glibc. And distros really are different operating systems.
I don’t imagine that you’re unaware of any of this, but: ld.so and libc.so are heavily interdependent in deliberately undocumented ways with Glibc and outright the same file with shared Musl. And while you might usually get away with using any old GCC with the right architecture and ABI (especially for C; cf the musl-gcc hack), technically it needs to be built to target a specific libc version (particularly via symbol versioning; I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases). The C compiler of course is tied to the C ABI, and this wouldn’t be really worth mentioning except for the time where the GCC devs accidentally the whole SysV i386 ABI and pretended that the stack was always 16-byte aligned, why do you ask, except on RHEL. The C++ parts I can’t really comment on.
I am not really sure. For ld.so and libc.so I may believe this. The C ABI is very stable, and if you use a new symbol from a newer glibc, you certainly depend on it, but this can also be avoided. In any case, I do not see what is fundamentally misdesigned here. I can't quite image how it could work differently. If you upgrade something so that the e ABI changed you natually need to update other components. Static linking certainly seems a very poor replacement for this.
> I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases
And that's the correct approach and also one that many have taken. We just need someone willing to maintain that as an easy mode SDK for everyone.
Who said that? This approach has many problems that have already been discussed here, not to mention the fact that it leaves Alpine and Bionic-based systems out in the cold.
Let me remind you that Bionic is the most widespread libc in the Linux world, and Alpine is the most popular Docker layer.
The world doesn't end with glibc. And it doesn't begin with it.
That's libstdc++ depending on a C ABI (or a GNU ABI to be precise) but that doesn't make it a C++ ABI. You can also have global constructors in C with GCC.
Somehow most of my portability issues seem to be caused by glibc, its symbol versioning and close ties to the dynamic loader. Minor versions aren't compatible, no two Linux distros ship the same version and you can't just provide your own without also patching in your own dynamic loader.
At least as far as the defaults on Linux go I consider C the root of all evil.
AFAIK glibc is backwards compatible as long as a) your program is running on a newer version (i.e. you're not trying to dynamically link against an older version) and b) you're not using hidden/undocumented symbols.
In which case as long as you're using the documented public API and compile your program with the oldest version of glibc you want to support (some Ubuntu from 4-5 years ago should cover pretty much every current desktop) you should be fine. And with something like Docker this is trivial to do.
Sure it is annoying that you cannot use your current distro (especially if you use some rolling distro) to make binaries for everyone, but it takes very little effort to work around that. The only issue i can think of is if you absolutely want to compile using the latest version of your compiler and you cannot build the compiler from source to work in the Docker (or whatever) contain to work against the older glibc.
In complex cases, it turns out that the old version of glibc also pulls in other libraries and the compiler, and you're stuck with a very ancient sysroot.
You may often find that you can't compile new library versions in such a sysroot and link them statically.
So, it looks good on paper, but forget about the ravines.
While I don't disagree with some of the pain you describe, you conveniently gloss over the fact that gnu developed a system that worked, and then made it free to everyone to consult and use.
BSD also did it. They did it better. Maybe more modern but AOSP also did it but at a different level of binary: instead of ELF, using compiled Java bytecode archives.
The root cause is alternative libc implementations. Are BSD syscalls considered stable? I remember Go moving to use libc on OpenBSD. Solaris also has the libc as the stable interface. Linux kernel is an outlier here guaranteeing stable syscalls but you wanting to use another libc is not Glibc's problem.
Except the other UNIX systems, starting with the original one, evolved from only having static linking to various ways to connect libraries and applications.
> all shared libraries depend on the specific glibc version to load them
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
It is not just compatibility. You cannot load them into the memory with your system dynamic loader. You need to also ship ld-linux.so with the new version of glibc you have, if you were to distribute your program independently.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
Windows loader is part of the system ABI, and programs' libc(s) are loaded by it.
One of the ways Windows manages to support multiple libc's is by being careful not to mix allocators; if a system API you call allocates on your behalf, your libc can't free it, the system API will offer a function to free it.
Windows loader is certainly available to user programs though; LoadLibrary has been around longer than many developers.
LoadLibrary will also not be able to load arbitrary libraries compiled for newer versions of Windows though. Just like with gcc, Windows also does not guarantee forwards compatibility - because that would men freezing the feature set the system libraries provide.
Windows handles this much better than any other OS. The API passes versions (== structure sizes on the calling side), and WinAPI can handle that.
As for older Windows systems not being able to load new DLLs, they can; the format hasn't changed in a very long time. I've had experience with installing some DLLs on Windows NT 3.51 and running a modern Firefox, which is about 20 years behind the times.
Targeting an older version of Windows is "set a define so that the system headers don't expose functions that didn't exist on older Windows". You don't need an old toolchain to target old systems, you ask the newer toolchains to target it.
MS haven't made this work arbitrarily far back, I believe they deprecated targeting Windows XP in one of the more recent toolchains, but that was purely a "not worth supporting" situation.
It kind of does but in a "it is an implementation detail that you cannot access and it doesn't matter" sense. Unlike ELF, PE32+ files do not have an "interpreter" that you can specify.
Why? What kind of single-responsibility-principle that predicates the C standard library to become the both syscall interface and the binary loader as well?
What happened to the Unix principle "do one thing and do it well"? Windows solved this quite modularly and it works perfectly well.
Glibc is terribly designed and it is the core component of almost any Linux userspace app today. Let's not try to find excuses for horrible designs.
I don't disagree that the loader and the libc should be decoupled, but it's not as simple a split as you're making it out to be.
The reason they're tightly coupled is because there needs to be a bridge between Sys-V ELF ABI and the C runtime, and it makes sense for the implementation of that bridge to use libc itself which means special-casing the bootstrapping of libc during program loading.
glibc isn't unique here btw - musl libc ships its own loader that does the exact same thing as rtld (parse auxv/envp off the stack, relocate itself and load libc, initialize the CRT, then load the rest of the program's SOs in order).
Even if they were fully decoupled it is not unreasonable for software targeting newer versions of an operating system to only run on that version or later. Linux is actually the outlier here where the syscall interface is stable enough to make this a non-obvious problem.
There's a universe where you can load multiple versions of the same libc into the same address space but that isn't ELF, and it's not common. aiui the only reason PE can support e.g. multiple versions of malloc/free in the same executable is due to two level namespaces in symbol relocation tables.
---
edit: another observation is that for a functioning ELF loader you basically need a three stage bootstrap. Stage one parses the stack passed by the kernel and performs self-relocation. You need that to make non-static function calls and read/write global variables. Stage two loads libc and initializes the CRT with stuff like environment variables, aux vector, any dso for syscalls passed by the kernel, etc. That has to coupled because POSIX specifies stupid stuff like "getenv" and "argv" requires the loader to know how to move from the stack passed by the kernel to the memory reserved by the CRT that gets used by the final executable. Finally in stage 3 it can relocate and load shared objects and run any constructor functions pre-main, before jumping to start where the precompiled CRT start (then jump to main goes).
So if you want an ELF loader on Sys-V you probably need to couple the loader to the libc, just for libc to work at all. Unless you want to specify a bunch of behavior and binary layout in the C standard.
None of this is designed by glibc, it's the simplest way to go from exec to main.
In what way are they "broken" when Linux runs fine on millions of boxes? Sure, it might be a pain for proprietary software, but if your app is open source it's not that hard to build it on whichever distro you want. If your app is popular enough the distro maintainer will build it for you
Yeah, it's mind boggling how everything hard depends on glibc, even critical graphics systems.
I've become obsessed with getting rid of it, especially after I realized that contributing to GNU itself was a dead end. Freestanding Linux programming turned out to be much more fun anyway.
All libraries out there should adopt the SQLite design: programmers provide it with all the necessary functions. Instead of libraries hard depending on glibc, we get to inject the libc-ish subset it needs. Then we can use whatever we want under the hood. I'm working on porting SQLite to freestanding Linux system calls so it can run with zero dependencies. Wish I could say the same for software like mesa, I'd need a lot of help for this one...
Modularizing (and versioning each part independently) libc would go a long way. There is also a need to separate the stuff needed for system integration with what is necessary for users of the C language to actually do stuff.
Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version.
> what is necessary for users of the C language to actually do stuff
Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all.
The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me.
Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.
> A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
Great choice for small programs, but what if I want hardware accelerated 3d?
Yeah, that's the annoying part. Been wondering about this for years, and graphics support was among the first issues raised on the lone lisp GitHub repository. At this point I've even started exploring the mesa codebase, made some patches but didn't submit them yet due to the AI stigma.
With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering.
For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it.
Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
> Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space.
And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
It's not a problem in the same way that things like snap or flatpak aren't problems. It works but it bloats things up considerably and makes you wonder where it all went so wrong. I mean, dozens of slightly incompatible runtimes inside a single process?
Those incompatible runtimes are separated by a linker that doesn't resolve all symbols globally, but rather scoped to the shared object they're expected from. They all coexist happily, and if you're so inclined you could resolve the same symbol from each, if you had reason to do so.
> Windows explicitly does not want you to link the system libc
This is categorically false; UCRT[1] is a thing. The 'U' stands for universal. Unlike Linux, Windows allows developers to choose their ABI boundary and also ship that boundary if they desire, or use the 'system' one and ask older platforms to install redistributables or Windows update packages. There's the old and new C runtimes in MSVCRT.DLL and UCRTBASE.DLL, the C++ runtime in VCRUNTIME140.DLL, Win32 in KERNEL32.DLL, USER32.DLL and more, and then the stable-ish kernel interfaces in NTDLL.DLL, in order of 'closeness to the kernel'.
And also, 'libc' is a UNIXism; on Windows the term is CRT, for 'C runtime'.
I think the grander point is that there is no real 'system CRT' on Windows; as I mentioned, there are multiple entry points each at an appropriate level of abstraction available to both platform and application developers (not that there is a real difference between the two, since platform developers may also write applications like Office). Many Windows platform libraries (WIL, for instance) themselves use UCRT instead of MSVCRT now. The latter exists, but it is by no means and has not ever been by any means the single entry point to the Windows platform, unlike glibc on most desktop Linux distributions.
You can completely skip all CRTs on Windows and still have a program that can call dynamic loader and other Win32 APIs. Here is a tutorial: https://nullprogram.com/blog/2023/02/15/
When you link with UCRT your specific program just gets a "view" of standard C functions. You can load DLLs that were linked with MSVCRT from a UCRT program. It just works
That's a pretty great article. Re. the stack probes, it's not hard to supply your own; Raymond Chen recently did a blog series on stack probes, the entries for amd64 and arm64 are here:
Indeed. I was just about to point out the fact that the CRT only provides the C standard library; Windows applications can run perfectly fine without one by linking only to the Windows API DLLs.
> Unlike Linux, Windows allows developers to choose their ABI boundary
Linux has a stable system call ABI. I can trash the entire user space and boot Linux with init=/my/program if I want.
It's not that Linux doesn't allow developers to choose, it's that Linux doesn't actually control the userspace ABIs. Userspace is whatever we want it to be, and people settled on "GNU/Linux".
This is a noble but practically worthless ideal. Hardly anything is written against the kernel ABI except glibc itself. End-user application developers don't write against the kernel interface. Platform library and application developers like KDE and GNOME don't really write against the kernel interface either. Nowadays even driver developers don't entirely write against the kernel interface, instead also linking in glibc and lib{std}c++ for things like shader compilers (which is nowadays usually just a fork of Clang). Like it or not, glibc is the singular entry point for desktop Linux, which is the point of essentially every comment under this post.
That 'desktop Linux ≡ glibc' is something that I daresay the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU. Linux's unstable user-mode is the entire reason for Docker's existence. It is bodge upon bodge and has led to much gnashing of teeth. The GNU/Linux community (I don't care for Stallman's meme rant here) have to grow a backbone and understand that an OS is much more than just a kernel, it is a platform, and currently the Linux platform is a disparate set of communities with highly divergent goals.
That's why I've always said the best Linux is Android, because Android is a platform.
I'm going to add system calls to this thing every day until I have every single one. Then I'll use them to make all of my applications.
Already have my first project planned: a custom Rust network stack for my QEMU virtual machines so I can firewall them off in userspace without root access.
> the Linux platform
The linux platform is these system calls. Everything else is userspace. This is exactly what makes it special.
> The linux platform is these system calls. Everything else is userspace. This is exactly what makes it special.
This is exactly the attitude I meant by 'the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU' and 'an OS is much more than just a kernel, it is a platform'.
By 'platform', I mean windowing toolkits, such as Wayland. Audio libraries like ALSA and PipeWire. Desktop idioms and libraries like DBus, and the entire freedesktop.org effort including Mesa and more. Init systems like systemd. Bootloaders like GRUB/systemd-boot. Networking, including systemd-networkd, iwd, and NetworkManager. Power and device management including PowerDevil, Bluez.
If all of this is considered 'user space' then my point is strongly vindicated. On most competitor OSs these comprise the 'platform'. You won't ever encounter a Windows or OS X developer arguing that USER32.DLL or Foundation is 'userspace' even if it is technically true and code from those libraries may only ever run in ring 3. From the average user's point of view, the only 'userspace' are end-user applications that they have installed, like Chrome, video games, etc. From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'. This includes such trivia like calculator and camera applications.
We need to talk about the practical realities of a platform, not a theoretical, unattainable, and as I mentioned, useless ideal. These practical realities mean shipping a unified set of useful 'batteries included' user-mode libraries and applications. If one wants to develop the platform itself, then sure, the kernel has a nice stable interface, but on most Linux distributions the kernel is built from source anyway, so even that nicety is moot.
> From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'.
Every Linux distribution is its own platform.
You decried Stallman's meme rant yet you just rewrote the entire rant in your own terms. Linux is the kernel, GNU's just the stuff on top. You can get rid of it if you want and make your own. Android did exactly that.
It's more than a little funny that you disparage syscalls being Linux's stable interface, and then immediately praise Android, which only works because glibc isn't uniquely privileged the way you seem to want.
Also, I daily drive Alpine Linux (using musl) and I'm quite happy to assure you that glibc really is only one option, not at all mandatory for desktop Linux.
> It's more than a little funny that you disparage syscalls being Linux's stable interface, and then immediately praise Android, which only works because glibc isn't uniquely privileged the way you seem to want.
I never disparaged it; I said having it doesn't make it any more useful that OSs that don't, because the practical reality is that the kernel interface is narrow and highly technical, and a stable kernel binary interface has little to do with a user's day-to-day experience with an OS.
I daresay Android would still work pretty well even sans the kernel ABI stability, because the kernel is usually built from source, and if it isn't, then every Android source tree has kernel prebuilts for the kernel version shipping. I also never said nor implied glibc should be uniquely privileged; I think you have misrepresented my comment.
VC-LTL Implements a good subset of modern CRT functions using the msvcrt.dll, which makes modern MSVC compiler to emit binaries that theoretically can run in Windows NT 4.1 or Windows 98 (until CFG broke them, but you can turn it off). Still can teach an old dog new tricks.
YY-Thunk is a little more funky, it tries to implement new Win32 userspace API for old system by emulating the behavior and expected values using userspace API in the old apartment, so you can try to implement a function in Windows 10 with functions in Windows Vista, and hopefully it will work out.
>A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
I'm sure those OSes make efforts to make said runtime binary compatible between executables.
I am not sure having a stable kernel ABI/API is really a goal in practice or intent in all cases.
GPU drivers are a perfect example - GPUs are hellishly complex, and their interfaces are super high-level - OpenGL/Vulkan is not something that should be implemented by the driver.
I think a general practice that is followed with GPU drivers is to move as much of the complex and non-privileged stuff out of the kernel, and only keep modesetting/power management/memory and control stuff in.
That reduces massively the amount of code that needs to run in Ring 0, but in exchange, the interface between the user and kernel blobs is non-stable. Programs talk to the userspace blob and there's no guarantee of compatibility between different kernel/userspace driver versions.
This is something even Linus accepts, with the only contention that the userspace blob needs to be open source as well. They're not maintaining something in tree, that they have no way of testing without proprietary external deps.
Yeah. I made an entire lisp for this, and I've recently revived my Rust liblinux crate too. I've been documenting the system calls as I add them. Hopefully one day it'll rival man7 as the Linux system call reference.
> Can't be done. The libc is legacy, it can't be changed without breaking everything... Every other OS forces a C runtime.
Windows never had this problem. If you really want it, you can actually skip CRT completely on Windows. Only thing you need is the initialization code that's statically linked anyway: https://nullprogram.com/blog/2023/02/15/ . You can still make Win32 calls with it. CRT is there as a convenience.
And since Linux distros are compiling everything from source anyway, this makes Linux a prime candidate for complete libc and complete ABI replacement. Thanks to stable kernel ABI, we can also use the same kernel even with Docker etc! Musl did it. Why can a completely modularized libc not?
Will it break a lot of programs in the userspace? YES! Hyrum's Law exists.
Will it be rejected by a significant part of user base? YES! They rejected systemd as well.
Will it take one or more decades to have something working? HELL YES! This is Linux we're talking about. Everybody has seen how Wayland played out. However, maybe if Valve has enough motivation to fix Linux binary problem, we maybe can get finally a binary-compatible distro in a decade or half a decade.
SQLite already does this with its VFS layer. You hand it the OS functions it needs instead of it grabbing them, etc.
You could look at going straight to syscalls, mesa would be a rewrite nightmare
I went zero-dependency on a production CMS. Rust, no framework, no external crates beyond argon2 for password hashing. Running live on client sites. The hard part wasn't building it, it was accepting that everything you reach for is pulling glibc or similar assumptions back in through the side door. SQLite's VFS model is the right pattern. More things should work that way.
The concept you mentioned is just dependency injection in software engineering design, but instead of giving a class and protocol/interface you give a bunch of function pointers, which is, well, some kind of interface anyway
Most everyone else is talking about the problem while you mapped the room.
Solo is basically pg83's answer to the architecture you just described. If there is no libc independent ABI, build your own loader and shim the boundary.
I can understand Linus's obsession with taste and the areas it was overlooked or traded.
Wasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years.
Or do the MS thing, and ship multiple versions like msvcrt
> Wasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years.
Guess what happened with LSB? Some distros like Debian never adopted it. It also broke backwards compatibility at random times. It is now abandoned.
> Or do the MS thing, and ship multiple versions like msvcrt
MS can do that since the CRT is decoupled from the system DLL loader and Windows ABI is designed to isolated loaded DLLs into their own symbol namespaces. So your program can load DLLs that depend on a different CRT version.
Glibc's design tightly couples the system loader with the specific libc. Moreover symbols in ELF files are loaded into a single global namespace. You cannot ship shared libraries that are compiled with a different version of Glibc or Musl libc.
Btw, absolutely insane that it's 2026, and Linux cannot do the most defining OS thing - that is, provide a standardized environment to run binaries against.
All solutions to this problem are hacky, complex and controversal and highly fragmented, where this should be BASIC functionality
It's not insane. Many people who were in the GNU organisation wanted the OS to be as hostile against binaries as possible to prevent people distributing proprietary software with GNU. That GNU became the userspace layer.
It isn't that this is 2026 and it was a bug that hasn't been fixed since. The userspace of GNU is designed against what you consider as basic functionality.
Well, now it's possible! Furthermore, SoLo binaries can run, without modification, on glibc-based distros, alpine, and soon on android/bionic (not committed yet).
> Why? Have people managed to break the ancient concept of shared libraries
If you break or remove a shared lib here, you may no longer be able to compile something from source. I had that happen in the past before I started to use more statically compiled programs (and busybox too).
Assuming everything works as-is via shared libraries at all times, makes little sense for ALL linux systems. For instance, some people upgrade glibc manually. Then you need a working base system to resume compilation. I do that for my customized gobolinux system, so I can use any program version as well as any glibc version (assuming I can still compile the program; many older programs no longer compile).
Because many folks don't understand UNIX systems introduced dynamic linking for several reasons, and they actually only had static linking for almost 20 years, since UNIX was known outside Bell Labs.
Additionally many other OSes have had both approaches since their early days, Xerox PARC ones.
For some strange reason they assume to know better than all those researchers.
These decisions, and these studies, were made a VERY long time ago. It's completely unclear why the decisions made then are relevant now, and why they can't be challenged.
It is like advocating that we should drop cars and go back to chariots, because wooden wheels don't get flat, while forgetting why they are mostly used for tourists nowadays.
If you can figure out your own ELF loader, you can figure out how to build a partially static executable that doesn't need this. You can mix static and dynamic linking. Build tooling around that is just shit.
In this scenario, I'll have to choose which libc I want to run. These won't be portable Linux binaries in the true sense of the word; I'll have to leave Alpine out, and possibly Android, which I don't want.
Fair point - while it's certainly possible to make a decision per individual library elsewhere, libc is the one thing that really will need to be linked dynamically for glibc, and it'll freeze a minimum version into the binary (i.e. it should be built against an old version of glibc).
I don't think Android is a target in most cases of this, so this would boil down to shipping 2* binaries, one glibc and one musl.
</i> I'd need to check how musl behaves for compatibility, I'm assuming it'll be a minimum like glibc.
All of this said - you'll need to ship multiple binaries anyway, these days: x86_64 + aarch64. Possibly more…
Can you really mix it? When I break or remove a shared lib, some binaries no longer work. With static libs or even better, e. g. statically compiled busybox, I don't have that issue, so I disagree on the claim that mixing solves everything as such. I keep the basic toolchain I use as statically compiled variant. The whole system works better if I can break it less easily.
> Do you belive the machine is taking offence? It can not.
Obviously, this is an offense to me.
> Telling me, a coder you never met or interacted with before, that I would introduce more bugs than The Product(tm) is pretty rude and prejudiced.
Don't exaggerate. I didn't say that you personally introduce more bugs (as you rightly pointed out, I don't know you and don't know how often you introduce bugs), I said that the average developer introduces more bugs than an SOTA LLM.
Every couple of years, I revisit my PL dev hobby and this time I decided to create a language/runtime with pre-emptive scheduling using instruction fuel. While I always do freestanding builds, this time I decided that I also wanted to support native FFI.
That is when I realized the true horror of (g)libc. It wants to inject itself at the root of the library/program and everything from threading to dlopen/dlsym is impacted. I tried a lot of workarounds including trying to implement a loader myself, but the complexity (and fragility) grew so much that I felt it was not worth it.
Finally, I retreated into the safe world of a freestanding runtime + syscalls. FFI, if it has to happen, will occur via IPC of some kind. A second process linked against glibc that will manage calls on behalf of the clean first one.
I've implemented the same thing for micron (more or less). One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE). That being said the way you're doing is also tricky(ish) because if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you. Doing this is safer if you control the entire runtime.
> One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE)
Yes, it's not trivial, but I hope that over time everything will settle down.
> if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you
No, I hook this to musl, which is statically linked into my binary, and I have complete control over it.
I have faced a similar issue in the past, and I don't understand how static binaries from the host are supposed to solve this.
From what I remember, GPU access on Linux 'works' by accessing specific FDs under /dev, which are vendor specific - this is what these libs do under the hood.
The libraries don't have any magic powers - if the FD is inaccessible, you won't be able to do anything.
So there's some vendor specific access needed in containers anyway (or a blanket allow, which is a BAD idea).
Also not sure why dynamic linking isn't good enough for this - the issue lies with the permissions, not how you load/link libraries.
i'm not sure how much people realize that the modern "graphics driver" is actually just "the kernel multiplexes userspace messages to/from the GPU and we've taught mesa to understand each family of GPU you'd ever care to support."
i helped somebody get Doom running on an old embedded system running some 4.x kernel. we just built everything, including mesa, statically and deployed that. imagine building everything static but loading the system's libgl.so instead (which was probably just a symlink or abstraction over mesa's own implementation). what's the benefit: we'd get older, less optimized graphics routines from 5 years ago?
if you're statically linking, then just bring your own graphics "driver". the kernel interfaces are stable enough. it's not conceptually different than embedding `syscall`s directly into your application the same way you do when statically linking libc.
> i'm not sure how much people realize that the modern "graphics driver" is actually just "the kernel multiplexes userspace messages to/from the GPU and we've taught mesa to understand each family of GPU you'd ever care to support."
In the case of OpenGL, it has a fairly sophisticated state tracker, which you can carry around with you. For example, I can statically compile ANGLE this way.
But even OpenGL/Vulkan have a shader compiler that's unique to the hardware (it's expensive to carry it around with every piece of hardware in the universe).
And I'm not even talking about libdrm, which is quite tightly tied to the kernel version.
I know exactly what I'm talking about, because I can also statically compile Mesa into my application.
I don’t think I’ve said it out loud more than a couple times in my life. But in general I think I spell out / pronounce the “dot” in file extensions unless it’s completely obvious from context.
Technically, you're right, it's a dynamic loader. Technically, it's pure dynamic loading.
If we look at the issue at its core, we're still a statically linked program in a hostile environment, forced to dynamically load device drivers from the system.
It's similar to Golang; on MacOS, it has to use libSystem, even though otherwise, these are the statically linked Go binaries we're used to and love.
Let me add a little more detail: if I use vdso with gettimeofday in a statically linked program on Linux, am I still a statically linked program, or not? :)
if you really want a single elf that could link to system libraries with a foreign libc (you shouldn't), the somehow correct-ish way to do it is map your preferred ldso manually , setup stack, call DT_ENTRY etc etc like how kernel does it, and just yank the libdl symbols.
It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens).
Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).
How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.
I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
Not sure what you’re trying to show with that bug report but it’s not a case of cross distro glibc issues. If I read correctly it’s a vanilla behavioral change that exposed preexisting UB in flash.
Not sure how the comments about alpine or bionic relate either to my claim that cross distro glibc is fine.
It depends on how we define the ABI. I see it as a set of client-visible invariants that they rely on. In my world, glibc changed the client's visible invariants, breaking the client. The client works on one glibc-based host, but not on another. What is this if not "a case of cross-distro glibc issues?"
Overall, both of our points of view on compatibility were discussed well in that thread; we probably shouldn't repeat ourselves. :)
ABI is not "whatever happens to work with this distro" but "what programs that comply with the API contract compile to". Overlapping memcpy arguments is an API contract violation and thus not something covered by the ABI either. This distinction is the entire reason why C has a separate memmove function. You can't just make up your own imaginary ABI contract and then blame the system when it doesn't fulfill it. That's going to result in self-inflicted plain on any OS.
Even Linus’s comments in that bug hedge things a bit:
> and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to.
(Emphasis mine.)
In the Flash case, they changed the direction memcpy works in, in a way that didn’t actually improve any performance (Linus did benchmarks to show it) and Linus’s point was that there’s no upside to the change, and only downsides, so on the whole it makes no sense to keep the changed memcpy.
You could imagine plenty of scenarios where existing behavior truly is broken, and show that no popular software depends on the brokenness (which is why distros are in a great place to do these kinds of smoke tests, they have thousands of packages they can test), and make a judgement call.
A “zero regressions no matter what” policy is impossible in practice due to Hyrum’s law, at some point you have to use common sense and draw the line somewhere sensible.
Did the distro apply a patch to glibc that broke things that others did not? Or did they pick up a newer version of glibc that other distros hadn’t?
Yes, different distributions run on different versions of glibc. And different versions of glibc may have subtle differences in runtime behavior. That’s still ultimately a glibc back compost issue, not some kind of bug specific to glibc.
This is not true. Glibc supports symbol versioning. You can use it to select old versions of used symbols. The result is a binary that can work on 20 year old distros the same as on the latest, compiled with latest compiler and Glibc.
You can also compile using old distro and old Glibc to get similar effect. Though you would miss the advances of the newer compilers.
I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
1) Why should I limit myself to the available APIs?
2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
Luckily Zig makes this quite easy to do. In Mach[0] we are able to just `zig build -Dtarget=x86_64-linux-gnu.2.28` to build GUI apps against an ~8 year old glibc version for maximum compatibility.
This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support.
Complete failure is strong words when lots and lots of Linux boxes are running just fine.
I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date.
Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue.
The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros.
When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works.
You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work.
It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want.
I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.
Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs.
But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
And we can't count on someone else packaging our work until we become incredibly popular.
> But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution.
Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those.
> And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Due to FUD, mostly.
Sane people do just link with an old enough version of glibc.
Ah yeah my mistake static != no external dependencies. I was thinking of golang static binaries (built without CGO) which do not have a dependency on glibc at all.
How are we supposed to take this stuff seriously if the author (sic) isn't even willing to write the readme? Claude exists! If I want some slop I can push the button myself.
README.md was written by me, and I, of course, used claude/codex for it. In general, I do everything through claude/codex, the reasons are described in https://github.com/pg83/solo/blob/main/CONTRIBUTING.md . And no, it's not low effort, and no, I don't see the point in wasting time de-claude-ifying the text just to avoid it looking like I didn't spend enough time on it.
I believe usually when someone complains about text written by a language model they are hoping to read human-written text instead of human-laundered LLM output.
You make it clear why you write all your code through a llm. But a README is not code. Presumably you would like people to read it. A machine authored readme reflects poorly on a project.
The author of the README is me, the machine just wrote it.
I am not a native speaker of English, my written English is simply terrible, no one wants to read the text that I wrote exactly :))
For context, I read Claude output every single day, I know what it is reliable with and what it is not. Or at least I have a feel for how much I can trust it.
It may not be clear to a non-english first language person, but when I read Claudes documentation, my brain immediately picks up claude-speak. Therefore, I expect the code to be generally correct, maybe, depending on how specific I was during my prompting. In no way shape or form do I really trust it, at least until i dig into the code and validate my mental model. And query the review for edge cases etc...
By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.
To a degree this is unfair, as it is like judging the quality of someone's work based on their accent.
But Claudes accent, has a high correlation with Claude, so unlike with people, where an accent has no bearing on technical ability, Claude being Claude does.
I would much rather read a typo ridden sentence than claudism, even just as a forward, explaining what you did vs the ai, and telling the users how much we should trust it.
Also, If you really insist on using AI to write, have another model rewrite docs/comments into regular English (opus 4.6 for example is much better than 4.7, 4.8, or 5.
It is open source so you can do whatever you like, but people (especially native English speakers) will discount your work, because Claude, especially opus 5 writes very very badly.
The people who say they would rather see a typo infested mess of a readme are serious. Or even just write in your native language and then have Claude translate.
Both of those are better indications of proof of effort than a Claude readme.
> By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.
I'm sorry, but I agree with the author: if a certain writing style makes you associate the work with low-quality, then that's your problem. The author shouldn't have to rewrite the readme just to avoid triggering your automatic unfounded associations. If you look at the substance of the work, including the test cases, then this is clearly not easy work that can be vibe coded in a single pass.
It's just like emdash. Everybody digs on how it's a signifier of LLM text, but I've used emdash for years because it's gramatically correct. I shouldn't have to stop using emdash just to avoid kneejerk reactions.
if 99 poor effort/quality projects have a readme that reads in a particular style, then you expect the 100th project with a readme in the same style to also be of poor effort/quality
statistically, it only makes sense for your expectations to immediately be low when you encounter this writing style because there are just so much slop out there
the author is free to keep that writing style but they should be aware that this will—at least on the surface level—make their project look exactly like the metric ton of slop we see posted everyday everywhere
That's like saying a large percentage of X-colored people are criminals, therefore when we see an X-colored person it's reasonable to expect him to be criminal. As a society we have decided that heuristics like this, and prejudicing individuals based on statistics of the group, are not acceptable.
Also, what do we call it when people don't look beyond the writing style, such as at the code, or at the rigorousness of the tests, or at other substances beyond writing style? Slop assessment?
LLMs often use emdashes in a distinct incorrect way. It's not just the existence of any emdash, although considering you were in a very small minority of older users, it now warrants increased scrutiny, unfortunately for you.
In any case, it's open source. If you don't like something, even if the project seems generally useful, go ahead and fix it. The PR came in. I'm an engineer and I can write good code, but that doesn't mean I can write good README.mds!
I almost always write my own prose, and when I have an LLM write me prose I edit it to make it my own, in my own style, and pithier than the LLM would make it.
I'm not saying everyone must do that. For me it's partly a point of pride, and partly empathy (editing for conciseness is showing empathy for others' time). But I am saying that people should consider doing it.
Disclaimer: This is a user’s perspective rather than a programmer’s perspective.
valid point. I am usually okay with LLM generated code since even if it might not be architecturally sound It is usually well commented and has tests and documentation for helping another agent/human debug any issues.
But, just the painful experience of debugging any dlopen related crashes and/or intermittent bugs; and the sheer amount of tokens burnt by an LLM chasing tangents when shown a stack trace; I wouldn’t touch this at least as a packager/consumer of certain apps for personal usage on older distros.
So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.
Modern models, when properly managed with a human in the loop, write higher-quality code than humans and introduce significantly fewer bugs.
Therefore, it's quite the opposite - you should expect fewer "dlopen-related crashes and/or intermittent bugs."
> So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.
The Linux ecosystem already settled on containers to solve the problem. Namely Docker, Flatpak, Podman ec.
They require admin privileges. A consumer on an enterprise servers might not have them, which is where I would guess glibc might not be latest one. If you are an admin then sure those solutions can be feasible.
Mapping parts of files into executable memory, and then executing them, had better be bulletproof! Exploiting this seems like a direct path to RCE, and it's likely that this sort of library is used by privileged code.
Purely academically, this is a very cool piece of code! Just hoping that it gets a thorough vetting before used by privileged/security-critical software :)
Well, ld.so already does this, and it's no big deal. The Python interpreter also does this when executing a .py script (code is code, whether it's machine-readable or human-readable).
In any case, we take testing very seriously—every glibc shim we've written is covered with tests, and we run our loader against 1000 of the most popular Debian packages. The project has 100% code coverage. Perhaps, if I have the time, I'll also do some fuzzing on this thing.
If you want to load the OpenGL/Vulkan vendor driver then unfortunately you don’t have much of a choice: those are linked against glibc, and I believe generally also against libwayland so screw off if you want a different protocol library (I might be wrong about the latter part). If you instead want to load plugins or whatnot into your statically linked executable, then personally I’d argue that you shouldn’t be emulating Linux dynamic linking semantics at all, because the whole late-bound global namespace thing is silly and wrong. (Solaris, which is where Glibc took this model from, moved away from it[1] as much as compatibility allowed, and so did Darwin[2], whereas Windows never made the mistake to begin with, but Glibc persisted and Musl copied it.)
This is a big forwards-compatibility risk. Suppose glibc adds a new symbol, and then a GPU driver adds a dependency on that symbol. The user wants to run an old executable with the updated GPU driver (maybe the old GPU driver doesn’t support their GPU). Normally, this would work fine: the user has to use a new copy of glibc, which will be compatible with both the new GPU driver and the old executable. But with your approach, the GPU driver is forced to use the glibc reimplementation which has been statically linked into the executable. Which, since the executable is old, can’t possibly implement the new symbol.
The same issue would occur if glibc adds a new version of an existing symbol and then the GPU driver is recompiled. (Or, for that matter, if a GPU driver adds a dependency on a symbol which glibc has always supported but which isn’t in the subset that you reimplemented, though in theory that could be solved if you reimplemented 100% of the symbols.)
Yes this is just asking for trouble - and all it does is solve a problem that doesn't really exist. Just dynamically link against the oldest glibc you want to support. Its annoying that Linux toolchains don't have built in easy mode support for that but its much easier to deal with than this thing will be when it breaks.
It's also not just new symbols, the loader semantics also aren't static and new enough libraries may not support older semantics - e.g. the loader used to use DT_HASH entries for symbol resolution but now they are no longer present on all distributions.
This is not a desirable solution on musl based systems. Whatever you do in that situation ends up horrible, so it is about finding the least bad solution. Which this seems like a workable variant of.
> Just dynamically link against the oldest glibc you want to support
I wish it would that simple for practical use cases.
I ship professional software for colorists for Hollywood studios and they absolutely love to never upgrade. We have to ship for RockyLinux 8. Sad.
RockyLinux doesn't sound so bad. It's not even EOL.
It's just this short of being EOL.
Almost three years?
>Rocky Linux 8 is supported by the Rocky Linux project until May 2029.
https://forums.rockylinux.org/t/what-is-eol-of-rl8/3316/3
Yes but only the very last version and only paid support. Don't mind to pay, it's just that no amount of payment would be a proper solution to ancient environment with everything.
We still have RHEL7 hosts in production. Every day it gets harder to find packages with a glibc 2.17 floor.
(I'm certain you know this already but I'm posting this for people who might want more context)
Part of the problem that people don't really realize is that it's not practical to just 'link against old glibc'.
In order to ensure compatibility with RHEL7 we need to link against RHEL7 libraries, which means we need to build on RHEL7. That means old, unpatched versions of glibc, libpcap, libcurl, openssl, or who knows what else, in case there's some backwards-incompatible change in newer RHEL versions.
Alternately, we can build/patch the libraries we use and compile them statically into the binary, which is vastly more maintenance work for us for very little benefit.
Meanwhile, compiling against older versions of libraries like glibc means we don't get the benefit of updates; not just new features in glibc, but things like more/better SIMD support in glibc algorithms, more/better optimizations in GCC, and so on.
Alternately, we can stop supporting RHEL7, like Redhat did, and build against RHEL8... Or we can build a separate version against each version of RHEL we want to support.
Funnily enough, we've been shipping RHEL7 RPMs of our product for years, and only recently realized that they won't actually run on RHEL7 because at some point the toolchain updated and now we're compiling against too new of a glibc version and then packaging it into a RHEL7 RPM. It wouldn't have worked on RHEL <8 for the past... few years? But no one uses RHEL7 so it went completely untested for ages and we didn't get any customer complaints.
Now we're re-labelling our RPMs as EL8, but it's just a cosmetic change so that we're claiming the version that we actually require.
static link everything? ship a docker image? ship your own userspace (like Oracle DB used (?) to)?
Yeah, to make things spicy, what we ship is a plugin, so there's limit of how static we can link. glibc stays dynamic and that's a huge limiting factor.
I've got plans to try to build on RockyLinux9 and package its glibc along with the app. Simple helloworld works, so I have a glimpse of hope that it would be possible to ship like that.
Neither of those fixes issues with system-calls not existing because the linux kernel was so old.
Something like https://github.com/wheybags/glibc_version_header may be of interest to wean off using older OS’s in your build system.
Indeed this is interesting. Thanks, appreciate it!
> Just dynamically link against the oldest glibc you want to support.
Just the other day I tried running an older binary and it failed with a glibc error, despite it being linked to a glibc version that's barely 5 releases behind the one on my system. So maybe glibc isn't backwards compatible after all...
Or maybe you misunderstood the error message.
Or may be not.
This means the user will update the binary with my program and continue using it happily.
I'm not offering a silver bullet, but the approach I've implemented is much better than what the industry currently offers.
Exactly. I think you're doing fine. It's weird that you're using C++ here, since that brings in a lot of crap, but then your programs are probably C++ anyway, so it's not a problem.
Honestly I'm impressed by this work. It's a thankless thing to have built, it's definitely feasible to build but more work than almost anyone would have been willing to do. What would be even neater is if Musl had this builtin.
So we are re-inventing patched a.out files, back when UNIX systems started to introduce dynamic loading, before ELF was invented?
Advocates of static linking keep forgetting once upon a time UNIX only had static linking, then we had overlays, and eventually dynamic linking came to be.
Of course, I remember those times very well.
And I also remember very well that dynamic linking appeared ONLY because we were catastrophically short on memory; everything else was added much later.
Now we have plenty of memory, and we can very well return to our blessed roots!
Not at all, the pain of doing plugins with UNIX IPC was another one.
Ah, you have lots of memory, we're very wealthy. /s
There is no RAM shortage in Ba-Sing-Se
I don't know much about musl.
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?
musl does not perfectly emulate all aspects of glibc, so trying to use libraries that assume glibc can sometimes lead to problems.
On the one hand, this is technically true, but on the other, what serious issues do you know that will cause problems in practice? I run tests on 1,000 of the most popular Debian packages.
Mostly about precompiled libraries (proprietary software) and libraries and software that use GNU extensions.
Not anymore, but for -years- it did DNS wrong because of the author being pedantic about an RFC wording.
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ ABI all depend into each other. You cannot change any of them independently. All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library and make calls like dlopen.
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
In what sense do binary interpreter / loader, GCC compiler, C library and system C/C++ ABI dependent on each other? I have certainly mixed different versions of all these components without problems so far.
When you compile GCC you need to provide a full glibc installation as your target. It is also a dependency of libstdc++.
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
I can't comment on the C++, I can imagine there plenty of issues, but for C I don't see this. You need some libc if you compile with gcc, but this generally does not introduce a hard version dependency on the specific version (there may be a minimum requirement if you compile against a new version that a symbol with a different ABI).
For example, the itanium unwind ABI implementation lies between these three entities.
The interpreter/loader is glibc and a key part of bootstrapping an executable built against glibc is loading libc itself before continuing on to load the program. Versioning is a problem when distributing binaries linked against a newer glibc to distros that ship an older one. The C compiler doesn't really care as much.
> The C compiler doesn't really care as much.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
It depends on functions defined in a standardized ABI.
You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well.
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
There is no libc independent ABI. ABI doesn't purely mean just calling conventions.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
Sorry, again this too vague for me. What is the exact problem with atomics and thread_local in C that would make the ABI dependent on a specific version of glibc? I know the ABI is not just calling convention and e.g. for atomics may involve calling a function from libatomic. But from my understanding, this is all part of a standardized ABI that does not change and can be provided by different implementations.
Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
Here read it from the lion's mouth: https://wiki.musl-libc.org/design-concepts
> Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
You still have functionality like `dlopen` with C or `pthreads` with C. ld-linux.so is the thing that prepares the stack frame, or the segment pointers (FS_BASE on x86_64).
When you have your main loaded __libc_start_main_impl is called from glibc and if you disassemble it you'll see this line:
call 0x7ffff7c28790 <_dl_audit_preinit@plt>
Guess where _dl_audit_preinit lives?
(gdb) info symbol _dl_audit_preinit _dl_audit_preinit in section .text of /lib64/ld-linux-x86-64.so.2
Moreover glibc has "magic" sections like .init_first. Only ld-linux.so that is compiled from glibc source knows how to handle that. https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/...
> I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
It didn't break, since glibc hasn't broken backwards compatibility of ld-linux.so and glibc combination lately. Last time they broke it was in 2024 for a really specific subset: https://sourceware.org/pipermail/libc-alpha/2024-December/16...
A breakage hasn't been observed doesn't mean that there is no tight-coupling between ld-linux.so and glibc that is compiled from glibc source.
If you want a really specific scenario:
- Write a very simple C file that contains a global (volatile if you want to stop the compiler optimizations) thread_local variable with C23 syntax
- Compile a simple .so file on a GNU distro targetting glibc ABI with GCC (pass -std=c23)
- start up a Musl distro (e.g. Alpine Docker image), copy that .so file in
- Write a C program that dlopens the GNU .so file
- Try accessing the thread_local variable you defined
- Watch the world burn
Thanks, I need to dig into this more. But I still not convinced there is a real issue here. Of course different infrastructure needs to exist for different feature. What exactly is the underlying issue with thread_local in your specific scenario? It seems Musl does not setup the the infrastructure needed for ABI-compliant thread-locaL data access?
THERE IS NO STANDARD, AGREED ON, RFC'd or EVEN BEHAVIORALLY DOCUMENTED ABI FOR MUSL TO FOLLOW!
It's not Musl's fault that Glibc has an undocumented, badly designed, tightly coupled implementation. It's not Musl's fault when they decide to not follow Glibc's design which they have to reverse engineer and rewrite from scratch. Then Glibc can break it in a future version anyway. They give 0 guarantees and have a track record of breaking things.
Sorry but I feel like you're trying to excuse Glibc out of their terrible design. Glibc didn't ask anybody for standardization. Glibc didn't document their behavior. Musl doesn't need to follow any undocumented behavior. Musl has its own implementation of thread_local variable and it works only for Musl dynamic binaries (which you cannot load with Glibc's ld-linux.so either). You need to tell your compiler to generate code that Musl prefers not Glibc. So effectively we have two ABIs.
There is no standard and any design that tightly couples the system binary loader to the C standard library is just extremely terrible design. Both Musl and Glibc do it. Both of them are terrible. Glibc was first and it set the horrible, binary hostile behavior "have no standards, trust Glibc, and recompile your programs again" as the "standard".
The entire Linux userspace depends on Glibc's terrible design. Due to Hyrum's Law, the exact needs of the complex and emergent behavior can only be surfaced via making a better designed, completely independent system loader and a completely independent libc and then fixing the entire tens of thousands of userspace libraries in the upcoming decades. It is just Sisyphean work to fix Linux userspace.
There is no simple "let libc-X obey the standard S" solution. There is no standard. Linux Standard Base project tried to set a standard. It failed.
I'll won't further discuss this with you. Maybe you have good intentions, maybe you're playing dumb. I'm not sure anymore. If you're the former, I'm not going to deliver every single bit of information. I provided enough resources and you can learn.
sorry you are just ranting. I was asking for specific details. The documentation for TLS seems this: https://www.uclibc.org/docs/tls.pdf
As a naive third party, wondering the same things he was at each point in the conversation, I think you definitely read him wrong.
But, thanks for spending the time! I learned a bunch!
Unless you use musl-gcc which is a proper GCC frontend with musl libc sysroot. The compiler is indeed (too) tightly coupled to its sysroot but linux is not special here. If anything interacts with the CRT it needs to know which CRT to use.
I'm not arguing that there aren't problems here, just that I think people have different expectations that are a little extreme. The libc implementation is a core part of the operating system, if you want to target that OS you should target that libc. Some distros (alpine) prefer musl libc, most prefer glibc. And distros really are different operating systems.
I don’t imagine that you’re unaware of any of this, but: ld.so and libc.so are heavily interdependent in deliberately undocumented ways with Glibc and outright the same file with shared Musl. And while you might usually get away with using any old GCC with the right architecture and ABI (especially for C; cf the musl-gcc hack), technically it needs to be built to target a specific libc version (particularly via symbol versioning; I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases). The C compiler of course is tied to the C ABI, and this wouldn’t be really worth mentioning except for the time where the GCC devs accidentally the whole SysV i386 ABI and pretended that the stack was always 16-byte aligned, why do you ask, except on RHEL. The C++ parts I can’t really comment on.
I am not really sure. For ld.so and libc.so I may believe this. The C ABI is very stable, and if you use a new symbol from a newer glibc, you certainly depend on it, but this can also be avoided. In any case, I do not see what is fundamentally misdesigned here. I can't quite image how it could work differently. If you upgrade something so that the e ABI changed you natually need to update other components. Static linking certainly seems a very poor replacement for this.
> I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases
And that's the correct approach and also one that many have taken. We just need someone willing to maintain that as an easy mode SDK for everyone.
> And that's the correct approach.
Who said that? This approach has many problems that have already been discussed here, not to mention the fact that it leaves Alpine and Bionic-based systems out in the cold.
Let me remind you that Bionic is the most widespread libc in the Linux world, and Alpine is the most popular Docker layer.
The world doesn't end with glibc. And it doesn't begin with it.
It could be fine if you care only about free-software desktop users, and not Google's propriety platform and hyperscalers.
> system C/C++ ABI
C++ abi should not be included in this. It is independent from the other pieces and historically a source of incompatibility on its own.
Saying "C/C++ abi" as if they are the same is looney tunes, the former is very simple and stable and the latter is very complex.
See: https://news.ycombinator.com/item?id=49355262
How libstdc++ initializes global variables absolutely depends on glibc and ld-linux.so. That is part of C++ ABI.
That's libstdc++ depending on a C ABI (or a GNU ABI to be precise) but that doesn't make it a C++ ABI. You can also have global constructors in C with GCC.
> the former is very simple
Somehow most of my portability issues seem to be caused by glibc, its symbol versioning and close ties to the dynamic loader. Minor versions aren't compatible, no two Linux distros ship the same version and you can't just provide your own without also patching in your own dynamic loader.
At least as far as the defaults on Linux go I consider C the root of all evil.
AFAIK glibc is backwards compatible as long as a) your program is running on a newer version (i.e. you're not trying to dynamically link against an older version) and b) you're not using hidden/undocumented symbols.
In which case as long as you're using the documented public API and compile your program with the oldest version of glibc you want to support (some Ubuntu from 4-5 years ago should cover pretty much every current desktop) you should be fine. And with something like Docker this is trivial to do.
Sure it is annoying that you cannot use your current distro (especially if you use some rolling distro) to make binaries for everyone, but it takes very little effort to work around that. The only issue i can think of is if you absolutely want to compile using the latest version of your compiler and you cannot build the compiler from source to work in the Docker (or whatever) contain to work against the older glibc.
This approach only works for very simple cases.
In complex cases, it turns out that the old version of glibc also pulls in other libraries and the compiler, and you're stuck with a very ancient sysroot. You may often find that you can't compile new library versions in such a sysroot and link them statically.
So, it looks good on paper, but forget about the ravines.
Still very simple compared to c++.
While I don't disagree with some of the pain you describe, you conveniently gloss over the fact that gnu developed a system that worked, and then made it free to everyone to consult and use.
BSD also did it. They did it better. Maybe more modern but AOSP also did it but at a different level of binary: instead of ELF, using compiled Java bytecode archives.
The root cause is alternative libc implementations. Are BSD syscalls considered stable? I remember Go moving to use libc on OpenBSD. Solaris also has the libc as the stable interface. Linux kernel is an outlier here guaranteeing stable syscalls but you wanting to use another libc is not Glibc's problem.
Yeah it's not Linux specific but rather true for most Unix systems that libc == base system interface library.
Except the other UNIX systems, starting with the original one, evolved from only having static linking to various ways to connect libraries and applications.
> all shared libraries depend on the specific glibc version to load them
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
See https://news.ycombinator.com/item?id=49355262 .
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
That's what I'm saying though, glibc-linked binaries are forwards (but not backwards) compatible.
It is not just compatibility. You cannot load them into the memory with your system dynamic loader. You need to also ship ld-linux.so with the new version of glibc you have, if you were to distribute your program independently.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
Windows doesn't even have a concept of a loader binary right? I think its hardcoded into the kernel/win32 itself.
Windows loader is part of the system ABI, and programs' libc(s) are loaded by it.
One of the ways Windows manages to support multiple libc's is by being careful not to mix allocators; if a system API you call allocates on your behalf, your libc can't free it, the system API will offer a function to free it.
Windows loader is certainly available to user programs though; LoadLibrary has been around longer than many developers.
LoadLibrary will also not be able to load arbitrary libraries compiled for newer versions of Windows though. Just like with gcc, Windows also does not guarantee forwards compatibility - because that would men freezing the feature set the system libraries provide.
Windows handles this much better than any other OS. The API passes versions (== structure sizes on the calling side), and WinAPI can handle that.
As for older Windows systems not being able to load new DLLs, they can; the format hasn't changed in a very long time. I've had experience with installing some DLLs on Windows NT 3.51 and running a modern Firefox, which is about 20 years behind the times.
Targeting an older version of Windows is "set a define so that the system headers don't expose functions that didn't exist on older Windows". You don't need an old toolchain to target old systems, you ask the newer toolchains to target it.
MS haven't made this work arbitrarily far back, I believe they deprecated targeting Windows XP in one of the more recent toolchains, but that was purely a "not worth supporting" situation.
EDIT: mixed up talking about older & newer
It kind of does but in a "it is an implementation detail that you cannot access and it doesn't matter" sense. Unlike ELF, PE32+ files do not have an "interpreter" that you can specify.
csrss.exe handles some of the loading functionality: https://en.wikipedia.org/wiki/Client/Server_Runtime_Subsyste...
Why? If you compile your program against an older glibc it will run on newer glibc. glibc is the loader.
Why? What kind of single-responsibility-principle that predicates the C standard library to become the both syscall interface and the binary loader as well?
What happened to the Unix principle "do one thing and do it well"? Windows solved this quite modularly and it works perfectly well.
Glibc is terribly designed and it is the core component of almost any Linux userspace app today. Let's not try to find excuses for horrible designs.
I don't disagree that the loader and the libc should be decoupled, but it's not as simple a split as you're making it out to be.
The reason they're tightly coupled is because there needs to be a bridge between Sys-V ELF ABI and the C runtime, and it makes sense for the implementation of that bridge to use libc itself which means special-casing the bootstrapping of libc during program loading.
glibc isn't unique here btw - musl libc ships its own loader that does the exact same thing as rtld (parse auxv/envp off the stack, relocate itself and load libc, initialize the CRT, then load the rest of the program's SOs in order).
Even if they were fully decoupled it is not unreasonable for software targeting newer versions of an operating system to only run on that version or later. Linux is actually the outlier here where the syscall interface is stable enough to make this a non-obvious problem.
There's a universe where you can load multiple versions of the same libc into the same address space but that isn't ELF, and it's not common. aiui the only reason PE can support e.g. multiple versions of malloc/free in the same executable is due to two level namespaces in symbol relocation tables.
---
edit: another observation is that for a functioning ELF loader you basically need a three stage bootstrap. Stage one parses the stack passed by the kernel and performs self-relocation. You need that to make non-static function calls and read/write global variables. Stage two loads libc and initializes the CRT with stuff like environment variables, aux vector, any dso for syscalls passed by the kernel, etc. That has to coupled because POSIX specifies stupid stuff like "getenv" and "argv" requires the loader to know how to move from the stack passed by the kernel to the memory reserved by the CRT that gets used by the final executable. Finally in stage 3 it can relocate and load shared objects and run any constructor functions pre-main, before jumping to start where the precompiled CRT start (then jump to main goes).
So if you want an ELF loader on Sys-V you probably need to couple the loader to the libc, just for libc to work at all. Unless you want to specify a bunch of behavior and binary layout in the C standard.
None of this is designed by glibc, it's the simplest way to go from exec to main.
> All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library
That's currently the real core of the problem.
The loader (and libdl) need to be decoupled from the glibc itself under Linux.
Without that, any attempt to ship static binaries (or any binary with a different Libc) will be a source of perpetual pain.
nss plugins and its associated pain (sssd and avahi) are an other examples of that.
libdl is the loader FWIW. It's all one big glibc codebase.
What we name the loader is typically ld.so, more precisely /lib/ld-linux-x86-64.so.2 under x86_64.
But I do agree with you, the glibc is all one giant codebase shipped in one giant blob, dispatch under several names.
Even libdl.so is barely more than a wrapper and some glue around the libc.so itself.
In what way are they "broken" when Linux runs fine on millions of boxes? Sure, it might be a pain for proprietary software, but if your app is open source it's not that hard to build it on whichever distro you want. If your app is popular enough the distro maintainer will build it for you
Yeah, it's mind boggling how everything hard depends on glibc, even critical graphics systems.
I've become obsessed with getting rid of it, especially after I realized that contributing to GNU itself was a dead end. Freestanding Linux programming turned out to be much more fun anyway.
All libraries out there should adopt the SQLite design: programmers provide it with all the necessary functions. Instead of libraries hard depending on glibc, we get to inject the libc-ish subset it needs. Then we can use whatever we want under the hood. I'm working on porting SQLite to freestanding Linux system calls so it can run with zero dependencies. Wish I could say the same for software like mesa, I'd need a lot of help for this one...
Modularizing (and versioning each part independently) libc would go a long way. There is also a need to separate the stuff needed for system integration with what is necessary for users of the C language to actually do stuff.
> Modularizing
Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version.
> what is necessary for users of the C language to actually do stuff
Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all.
The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me.
Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.
> A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
Great choice for small programs, but what if I want hardware accelerated 3d?
exactly
Yeah, that's the annoying part. Been wondering about this for years, and graphics support was among the first issues raised on the lone lisp GitHub repository. At this point I've even started exploring the mesa codebase, made some patches but didn't submit them yet due to the AI stigma.
With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering.
For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it.
Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
Vulkan is designed to only load core statically and the whole extension spaghetti dynamically, using driver entry points.
https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfac...
Modern 3D APIs are built around passing around a few buffers per frame and long command queues. Should be doable with IPC.
> Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space.
And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
> You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space.
That only massively compounds the problem.
> And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
NT system calls are not stable. You still need to link against ntdll.dll at the very least, like a forced Linux vDSO.
> That only massively compounds the problem.
The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
It's not a problem in the same way that things like snap or flatpak aren't problems. It works but it bloats things up considerably and makes you wonder where it all went so wrong. I mean, dozens of slightly incompatible runtimes inside a single process?
Those incompatible runtimes are separated by a linker that doesn't resolve all symbols globally, but rather scoped to the shared object they're expected from. They all coexist happily, and if you're so inclined you could resolve the same symbol from each, if you had reason to do so.
> Windows explicitly does not want you to link the system libc
This is categorically false; UCRT[1] is a thing. The 'U' stands for universal. Unlike Linux, Windows allows developers to choose their ABI boundary and also ship that boundary if they desire, or use the 'system' one and ask older platforms to install redistributables or Windows update packages. There's the old and new C runtimes in MSVCRT.DLL and UCRTBASE.DLL, the C++ runtime in VCRUNTIME140.DLL, Win32 in KERNEL32.DLL, USER32.DLL and more, and then the stable-ish kernel interfaces in NTDLL.DLL, in order of 'closeness to the kernel'.
And also, 'libc' is a UNIXism; on Windows the term is CRT, for 'C runtime'.
[1]: https://learn.microsoft.com/en-gb/cpp/porting/upgrade-your-c...
By "system libc" I'm only referring to msvcrt.dll, found in the Windows directory. Quoting Raymond Chen:
> At some point, the decision was made to just give up and declare it an operating system DLL, to be used only by operating system components.
https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...
I think the grander point is that there is no real 'system CRT' on Windows; as I mentioned, there are multiple entry points each at an appropriate level of abstraction available to both platform and application developers (not that there is a real difference between the two, since platform developers may also write applications like Office). Many Windows platform libraries (WIL, for instance) themselves use UCRT instead of MSVCRT now. The latter exists, but it is by no means and has not ever been by any means the single entry point to the Windows platform, unlike glibc on most desktop Linux distributions.
You can completely skip all CRTs on Windows and still have a program that can call dynamic loader and other Win32 APIs. Here is a tutorial: https://nullprogram.com/blog/2023/02/15/
When you link with UCRT your specific program just gets a "view" of standard C functions. You can load DLLs that were linked with MSVCRT from a UCRT program. It just works
That's a pretty great article. Re. the stack probes, it's not hard to supply your own; Raymond Chen recently did a blog series on stack probes, the entries for amd64 and arm64 are here:
amd64: https://devblogs.microsoft.com/oldnewthing/20260319-00/?p=11...
arm64: https://devblogs.microsoft.com/oldnewthing/20260320-00/?p=11...
That was introduced in Windows 10, Windows history is a long one.
Usually Windows developers will use Win32 directly, ZeroMemory() instead of memset(), and so on.
Indeed. I was just about to point out the fact that the CRT only provides the C standard library; Windows applications can run perfectly fine without one by linking only to the Windows API DLLs.
> Unlike Linux, Windows allows developers to choose their ABI boundary
Linux has a stable system call ABI. I can trash the entire user space and boot Linux with init=/my/program if I want.
It's not that Linux doesn't allow developers to choose, it's that Linux doesn't actually control the userspace ABIs. Userspace is whatever we want it to be, and people settled on "GNU/Linux".
> Linux has a stable system call ABI
This is a noble but practically worthless ideal. Hardly anything is written against the kernel ABI except glibc itself. End-user application developers don't write against the kernel interface. Platform library and application developers like KDE and GNOME don't really write against the kernel interface either. Nowadays even driver developers don't entirely write against the kernel interface, instead also linking in glibc and lib{std}c++ for things like shader compilers (which is nowadays usually just a fork of Clang). Like it or not, glibc is the singular entry point for desktop Linux, which is the point of essentially every comment under this post.
That 'desktop Linux ≡ glibc' is something that I daresay the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU. Linux's unstable user-mode is the entire reason for Docker's existence. It is bodge upon bodge and has led to much gnashing of teeth. The GNU/Linux community (I don't care for Stallman's meme rant here) have to grow a backbone and understand that an OS is much more than just a kernel, it is a platform, and currently the Linux platform is a disparate set of communities with highly divergent goals.
That's why I've always said the best Linux is Android, because Android is a platform.
> Hardly anything is written against the kernel ABI except glibc itself.
I'm working hard to change that.
Here's an entire lisp interpreter written in freestanding C using nothing but Linux system calls:
https://github.com/lone-lang/lone
It has a system-call primitive too, which means it can do anything.
Here's a TCP echo server written in lone lisp:
https://github.com/lone-lang/lone/blob/master/examples/tcp-e...
It works, and you don't need libc.
> Like it or not, glibc is the singular entry point for desktop Linux
Well I don't like it, and I've made it my mission to fix it.
Maybe lisp isn't the answer, Rust is. I've revived my liblinux crate too.
https://github.com/matheusmoreira/liblinux
I'm going to add system calls to this thing every day until I have every single one. Then I'll use them to make all of my applications.
Already have my first project planned: a custom Rust network stack for my QEMU virtual machines so I can firewall them off in userspace without root access.
> the Linux platform
The linux platform is these system calls. Everything else is userspace. This is exactly what makes it special.
https://www.matheusmoreira.com/articles/linux-system-calls
> The linux platform is these system calls. Everything else is userspace. This is exactly what makes it special.
This is exactly the attitude I meant by 'the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU' and 'an OS is much more than just a kernel, it is a platform'.
By 'platform', I mean windowing toolkits, such as Wayland. Audio libraries like ALSA and PipeWire. Desktop idioms and libraries like DBus, and the entire freedesktop.org effort including Mesa and more. Init systems like systemd. Bootloaders like GRUB/systemd-boot. Networking, including systemd-networkd, iwd, and NetworkManager. Power and device management including PowerDevil, Bluez.
If all of this is considered 'user space' then my point is strongly vindicated. On most competitor OSs these comprise the 'platform'. You won't ever encounter a Windows or OS X developer arguing that USER32.DLL or Foundation is 'userspace' even if it is technically true and code from those libraries may only ever run in ring 3. From the average user's point of view, the only 'userspace' are end-user applications that they have installed, like Chrome, video games, etc. From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'. This includes such trivia like calculator and camera applications.
We need to talk about the practical realities of a platform, not a theoretical, unattainable, and as I mentioned, useless ideal. These practical realities mean shipping a unified set of useful 'batteries included' user-mode libraries and applications. If one wants to develop the platform itself, then sure, the kernel has a nice stable interface, but on most Linux distributions the kernel is built from source anyway, so even that nicety is moot.
> By 'platform', I mean
You mean Linux distributions.
> From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'.
Every Linux distribution is its own platform.
You decried Stallman's meme rant yet you just rewrote the entire rant in your own terms. Linux is the kernel, GNU's just the stuff on top. You can get rid of it if you want and make your own. Android did exactly that.
It's more than a little funny that you disparage syscalls being Linux's stable interface, and then immediately praise Android, which only works because glibc isn't uniquely privileged the way you seem to want.
Also, I daily drive Alpine Linux (using musl) and I'm quite happy to assure you that glibc really is only one option, not at all mandatory for desktop Linux.
> It's more than a little funny that you disparage syscalls being Linux's stable interface, and then immediately praise Android, which only works because glibc isn't uniquely privileged the way you seem to want.
I never disparaged it; I said having it doesn't make it any more useful that OSs that don't, because the practical reality is that the kernel interface is narrow and highly technical, and a stable kernel binary interface has little to do with a user's day-to-day experience with an OS.
I daresay Android would still work pretty well even sans the kernel ABI stability, because the kernel is usually built from source, and if it isn't, then every Android source tree has kernel prebuilts for the kernel version shipping. I also never said nor implied glibc should be uniquely privileged; I think you have misrepresented my comment.
> There's the old and new C runtimes in MSVCRT.DLL
Some advertisement for VC-LTL: https://github.com/Chuyu-Team/VC-LTL5 and https://github.com/Chuyu-Team/YY-Thunks
VC-LTL Implements a good subset of modern CRT functions using the msvcrt.dll, which makes modern MSVC compiler to emit binaries that theoretically can run in Windows NT 4.1 or Windows 98 (until CFG broke them, but you can turn it off). Still can teach an old dog new tricks.
YY-Thunk is a little more funky, it tries to implement new Win32 userspace API for old system by emulating the behavior and expected values using userspace API in the old apartment, so you can try to implement a function in Windows 10 with functions in Windows Vista, and hopefully it will work out.
>A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
I'm sure those OSes make efforts to make said runtime binary compatible between executables.
Yes, but the whole idea is to delete the runtime binary and talk to the kernel directly.
That's when you run into the Darth Vader of binary interfaces.
> I have altered the ABI. Pray I do not alter it further. -- De Raadt
I am not sure having a stable kernel ABI/API is really a goal in practice or intent in all cases.
GPU drivers are a perfect example - GPUs are hellishly complex, and their interfaces are super high-level - OpenGL/Vulkan is not something that should be implemented by the driver.
I think a general practice that is followed with GPU drivers is to move as much of the complex and non-privileged stuff out of the kernel, and only keep modesetting/power management/memory and control stuff in.
That reduces massively the amount of code that needs to run in Ring 0, but in exchange, the interface between the user and kernel blobs is non-stable. Programs talk to the userspace blob and there's no guarantee of compatibility between different kernel/userspace driver versions.
This is something even Linus accepts, with the only contention that the userspace blob needs to be open source as well. They're not maintaining something in tree, that they have no way of testing without proprietary external deps.
> A change in paradigm is necessary. Freestanding C, not hosted C.
Or.. any other programming language. A lot of systems programming is done in Rust nowadays.
Yeah. I made an entire lisp for this, and I've recently revived my Rust liblinux crate too. I've been documenting the system calls as I add them. Hopefully one day it'll rival man7 as the Linux system call reference.
> Can't be done. The libc is legacy, it can't be changed without breaking everything... Every other OS forces a C runtime.
Windows never had this problem. If you really want it, you can actually skip CRT completely on Windows. Only thing you need is the initialization code that's statically linked anyway: https://nullprogram.com/blog/2023/02/15/ . You can still make Win32 calls with it. CRT is there as a convenience.
And since Linux distros are compiling everything from source anyway, this makes Linux a prime candidate for complete libc and complete ABI replacement. Thanks to stable kernel ABI, we can also use the same kernel even with Docker etc! Musl did it. Why can a completely modularized libc not?
Will it break a lot of programs in the userspace? YES! Hyrum's Law exists.
Will it be rejected by a significant part of user base? YES! They rejected systemd as well.
Will it take one or more decades to have something working? HELL YES! This is Linux we're talking about. Everybody has seen how Wayland played out. However, maybe if Valve has enough motivation to fix Linux binary problem, we maybe can get finally a binary-compatible distro in a decade or half a decade.
We have a similar no dependency philosophy
SQLite already does this with its VFS layer. You hand it the OS functions it needs instead of it grabbing them, etc.
You could look at going straight to syscalls, mesa would be a rewrite nightmare
I went zero-dependency on a production CMS. Rust, no framework, no external crates beyond argon2 for password hashing. Running live on client sites. The hard part wasn't building it, it was accepting that everything you reach for is pulling glibc or similar assumptions back in through the side door. SQLite's VFS model is the right pattern. More things should work that way.
The concept you mentioned is just dependency injection in software engineering design, but instead of giving a class and protocol/interface you give a bunch of function pointers, which is, well, some kind of interface anyway
Most everyone else is talking about the problem while you mapped the room.
Solo is basically pg83's answer to the architecture you just described. If there is no libc independent ABI, build your own loader and shim the boundary.
I can understand Linus's obsession with taste and the areas it was overlooked or traded.
Wasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years.
Or do the MS thing, and ship multiple versions like msvcrt
> Wasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years.
Guess what happened with LSB? Some distros like Debian never adopted it. It also broke backwards compatibility at random times. It is now abandoned.
> Or do the MS thing, and ship multiple versions like msvcrt
MS can do that since the CRT is decoupled from the system DLL loader and Windows ABI is designed to isolated loaded DLLs into their own symbol namespaces. So your program can load DLLs that depend on a different CRT version.
Glibc's design tightly couples the system loader with the specific libc. Moreover symbols in ELF files are loaded into a single global namespace. You cannot ship shared libraries that are compiled with a different version of Glibc or Musl libc.
Btw, absolutely insane that it's 2026, and Linux cannot do the most defining OS thing - that is, provide a standardized environment to run binaries against.
All solutions to this problem are hacky, complex and controversal and highly fragmented, where this should be BASIC functionality
It's not insane. Many people who were in the GNU organisation wanted the OS to be as hostile against binaries as possible to prevent people distributing proprietary software with GNU. That GNU became the userspace layer.
It isn't that this is 2026 and it was a bug that hasn't been fixed since. The userspace of GNU is designed against what you consider as basic functionality.
musl has no problem building and using shared libraries.
What you can't do is build something statically with musl and then reliably dlopen shared libraries built with glibc.
Well, now it's possible! Furthermore, SoLo binaries can run, without modification, on glibc-based distros, alpine, and soon on android/bionic (not committed yet).
Only on current glibc distributions. Your SoLo-program WILL break on future distributions. This is not a sensible tradeoff.
Who says it is not a sensible tradeoff? It may be a much better tradeoff than not being able to run it at all.
Any software WILL break on future distributions, sooner or later.
> Why? Have people managed to break the ancient concept of shared libraries
If you break or remove a shared lib here, you may no longer be able to compile something from source. I had that happen in the past before I started to use more statically compiled programs (and busybox too).
Assuming everything works as-is via shared libraries at all times, makes little sense for ALL linux systems. For instance, some people upgrade glibc manually. Then you need a working base system to resume compilation. I do that for my customized gobolinux system, so I can use any program version as well as any glibc version (assuming I can still compile the program; many older programs no longer compile).
Because many folks don't understand UNIX systems introduced dynamic linking for several reasons, and they actually only had static linking for almost 20 years, since UNIX was known outside Bell Labs.
Additionally many other OSes have had both approaches since their early days, Xerox PARC ones.
For some strange reason they assume to know better than all those researchers.
These decisions, and these studies, were made a VERY long time ago. It's completely unclear why the decisions made then are relevant now, and why they can't be challenged.
It is like advocating that we should drop cars and go back to chariots, because wooden wheels don't get flat, while forgetting why they are mostly used for tourists nowadays.
If you can figure out your own ELF loader, you can figure out how to build a partially static executable that doesn't need this. You can mix static and dynamic linking. Build tooling around that is just shit.
In this scenario, I'll have to choose which libc I want to run. These won't be portable Linux binaries in the true sense of the word; I'll have to leave Alpine out, and possibly Android, which I don't want.
You don't strictly need to; you can write freestanding C, and use the Linux kernel directly.
Fair point - while it's certainly possible to make a decision per individual library elsewhere, libc is the one thing that really will need to be linked dynamically for glibc, and it'll freeze a minimum version into the binary (i.e. it should be built against an old version of glibc).
I don't think Android is a target in most cases of this, so this would boil down to shipping 2* binaries, one glibc and one musl.
</i> I'd need to check how musl behaves for compatibility, I'm assuming it'll be a minimum like glibc.
All of this said - you'll need to ship multiple binaries anyway, these days: x86_64 + aarch64. Possibly more…
Can you really mix it? When I break or remove a shared lib, some binaries no longer work. With static libs or even better, e. g. statically compiled busybox, I don't have that issue, so I disagree on the claim that mixing solves everything as such. I keep the basic toolchain I use as statically compiled variant. The whole system works better if I can break it less easily.
I'm not sure that's particularly relevant; if I null out bits of the statically linked binary to remove some code paths, they break too.
How this differs (is better!) from prior art - https://github.com/pg83/solo#how-this-differs-from-prior-wor...
Note: That md file is LLM spew (like most of rest of codebase).
https://github.com/pg83/solo/commits/main/README.md
https://github.com/pg83/solo/commits/main/
> spew
Rude.
It only gets better from there - https://github.com/pg83/solo/blob/main/CONTRIBUTING.md!
Do you belive the machine is taking offence? It can not.
Telling me, a coder you never met or interacted with before, that I would introduce more bugs than The Product(tm) is pretty rude and prejudiced.
> The project author believes that, with capable human direction, modern LLMs write code faster than people and introduce fewer bugs.
> Do you belive the machine is taking offence? It can not.
Obviously, this is an offense to me.
> Telling me, a coder you never met or interacted with before, that I would introduce more bugs than The Product(tm) is pretty rude and prejudiced.
Don't exaggerate. I didn't say that you personally introduce more bugs (as you rightly pointed out, I don't know you and don't know how often you introduce bugs), I said that the average developer introduces more bugs than an SOTA LLM.
No sense in arguing with LLM haters (or zealots). It's approaching religious levels of discourse. Let them have their opinion and move on.
> Obviously, this is an offense to me.
But why? You didn't write it. You said so yourself, you wrote the initial text but it was all reworded by claude.
It's still his project you're insulting.
Is it? In what sense?
In the sense that he had the idea and iterated on it until it became something real that actually works.
Agree. The work itself looks solid. Good job to the author.
Thanks a lot, guys!
I agree. Even if the README was polished with AI, the technical work behind it is impressive. Let's focus on the code.
Of course it does. I once called the thing a /communist fuckface/ for speaking the wrong language and it responded in a way an offended person would.
I could choose to believe in a social fiction and you can't stop me.
What are your plans around the maintenance of this project, how would you feel about solo being incorporated into the musl build for graphics.gd ?
Every couple of years, I revisit my PL dev hobby and this time I decided to create a language/runtime with pre-emptive scheduling using instruction fuel. While I always do freestanding builds, this time I decided that I also wanted to support native FFI.
That is when I realized the true horror of (g)libc. It wants to inject itself at the root of the library/program and everything from threading to dlopen/dlsym is impacted. I tried a lot of workarounds including trying to implement a loader myself, but the complexity (and fragility) grew so much that I felt it was not worth it.
Finally, I retreated into the safe world of a freestanding runtime + syscalls. FFI, if it has to happen, will occur via IPC of some kind. A second process linked against glibc that will manage calls on behalf of the clean first one.
I've implemented the same thing for micron (more or less). One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE). That being said the way you're doing is also tricky(ish) because if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you. Doing this is safer if you control the entire runtime.
> One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE)
Yes, it's not trivial, but I hope that over time everything will settle down.
> if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you
No, I hook this to musl, which is statically linked into my binary, and I have complete control over it.
I have faced a similar issue in the past, and I don't understand how static binaries from the host are supposed to solve this.
From what I remember, GPU access on Linux 'works' by accessing specific FDs under /dev, which are vendor specific - this is what these libs do under the hood.
The libraries don't have any magic powers - if the FD is inaccessible, you won't be able to do anything.
So there's some vendor specific access needed in containers anyway (or a blanket allow, which is a BAD idea).
Also not sure why dynamic linking isn't good enough for this - the issue lies with the permissions, not how you load/link libraries.
i'm not sure how much people realize that the modern "graphics driver" is actually just "the kernel multiplexes userspace messages to/from the GPU and we've taught mesa to understand each family of GPU you'd ever care to support."
i helped somebody get Doom running on an old embedded system running some 4.x kernel. we just built everything, including mesa, statically and deployed that. imagine building everything static but loading the system's libgl.so instead (which was probably just a symlink or abstraction over mesa's own implementation). what's the benefit: we'd get older, less optimized graphics routines from 5 years ago?
if you're statically linking, then just bring your own graphics "driver". the kernel interfaces are stable enough. it's not conceptually different than embedding `syscall`s directly into your application the same way you do when statically linking libc.
> i'm not sure how much people realize that the modern "graphics driver" is actually just "the kernel multiplexes userspace messages to/from the GPU and we've taught mesa to understand each family of GPU you'd ever care to support."
In the case of OpenGL, it has a fairly sophisticated state tracker, which you can carry around with you. For example, I can statically compile ANGLE this way.
But even OpenGL/Vulkan have a shader compiler that's unique to the hardware (it's expensive to carry it around with every piece of hardware in the universe).
And I'm not even talking about libdrm, which is quite tightly tied to the kernel version.
I know exactly what I'm talking about, because I can also statically compile Mesa into my application.
Do people say "so", "ess-oh" or "dot-ess-oh"? The title "a .so" is clunky to the "ess-oh" gang.
I don’t think I’ve said it out loud more than a couple times in my life. But in general I think I spell out / pronounce the “dot” in file extensions unless it’s completely obvious from context.
Well .so is short for shared object so I think "a .so" is correct.
That's not generally how the English language works. We say "an SoC" but "a system-on-chip".
If you dynamically sold an SO are you still static even if you did it "custom" ? At that point it's a dynamic loader in another name?
Technically, you're right, it's a dynamic loader. Technically, it's pure dynamic loading.
If we look at the issue at its core, we're still a statically linked program in a hostile environment, forced to dynamically load device drivers from the system.
It's similar to Golang; on MacOS, it has to use libSystem, even though otherwise, these are the statically linked Go binaries we're used to and love.
Let me add a little more detail: if I use vdso with gettimeofday in a statically linked program on Linux, am I still a statically linked program, or not? :)
if you really want a single elf that could link to system libraries with a foreign libc (you shouldn't), the somehow correct-ish way to do it is map your preferred ldso manually , setup stack, call DT_ENTRY etc etc like how kernel does it, and just yank the libdl symbols.
see cosmo_dlopen for an example implementation.
The README.md link describes cosmopolitan's approach, among other things, and it's worse than mine.
You normally do this sort of thing with plan old `dlopen` and `dlsym` and make your own "plugin" system.
Is the 'hello.jpg' reference intentional?
It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens).
Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).
How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.
I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
https://bugzilla.redhat.com/show_bug.cgi?id=638477 is the most "famous" example.
There are also much less well-known "little things" that regularly pop up here and there.
> If your CI uses an older glibc you should be fine AFAIK.
In any case, my binaries work not only under glibc, but also under Alpine, and (work in progress) under android/bionic.
Not sure what you’re trying to show with that bug report but it’s not a case of cross distro glibc issues. If I read correctly it’s a vanilla behavioral change that exposed preexisting UB in flash.
Not sure how the comments about alpine or bionic relate either to my claim that cross distro glibc is fine.
It depends on how we define the ABI. I see it as a set of client-visible invariants that they rely on. In my world, glibc changed the client's visible invariants, breaking the client. The client works on one glibc-based host, but not on another. What is this if not "a case of cross-distro glibc issues?"
Overall, both of our points of view on compatibility were discussed well in that thread; we probably shouldn't repeat ourselves. :)
ABI is not "whatever happens to work with this distro" but "what programs that comply with the API contract compile to". Overlapping memcpy arguments is an API contract violation and thus not something covered by the ABI either. This distinction is the entire reason why C has a separate memmove function. You can't just make up your own imaginary ABI contract and then blame the system when it doesn't fulfill it. That's going to result in self-inflicted plain on any OS.
It's so good that Linus thinks differently!
Even Linus’s comments in that bug hedge things a bit:
> and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to.
(Emphasis mine.)
In the Flash case, they changed the direction memcpy works in, in a way that didn’t actually improve any performance (Linus did benchmarks to show it) and Linus’s point was that there’s no upside to the change, and only downsides, so on the whole it makes no sense to keep the changed memcpy.
You could imagine plenty of scenarios where existing behavior truly is broken, and show that no popular software depends on the brokenness (which is why distros are in a great place to do these kinds of smoke tests, they have thousands of packages they can test), and make a judgement call.
A “zero regressions no matter what” policy is impossible in practice due to Hyrum’s law, at some point you have to use common sense and draw the line somewhere sensible.
> It depends on how we define the ABI. I see it as a set of client-visible invariants that they rely on
By that definition, any change of any kind will break ABI, Hyrum’s law and spacebar heating and all that.
Did the distro apply a patch to glibc that broke things that others did not? Or did they pick up a newer version of glibc that other distros hadn’t?
Yes, different distributions run on different versions of glibc. And different versions of glibc may have subtle differences in runtime behavior. That’s still ultimately a glibc back compost issue, not some kind of bug specific to glibc.
> Quite frankly, I find your attitude to be annoying and downright stupid.
- Linus Torvalds
That bug report was a good read.
according to appimage recommendations as long as you build against glibc with an earlier version than the system it's run on it should be fine.
https://docs.appimage.org/reference/best-practices.html
I hear you about WINE though.
This is not true. Glibc supports symbol versioning. You can use it to select old versions of used symbols. The result is a binary that can work on 20 year old distros the same as on the latest, compiled with latest compiler and Glibc.
You can also compile using old distro and old Glibc to get similar effect. Though you would miss the advances of the newer compilers.
https://news.ycombinator.com/item?id=49358538
That irony is delicious. Build to WIN32 APIs to ship on Linux.
I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
It's a pretty poor solution, to be honest.
1) Why should I limit myself to the available APIs?
2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
Luckily Zig makes this quite easy to do. In Mach[0] we are able to just `zig build -Dtarget=x86_64-linux-gnu.2.28` to build GUI apps against an ~8 year old glibc version for maximum compatibility.
This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support.
[0] https://machengine.org
Complete failure is strong words when lots and lots of Linux boxes are running just fine.
I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date.
Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue.
The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros.
When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works.
You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work.
It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want.
I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.
I disagree that this is a niche problem.
Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs.
But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
And we can't count on someone else packaging our work until we become incredibly popular.
> But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution.
Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those.
> And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Due to FUD, mostly.
Sane people do just link with an old enough version of glibc.
So not completely static, since it must link against a libc :P
The binary itself is completely static; the link even provides commands on how to check this!
Ah yeah my mistake static != no external dependencies. I was thinking of golang static binaries (built without CGO) which do not have a dependency on glibc at all.
Why not just pass the GPU to a docker container?
Available in README - https://github.com/pg83/solo#how-this-differents-from-prior-...
Claude-ware
How are we supposed to take this stuff seriously if the author (sic) isn't even willing to write the readme? Claude exists! If I want some slop I can push the button myself.
Isn't this a README? https://github.com/pg83/solo/blob/main/README.md
Parents point is the readme was written by Claude. Signals low effort.
README.md was written by me, and I, of course, used claude/codex for it. In general, I do everything through claude/codex, the reasons are described in https://github.com/pg83/solo/blob/main/CONTRIBUTING.md . And no, it's not low effort, and no, I don't see the point in wasting time de-claude-ifying the text just to avoid it looking like I didn't spend enough time on it.
I believe usually when someone complains about text written by a language model they are hoping to read human-written text instead of human-laundered LLM output.
You make it clear why you write all your code through a llm. But a README is not code. Presumably you would like people to read it. A machine authored readme reflects poorly on a project.
The author of the README is me, the machine just wrote it. I am not a native speaker of English, my written English is simply terrible, no one wants to read the text that I wrote exactly :))
I think we're trying to tell you that we actually would prefer it.
For context, I read Claude output every single day, I know what it is reliable with and what it is not. Or at least I have a feel for how much I can trust it.
It may not be clear to a non-english first language person, but when I read Claudes documentation, my brain immediately picks up claude-speak. Therefore, I expect the code to be generally correct, maybe, depending on how specific I was during my prompting. In no way shape or form do I really trust it, at least until i dig into the code and validate my mental model. And query the review for edge cases etc...
By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.
To a degree this is unfair, as it is like judging the quality of someone's work based on their accent.
But Claudes accent, has a high correlation with Claude, so unlike with people, where an accent has no bearing on technical ability, Claude being Claude does.
I would much rather read a typo ridden sentence than claudism, even just as a forward, explaining what you did vs the ai, and telling the users how much we should trust it.
Also, If you really insist on using AI to write, have another model rewrite docs/comments into regular English (opus 4.6 for example is much better than 4.7, 4.8, or 5.
It is open source so you can do whatever you like, but people (especially native English speakers) will discount your work, because Claude, especially opus 5 writes very very badly.
The people who say they would rather see a typo infested mess of a readme are serious. Or even just write in your native language and then have Claude translate.
Both of those are better indications of proof of effort than a Claude readme.
> Or even just write in your native language and then have Claude translate.
That's exactly what I did.
> By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.
I'm sorry, but I agree with the author: if a certain writing style makes you associate the work with low-quality, then that's your problem. The author shouldn't have to rewrite the readme just to avoid triggering your automatic unfounded associations. If you look at the substance of the work, including the test cases, then this is clearly not easy work that can be vibe coded in a single pass.
It's just like emdash. Everybody digs on how it's a signifier of LLM text, but I've used emdash for years because it's gramatically correct. I shouldn't have to stop using emdash just to avoid kneejerk reactions.
if 99 poor effort/quality projects have a readme that reads in a particular style, then you expect the 100th project with a readme in the same style to also be of poor effort/quality
statistically, it only makes sense for your expectations to immediately be low when you encounter this writing style because there are just so much slop out there
the author is free to keep that writing style but they should be aware that this will—at least on the surface level—make their project look exactly like the metric ton of slop we see posted everyday everywhere
That's like saying a large percentage of X-colored people are criminals, therefore when we see an X-colored person it's reasonable to expect him to be criminal. As a society we have decided that heuristics like this, and prejudicing individuals based on statistics of the group, are not acceptable.
Also, what do we call it when people don't look beyond the writing style, such as at the code, or at the rigorousness of the tests, or at other substances beyond writing style? Slop assessment?
On the other hand I know how to tell if the underlying idea is good, and if it is I can decide to put up with reading through LLM outputs.
LLMs often use emdashes in a distinct incorrect way. It's not just the existence of any emdash, although considering you were in a very small minority of older users, it now warrants increased scrutiny, unfortunately for you.
I would rather read "terrible" written english from a human than I would read "perfect" written english from Claude.
In any case, it's open source. If you don't like something, even if the project seems generally useful, go ahead and fix it. The PR came in. I'm an engineer and I can write good code, but that doesn't mean I can write good README.mds!
I almost always write my own prose, and when I have an LLM write me prose I edit it to make it my own, in my own style, and pithier than the LLM would make it.
I'm not saying everyone must do that. For me it's partly a point of pride, and partly empathy (editing for conciseness is showing empathy for others' time). But I am saying that people should consider doing it.
Implicit assumption of gp is that the README is llm authored. (Which, I agree is how it reads to me.)
Tell me, are there any substantive comments on the text, on what has been done, and on the technical implementation, and not on the form?
Why would I ask you about this work? The root of my question is why is this type of output shared on github, rather than the inputs?
Rough
Disclaimer: This is a user’s perspective rather than a programmer’s perspective.
valid point. I am usually okay with LLM generated code since even if it might not be architecturally sound It is usually well commented and has tests and documentation for helping another agent/human debug any issues.
But, just the painful experience of debugging any dlopen related crashes and/or intermittent bugs; and the sheer amount of tokens burnt by an LLM chasing tangents when shown a stack trace; I wouldn’t touch this at least as a packager/consumer of certain apps for personal usage on older distros. So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.
Modern models, when properly managed with a human in the loop, write higher-quality code than humans and introduce significantly fewer bugs. Therefore, it's quite the opposite - you should expect fewer "dlopen-related crashes and/or intermittent bugs."
> Modern models, when properly managed with a human in the loop, write higher-quality code than humans and introduce significantly fewer bugs.
I don't think anybody believes this, and interjecting it into every thread is not really convincing anyone.
where is the study confirming this?
> So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.
The Linux ecosystem already settled on containers to solve the problem. Namely Docker, Flatpak, Podman ec.
https://github.com/pkgforge-dev/Anylinux-AppImages/blob/main...
Wanna see those containers running in those old systems. Good luck getting flatpak to run on a kernel without PR_SET_NO_NEW_PRIVS.
They require admin privileges. A consumer on an enterprise servers might not have them, which is where I would guess glibc might not be latest one. If you are an admin then sure those solutions can be feasible.
> backed by its own ELF loader (x86-64 and aarch64) and a glibc ABI bridge
Yacks
Please elaborate and explain to people with less knowledge why this is bad.
Mapping parts of files into executable memory, and then executing them, had better be bulletproof! Exploiting this seems like a direct path to RCE, and it's likely that this sort of library is used by privileged code.
Purely academically, this is a very cool piece of code! Just hoping that it gets a thorough vetting before used by privileged/security-critical software :)
Well, ld.so already does this, and it's no big deal. The Python interpreter also does this when executing a .py script (code is code, whether it's machine-readable or human-readable).
In any case, we take testing very seriously—every glibc shim we've written is covered with tests, and we run our loader against 1000 of the most popular Debian packages. The project has 100% code coverage. Perhaps, if I have the time, I'll also do some fuzzing on this thing.
similar energy to my https://github.com/lunixbochs/crossldso
Oh, cool, another prior art I didn't know :)
If you want to load the OpenGL/Vulkan vendor driver then unfortunately you don’t have much of a choice: those are linked against glibc, and I believe generally also against libwayland so screw off if you want a different protocol library (I might be wrong about the latter part). If you instead want to load plugins or whatnot into your statically linked executable, then personally I’d argue that you shouldn’t be emulating Linux dynamic linking semantics at all, because the whole late-bound global namespace thing is silly and wrong. (Solaris, which is where Glibc took this model from, moved away from it[1] as much as compatibility allowed, and so did Darwin[2], whereas Windows never made the mistake to begin with, but Glibc persisted and Musl copied it.)
[1] https://www.linker-aliens.org/blogs/rie/entry/direct_binding...
[2] https://web.archive.org/web/20011004090044/http://developer....