A few folks have asked me the generic ABI status (unmaintained?) and the availability of an up-to-date specification (no). I compiled “History” and “Evolution of the generic ABI” in the blog post.
I have two specific questions:
- Key features (symbol visibility, section groups, SHF_MERGE, etc) were all available as of April 2001. Where can we find the discussion mailing lists? Are they still available?
- How does the ABI end up being “All rights reserved” by SCO? Tool Interface Standard (TIS) Portable Formats Specification, version 1.2 effectively put the specification in the public domain.
One of the things I've been (very slowly) putting together is a master list of all the references one needs to build a compiler toolchain--such as processor ISA manuals, ABI specifications, language standards, even things like IEEE 754 or DWARF.
While IEEE 754 is an issue because it's not freely available for most people, the only document that has truly stumped me is the ELF specification. From your blog post, it seems like my failure to find the most up-to-date specification is simply because one doesn't exist.
The ELF format isn't the one that I'm the most concerned about, but for example compressed sections were updated in the past decade. The ABI for x86-64 is out-of-date (actually, the site it links to is now a 404, but my recollection was that was a version around 2013)--missing things like _Float16 rules, DWARF register naming for AMX and APX registers, microarchitecture levels, new relocation types,
I'm aware of that (and peeved that they've got runners set up for the x86-64 ABI but not the i386 ABI, so there's no nice PDF version of the i386 version I can use).
I file things like linkers, loaders, and debuggers into the same bucket as compilers, so all of the ELF extensions are interesting to me, even if compilers directly don't interact with them. I'm also interested in things like the loader-debugger rendezvous interface. Basically any document anyone who touches any project in llvm-project might need to read.
And then there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts.
In case this is of interest, here's an attempt at gathering the references, including random blog posts (as well as articles, documentation, papers, software, and talks), on these topics:
> even if compilers directly don't interact with them
Essentially they do as they put directives into the asm files to put things in specific sections, to define certain symbol visibility etc all of which depend on the object file format (e.g. a classic BSD a.out file, using a format I called “a.out” in BFD, can’t represent most of that). So the compiler does need to know. And the assembler absolutely needs to know since it generates object files!
So yes an a.out file can be an ELF file (always is on Linux) or could be an “a.out” file. Sorry! In retrospect I should have called that format “bsd”.
> there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts.
The way I dealt with it was write an ELF dumper for the tables, and then run various C++ code snippets through g++ and look at how the tables were constructed.
So far, the list of resources is a collection of PDFs I've downloaded plus a script to download the latest version of every relevant specification. (Although it does seem that Intel likes breaking its links to the latest ISA PDF every now and then). I don't have a great solution for things that are essentially web-only pages, which is how the Itanium ABI is handled (and unlike C, C++ makes it hard to find the relevant working draft copies, instead preferring to use the HTML-based https://eel.is/c++draft for that).
I probably will make it publicly available eventually, it just hasn't been a priority for me.
The most widespread standard for archiving collections of web pages is WARC - it is a publicly available ISO standard so it's very future-proof. WACZ (https://specs.webrecorder.net/wacz/1.1.1/) is a more recent take that is basically zipped WARC + index + metadata, and Internet Archive can generate them when archiving is requested (https://web.archive.org/save/).
In addition to that, I would also suggest using https://zimit.kiwix.org. It's a very compact representation that has full text index support for convenient searching, the format is open and documented, and first-party readers are available on all major platforms. And for stuff like this, Kiwix might actually be interested in adding them to their official library at https://library.kiwix.org/ if you contact them.
The first thing I do when porting to a new platform is build an object file format pretty-printer. It's the only way to figure out how it really works.
The written specifications for them are often a bit of a fairy tale. The real specification is whatever the local C compiler generates.
The other gatekeeper is what the linker will accept. That can sometimes be a nightmare.
I fell very far down this rabbit hole trying to figure out the history of library link orders and symbol resolution. One fun outcome of which was finding that the first ever public release of a manpage for ld.so on Linux listed RPATH as deprecated. Finding mailing lists from before the source ware transition is hard, some of it exists if you follow the path back to the Solaris forums, but only some. I’ll look through my archives for the features you listed, but I don’t think I uncovered anything about them specifically. Good luck!
> Second draft published May 3, 1999.
> ...
> New dynamic section tags DT_RUNPATH and DT_FLAGS added. Dynamic section tag DT_RPATH moved to level 2.
In glibc, DT_RPATH and DT_RUNPATH have different semantics regarding precedence related to LD_LIBRARY_PATH. In FreeBSD rtld and musl, DT_RPATH and DT_RUNPATH are equivallent.
> How does the ABI end up being “All rights reserved” by SCO?
Text of the spec is copyrighted by the SCO, they did not put it in public domain. That's what "All rights reserved" was intended to mean even though this phrase has not been meaning anything since 2000.
It's concerning that a huge open source ecosystem like Linux depends on a closed specification. That is, I read it as that the spec cannot be continued, developed, and evolved or republished without full redefinition and respecification.
Has anyone looked at creating a new object format for Linux? A non-open spec seems a minor issue, really, in an era when we put binary blobs in the kernel (hi Nvidia.) But the more decades I work in closed source, the more I value open source, and believe keeping _everything_ open.
I wouldn't think you'd need to create a new format, just write up a spec of what all available FOSS implementations are actually doing and agree that that's the standard going forward. Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF.
(Of course this assumes that doing this is sufficient to make it legally not derived from the original spec as far as copyright law cares about, which is beyond my non-lawyer ability to be that confident in.)
> Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF.
OpenVMS uses ELF too. Rather niche proprietary OS but still maintained and in production use. (Not a new thing with x86-64 port, ELF was adopted during the Alpha to Iranian transition.)
As do many RTOS (both open source and proprietary)
Lol. Although, Itanium does have an entertaining history of having caused a lot of positive changes even though it crashed and burned itself - it's where we got UEFI, and apparently a good time for some OSs to pick up ELF.
> Has anyone looked at creating a new object format for Linux?
No, because that's overkill for this problem, because the de facto standard is "What do modern free Unicies do", where those are basically Linux/BSD/Illumos. Besides, it's not unheard of in Unix history to have free implementations of proprietary standards, where those implementations became de facto standards anyway.
Even if it was a seriously pressing issue, writing a new specification that covers ELF as it exists today is going to be many, many, many times easier than introducing a new executable format across the ecosystem, they're not even in the same league or sport in terms of effort. If you were going to introduce a whole new object file format it would be much better to do it and justify it on the basis of fundamental technical/architectural changes.
> It's concerning that a huge open source ecosystem like Linux depends on a closed specification.
Linux is written in C, whose spec is proprietary. The larger stack has countless ISO standard references etc., all of which refer to proprietary standards you need to buy.
Yeah, it's not like they could copyright the ELF format anyway, only that particular description, or any particular implementation they had. Pretty much everyone seems to agree that file formats themselves aren't copyrightable expression, though I have trouble finding any U.S. case law to that effect.
(Personally, that makes me a bit uneasy: for instance, if the copyrighted spec lists the file sections in a certain order, and your implementation happens to output them in the same order even if it doesn't have to, then have you infringed on the owner's copyright of that particular arrangement?)
Meanwhile, they could have gotten a patent on (some parts of) the format, but that doesn't seem to be the case here.
The rule in general (not just for software) is that patents are for functional elements where as copyrights are for non-functional elements; they are exclusive, a element can only be patentable or copyrightable, not both.
If it is necessary for operation or inter-operation [1] then they can probably not enforce copyright or trademark.
Sure, but usually in any file format, an encoder has some choice in how to produce its output, with no functional difference either way. For instance, different PNG encoders might include the necessary chunks in a different order in the file. And my concern is that some such choices, if they align with choices described in the copyrighted spec, might be protectable as a creative arrangement or similar. (E.g., a spec might list the chunks in such an order that their names form a creative acronym.)
Of course, the defense against this would be to scramble (or normalize) all non-functional choices compared to anything in the spec. But you have to be careful to make sure there's nothing left of the spec's non-functional influence.
At least Oracle v. Google appears to provide some ammunition here in favor of implementers: the court found the transformative use in that case enough to trump even the byte-for-byte copying of the API signatures. So perhaps interoperability could similarly trump such non-functional copying from the spec. But overall, it's still on shakier ground than I'd like.
I feel that The SCO Group's role in the evolution of the System V ABI seems to have been more of a curator/editor than an innovator, inheriting the System V ABI from previous entities.
Given that the Tool Interface Standard (TIS) Committee has essentially released the ELF-related chapters into the public domain, and others have made changes,
it's unclear what specific rights The SCO Group (and now Xinuos) could claim to reserve.
(That said, their maintenance work needs to be remembered.)
> Q18: How can you get a single binary to work identically across all these diverse systems?
> Most Unix-on-Intel binary packages are already largely similar. Almost all such operating systems use the "ELF" binary 'packaging'; the various operating systems have small but significant differences, though, that make each system's ELF binary unusable on others'.
Though the scope has diminished with the decline of proprietary unixen, there is a nice bright spot with APE binaries ( https://justine.lol/ape.html ) which are in some regards even more portable since they work on Darwin (which natively uses Mach-O) and NT (which natively uses PE).
Yes! I immediately thought about APE when I read The 86open Project's FAQ. However, I feel that APE is less related to the blog post so decide not to add the link to the post.
> Despite this uncertainty, innovation within the ELF ecosystem should continue.
> In practice, achieving consensus among major toolchain vendors (GNU and LLVM) may be sufficient, even without formal approval from the generic ABI.
That makes it sound easy. GNU projects seem to be very conservatively maintained. LLVM has around one trillion open pull requests.
I came up with a little mechanism to get the kernel to automatically load into memory data embedded into the program. I used it in my programming language to embed code into the interpreter so that they can be automatically loaded and executed.
Only the maintainer of the relatively new mold linker cared to implement a helpful feature to make this kind of thing possible and easy. I requested the same feature in GNU ld and the idea wasn't exactly received with enthusiasm. I'm not sure LLVM ever received it at all.
And that was just a linker option to add some extra PT_NULL segments for easy and efficient patching. Can't even imagine the effort it would take to actually change something about this ABI.
Isn't that's what NOTE segments are for? It can have arbitrary data embedded in it, that's how GCC implements most of its insane dynamic-linking-related extensions IIRC. Of course, it has to overlap with some LOAD segment to be useful.
I considered using them. Ended up not using them but they did inspire the actual solution.
At the time I wasn't able to find much information about those things and how they're implemented and used in practice. Couldn't be sure if they were a good fit for what I wanted to do.
The best functional description of these notes I've ever found actually came from some old Linux standard base document. Apparently people would use their company's name as the note's name and enumerate a bunch of metadata types that are specific to their company. To me that sounded a lot like the "email the guys in charge to ask them for their blessing" situation described in TFA. I'm just a random guy and I just wasn't gonna show up out of nowhere in somebody's mailbox asking for some official designation for my hobby language.
There was no practical reason to use these things either. Turns out the kernel doesn't care about PT_NOTE segments at all. They don't get loaded into memory automatically. I'd still need to add a PT_LOAD segment for the data even if I had used this stuff.
The PT_LOAD segments are the only ones that really matter. Those are the segments the kernel cares about. Ideally I would have been able to simply use a PT_LOAD segment for the data and nothing else. When I tried it though, I discovered they had two limitations: alignment requirements and lack of type information. They have to be aligned to some page boundaries which obfuscates the true size of the data they contain. I also can't easily tell them apart from each other, I'd need to put some header in the data itself in order to find the correct segment which risks false positives.
Those two problems are solved by the PT_LONE segment. It describes the true size of the data block inside the PT_LOAD. It provides a unique magic number for finding it while iterating over the program header table. No possibility for mistakes. When it's found the program gets a pointer and a size and the data is already in memory because it's contained inside a properly aligned PT_LOAD segment. This makes the kernel do a ton of heavy lifting before the program has even begun execution.
This is what the PT_NOTE segment boils down to anyway. It's just a memory area. The standard just happens to mandate the presence of a header structure on those segments. The PT_LONE segment doesn't have that requirement so I just replaced that opaque binary header with S-expressions that my interpreter simply parses instead. Much nicer in my opinion.
> you can always leave the section table in as well
That was the first thing I tried. The objcopy tool even has options that let you embed arbitrary files inside the binary by putting them into a section. Seemed incredibly easy at first.
Turns out that ELF sections are irrelevant to the kernel: it only maps in the PT_LOAD segments. Had to ask lots of questions on stackoverflow and read the Linux ELF loader source code to figure this stuff out. Then there's the fact that linkers don't generate PT_LOAD segments covering the program's sections. Apparently only the dynamic linker uses that stuff so there was never any reason to make the kernel load it. It's just how things are done and it doesn't look like things are about to change to accomodate other use cases.
Since this is not done, anyone who wants to embed data into the ELF has to somehow find and open the running program's own executable, read the thing into memory and parse its contents a second time. This is just bad on so many levels. It's fragile and duplicates work the kernel has already done.
The proper solution is to make the kernel load the data as part of the program's memory image. That requires editing the segments, not the sections. Tools for editing sections are quite featureful. Tools for editing the segments didn't actually exist.
So I made a little tool to patch this data into an ELF file. This turned out to be very hard and annoying. There were two possibilities: either move the table to the end of the file, or reinvent the linker. The first option leaves a huge hole in the file where the original used to be but it does allow adding entries without relinking everything. The second option doesn't waste space but is orders of magnitude more complexity than I had signed up for.
Took a third option by asking for linker support instead. The mold maintainer added a couple lines of code and suddenly my ELF files had PT_NULL entries tailor made for patching which is just awesome and the best possible solution.
I asked GNU ld maintainers to add this feature as well. They suggested I try using linker scripts instead. I had actually tried that before emailing them and it's actually what motivated me to just ask for this. I seriously hope I never have to write linker scripts ever again.
I opened an issue on the LLVM issue tracker but never heard back. I also tried to implement it in the patchelf utility but couldn't figure out how it works.
I'd be interested in reading about the history of object file formats in general. I'm wondering what motivates them being so relatively complicated for something that seems so conceptually simple; is it historical baggage or is it essential complexity?
- 2011- Xinous, but Xinous has stopped updating https://www.sco.com/developers/gabi/latest/contents.html . It seems that Xinous has moved on from System V based OpenServer/UnixWare. The newer OpenServer seems to be based on FreeBSD. They likely no longer care about the System V ABI.
Nowadays, people make proposals to the generic-abi Google Group.
Essentially, a proposal is considered "standardized" if it receives approval from GNU, LLVM, and Ali Bahrami (Solaris representative).
Many GNU extensions are implemented by LLVM and adopted by BSD and newer ELF-based OSes.
For extensions that are considered generic enough, it's recommended to propose them through the generic-abi Google Group.
psABI documents generally prefer generic extensions over GNU or LLVM-specific ones.
I have two specific questions:
- Key features (symbol visibility, section groups, SHF_MERGE, etc) were all available as of April 2001. Where can we find the discussion mailing lists? Are they still available?
- How does the ABI end up being “All rights reserved” by SCO? Tool Interface Standard (TIS) Portable Formats Specification, version 1.2 effectively put the specification in the public domain.