Last modified on 2025-03-21
Hubris
Hubris or exHubris is an embedded operating system without dynamic task loading. For this usecase, the combining of separately compiled components happens at (Hubris) build time rather than on the host machine. This gives an interesting set of potential design tradeoffs. The lack of dynamism means that a full compiler is available when combining the modules. The real-time constraints of the system however mean that additional runtime overhead must be carefully considered. Given that the project is currently functional, I can contact the team to see if there is anything that they desire from us.Rubicon
Rubicon has been explored in the extant work post. For Rubicon to work as it does now, it only needs to expose existing linker flags.Large utility libraries
Large libraries implementing commonly used functionality are often loaded as dynamic libraries. These libraries vary in what they provide, and examples would belibpng, jemalloc, sqlite, lzma, and many others.
The utility of dynamic linking lies in reducing the amount of copies of the library that need to exist, not just on disk, but also in ram and the cpu cache.
In many cases the data passed through the dynamic boundary is raw binary data (representable by a Vec<u8> or a &[u8]).
The binary interface to these libraries is often stable, with updates adding functions or variants.
Multiple versions of these libraries can peacefully coexist, making ABI breakage manageable.
Breaking changes cannot be coordinated between libraries and users as they are organizationally separated.
Library multiversioning
Many utility libraries use optimized CPU instructions that may not be available on all systems, e.g. AVX-512 or T4 crypto instructions. Correctness is vital here, as executing an unsupported instruction is Undefined Behavior, unless the instruction is defined undefined. A common solution for open source projects is to recompile the library with the desired-march and -mtune options set.
This solution does require the system administrator to have access to source code, as well as needing to run package distribution.
Another option is to ship a “fat binary” that contains multiple versions of the code.
Such a binary contains multiple instantiations of functions using different instructions.
A subroutine is then used to select the correct version at run-time.
The Solaris/OpenSolaris/illumos systems provide a synthesis of both of the solutions above, using dynamic libraries.
Symbols can be annotated (by the linker) with the required hardware capability, and the link-editor will resolve the reference to the “most capable” version.
This even works for private symbols, whose privacy will be respected.
A major advantage of this solution comparsed to the above is that the search cost only has to be paid once, rather than every invocation.
Another benefit is that only the functions that benefit from hardware extensions have to be multiversioned.
Glibc also supports hardware capabilities specific library search paths automatically — rather than relying on LD_PRELOAD.
An object located in /usr/lib/sse2/ will only be resolved if the current system supports SSE2 instructions.
This has the disadvantages of increasing the search path drastically, as well as duplicating the whole object file for a particular capability set.
Even more powerfully, glibc+gcc offer the indirect function mechanism.
An indirect function is given a user defined (or compiler generated) function resolver, which returns the function pointer to the desired function.
Once resolved, this choice is final and cannot be changed for the lifetime of the process.
It is evenb more flexible than the illumos version, as arbitrary code can be used to decide which pointer to return.
Dynamically selected parts of a program
A program may want to delay the choice of a specific implementation of an interface. In essence this is the same as loading a utility library, but it adds more dimensions of complexity. Having an implementation neutral “header file” that can be implemented against ensures that independent implementations are interchangeable. These libraries can be loaded either on startup (LD_PRELOAD) or loaded when needed (dlopen(3)).
Which option is preferred depends mainly on three factors:
- Startup performance: systems like
systemddelay loading of packages like lzma until it is needed in order to reduce initial startup time. - Security: By resolving all symbols before
mainis called, the GOT and the PLT can be marked as read-only. This removes a large vector for Return Oriented Programming. - Required dynamism: Often a program will want to decide which implementation to load by doing some computation.
This does not preclude using
LD_PRELOAD, but doing so would mean that the decision would need to be made before the process is started. Often a bash script or a launcher will be used for this, but the developer ergonomics of these solutions may or may not be desirable.
Large frameworks
Large framework libraries (e.g. GTK, QT, X11) have slightly different restrictions on dynamic loading. The primary distinction is that it is highly undesirable for multiple versions of these libraries to be loaded at the same time. These frameworks provide objects that the program can be interacted with. Methods on these objects may change implementation, and thevtable attached
to an object may have entries added arbitrarily.
The desired performance characteristics also differ.
Many function and method calls will cross the linking boundary.
As a lot of these libraries are UI frameworks and therefore real-time systems,
the predicability of overhead is important, but with generous timing
requirements.
Plugins
Plugin systems are similar to taking the other side of large frameworks. An important requirents for a lot of plugin systems is the ability to unload plugins. Various safety methodologies can be employed to achieve safety in this context. A mechanism for explicitly declaring what state should be private to the plugin and what state is shared with the framework. Loading multiple instances of a library may also be desirable. Depending on the implementation of state sharing, this may be in conflict with loading a global handler.Linux kernel modules
Linux kernel modules are a concrete instance of plugins. Unloading modules is done unsafely, it is the modules responsibility to ensure that no dangling references remain after it has been unloaded. Given that the Linux kernel does not have a stable internal API, care must be taken to verify that the module has been compiled against the correct kernel version.libc
Most programs need to be linked against the C standard library. Developers oflibcs are often careful to maintain ABI compatibility, with infrequent
changes to the interface. Performance overhead becomes important because most
interactions with the system are done over libc. The interface of libc is
captured by c types, but projects like Redox OS
may want to extend their standard system interface to include more complicated
types.