Understanding the classical model for linking: Taking symbols along for the ride


Last time, we learned the basics of the classical model for linking. Today, we’ll look at the historical background for that model, and how the model is exploited by libraries.

In the classical model, compilers and assemblers consume source code and spit out an OBJ file. They do as much as they can, but eventually they get stuck because they don’t have the entire module at their disposal. To record the work remaining to be done, the OBJ file contains various sections: a data section, a code section (historically and confusingly called text), an uninitialized data section, and so on. The linker resolves symbols, and then for each OBJ file that got pulled into the module, it combines all the code sections into one giant code section, all the data sections into one giant data section, and so on.

One thing you may have noticed is that the unit of consumption is the OBJ file. If an OBJ file is added to the module, the whole thing gets added, even if you needed only a tiny part of the OBJ file. Historically, the reason for this rule is that the compilers and assemblers did not include information in the OBJ file to indicate how to separate all the little pieces. It’s like if somebody said, “Can you get me a portable mp3 player?” and the only thing available in the library was a smartphone. Sure, it plays mp3 files, but there’s a lot of other electronic junk in there that you didn’t ask for, but it came along for the ride. And you don’t know how to disassemble the smartphone and extract just the mp3-player part.

This behavior is actually exploited as a feature, because it allows for tricks like this:

/* magicnumber.h */
extern int magicNumber;

/* magicnumber.c */
int magicNumber;

class InitMagicNumber
{
 InitMagicNumber()
 {
    magicNumber = ...;
 }
}
g_InitMagicNumber;

I’m not going to go into the magic of how the compiler knows to construct the g_Init­Magic­Number object at module entry; I’ll let you read up on that.

The point is that if anybody in the module refers to the magic­Number variable, then that causes magic­number.obj to be pulled into the module, which brings in not just the magic­Number variable, but also the g_Init­Magic­Number object, which initializes the magic number when the process starts.

One place the C runtime library took advantage of this was in deciding whether or not to include floating point support.

As you may recall, the 8086 processor did not have native floating support. You had to buy the 8087 coprocessor for that. It was therefore customary for programs of that era to include a floating point library if they did any floating point arithmetic. The library would redirect floating point operations from the coprocessor to the emulator.

The floating point emulation library was pretty hefty, and it would have been a waste to include it for programs that didn’t use floating point (which was most of them), so the compiler used a trick to allow it to pull in the floating point library only if the program used floating point: If you used floating point, then the compiler added a needed symbol to your OBJ file: __fltused.

That magical __fltused symbol was marked as provided by… the floating point emulation library!

The linker found the symbol in an OBJ in the floating point emulation library, and that served as the loose thread that caused the rest of the floating point emulation library to be pulled into your module.

Next time, we’ll look at the interaction between OBJ files and LIB files.

Bonus reading: Larry Osterman gives another example of this trick.

Comments (15)
  1. Joshua says:

    FYI: you can't split by deciding when one symbol starts the previous ends. Symbols can share code. Most famously, malloc and realloc in certain old libraries were about 10 bytes apart.

  2. Joseph Koss says:

    @Joshua

    While it is certainly true that one procedure can "validly" spill into another, it could be easily argued that this was bad practice even when memory was tight. We are talking about an extra 5 bytes for the routine that spills to instead call the function that it spills into.

    Of course, back then programming was quite a bit more clever than it is today so the definition of "bad practice" was also different. It wasnt uncommon to calculate a code offset at runtime based on the size of the instruction(s) that repeatedly perform some operation in an unrolled loop manner (ex: the assembly precursor to Duffs Device)

    [The linker is language-agnostic. By the time the OBJ reaches the linker, it's just a bunch of bytes with symbolic labels attached. Some languages support multiple entry points for a single function. (Even C considered such support but eventually decided against it.) -Raymond]
  3. Evan says:

    Thanks, Raymond, for the g_InitMagicNumber example. I've wondered before, and again prompted by yesterday's post, about how useful it would be if the compiler did something like automatically split out each symbol into its own object file to reduce dependencies, but that would be a good reason why that wouldn't work. I've used stuff like your example on a few occasions (I even have "ConstructionRunner" and "DestructionRunner" classes that I pull around to various projects — you write 'void foo() { … }; ConstructionRunner dummy(foo);'), so I find it very sympathetic.

  4. @joshua:

    This is why gods gave the man the great power of COMDAT.

  5. Joshua says:

    [The magic is how the constructor manages to run when there is no explicit caller. -Raymond]

    Maybe I should show Raymond how to make code run without an implicit caller either.

  6. Myria says:

    @alegr1:

    It's sad that MASM doesn't have a way to declare COMDATs.  If you use /Gy /Fa on cl.exe's command line to generate function COMDATs and output assembly source, the COMDAT flag is only a comment.  If you were to assemble the output file, it wouldn't have COMDATs.

    @Brian_EE:

    Global constructors work in Visual C++ by the compiler putting a pointer to the constructor into a specifically-named .obj section.  Part of the msvcrt startup code goes through the list of pointers in this section and calls each function; in Visual Studio 2010, that's the function _initterm in crt0dat.c.  See also crt0init.c.

  7. "I'm not going to go into the magic of …."

    Wait. You created a global object of type InitMagicNumber, so the object must get constructed. How is that magic? Perhaps you mis-spelled *logic* ?

    [The magic is how the constructor manages to run when there is no explicit caller. -Raymond]
  8. Neil says:

    Thus the note in *printf pointing out that you couldn't use any of the floating-point formats unless you actually did some floating-point operation that caused the floating-point library to get linked in.

  9. Neil says:

    Another version of the trick is when you have a dllexport symbol in a library; a fake reference to the symbol in an unused function in an explicit object file will cause the exported symbol to get resolved. Firefox used to do this but now they just skip the intermediate library step and simply link all their bazillion object files together.

  10. @Myria:

    I understand how that works. I was questioning why Raymond considered it to be "magic".

    And in DSP-land the section is called .init_array and is called in c_int00.

    [It's even more magical than the __fltused magic, since it employs a trick beyond merely resolving symbols. -Raymond]
  11. Evan says:

    I suspect the use of "magic" to describe the operating of constructors and destructors is because by my understanding if you went back to, I dunno, 1980 runtimes, they wouldn't get run. The g_InitMagicNumber symbol would be present, but wouldn't do anything. (1980 is arbitrary. Maybe it'd have to be 1975 or something.)

    As Raymond's latest response (to Brian_EE) suggests, even something like __fltused fits into the classical model of linking, and you'd be able to pull that trick with 1980 linkers.

    [They get run even under the classical model, but by a technique not described in this series of articles (but described by Larry). -Raymond]
  12. Mark says:

    I don't get why you need __fltused.  I thought that something like 1.2 + 3.4 gets compiled into something like __fltadd(1.2, 3.4).  Then you can resolve __fltadd instead of a bogus symbol like __fltused.

  13. Gregory says:

    Why do we want magicNumber initialization to happen through g_InitMagicNumber exactly?

  14. @Gregory: Um, simply because it is a contrived example used to describe the issue in general terms maybe?

  15. Gregory says:

    Can someone come up with a more concrete scenario then?

    [See the entire printf discussion. -Raymond]

Comments are closed.