Why not use weak linking to solve the retargetable library problem?


In response to the problem of creating a retargetable library, there was disbelief that the Windows linker doesn't support weak symbols. (I'm assuming they meant "loader", not "linker".)

Back in the days of 16-bit Windows, the loader used weak linking. If you imported a function and it didn't exist, you got a null pointer. You were then on the hook not to call the function unless you first checked that it was there. And it was a disaster. Programs crashed left and right because they didn't check whether the function actually existed before calling it. The designers of Win32 decided that if you wanted weak linking, you had to do it explicitly via Load­Library and Get­Proc­Address.

Okay, but now we've come full circle, because Load­Library and Get­Proc­Address of system DLLs is not permitted in universal Windows apps. Why not let universal Windows apps link to nonexistent functions, and put the burden on them to check the pointer before calling it?

Well, first of all, that's sort of taking a step backward. "Hey, here's a new programming platform. It's harder to use than the old one."

Second, how would you enforce this policy? The Windows App Certification Kit acceptance test would see that there is an attempt to import the Initialize­Critical­Section function. Now it needs to reverse-engineer the code to verify that the program never calls Initialize­Critical­Section. This eventually turns into the Halting Program, which is unsolvable.

Furthermore, the issue isn't that the Initialize­Critical­Section function doesn't exist. The function exists just fine. It's just that universal Windows apps in the Windows Store are not allowed to call it. This means that you have to come up with a way for the operating system to decide at run time whether the import table entry for Initialize­Critical­Section should be null or not. This means that the loader must now be aware of Windows Store policies, and whenever the Windows Store changes its policy, a system update needs to be delivered in order to implement that policy.

And even if you somehow got the loader to enforce Windows Store policies, you have to tell the loader, "Oh wait, this program didn't get installed via the Windows Store." If a program gets installed by means other than the Windows Store, then Windows Store policies don't apply. The loader would have to check somehow whether Windows Store policies are in effect for the app.

Therefore, the answer to the original question is twofold. First of all, no, we don't have weak symbols, on purpose; if you want weak symbols, you can do it yourself with Load­Library and Get­Proc­Address. Second, weak symbols don't actually solve the problem, because it leads to making low-level components enforce a high-level policy.

Comments (33)
  1. Cesar says:

    > (I’m assuming they meant “loader”, not “linker”.)

    They probably meant "dynamic linker". As in, the thing in Unix which links the dynamic libraries into your process when it's executed (or when you call dlsym()).

    Yes, this is probably yet another case of a developer coming from the Unix world. In this case, the Unix-based MacOS world, as shown by the next tweet in that link.

    1. Joshua says:

      I at least really did mean linker. My interpretation was the windows phone environment was running a stripped-down version of the windows OS libraries where the function didn't exist.

      So it is a policy problem, not a technical one. Well in that case, the answer should have been mass outcry that the policy is not wanted. And in fact that's what happened. Windows RT died, and the reason it died was lack of apps, and the lack of apps was due to the important ones not being ported. I could elaborate but there's no point. fin.

    2. Rosyna says:

      Yes, I did indeed mean linker in the sense of "dynamic linked" as dyld is responsible for dynamically linking binaries and hooking up symbols at runtime, if needed.

  2. Pierre B. says:

    Games want to save different gamers high-score and progress in different file. Obviously, NTFS cannot support having multiple files, that would be a low-level component enforcing a high-level requirement!

    It only is because of how the issue is framed. Having a manifest in an executable that specify which loader restriction it wish to conform to makes the design all low-level. That some of these 'loader-set' be designed and supplied by the Windows Store doesn't make a difference. Making it generic allows different behaviour to be designed (a set for drivers, a set for shell component, a set for untrusted app, a set for browser plugins...)

    Furthermore, you can easily design them to be revocable, with the revocation containing localisable user-level explanation and remediation steps. For the Windows Store, something like: "This application is obsolete. You will need to install a more recent version, if available, from the Windows Store."

    (Also, the Win16 fiasco, as I understand it, was due to the fact that the loader provided a NULL pointer for any and every function it failed to find. Any function pointer could suddenly be NULL. By having a known set of functions be weak-loaded and have the set be defined, enforced and aggred upon using manifests, makes the application squarely responsible. Furthermore, the Windows Store cannot possibly be accountable for every possible bugs a app can have?)

    1. Suppose the Store policy changes. Now a new policy file needs to be deployed to the device before any apps that take advantage of the new policy can be acquired from the Store. But Phone updates are notoriously slow. "Yes, Flappy Candy Mafia Farm with Friends is now available for Windows Phone, but you have to wait until the next update in order to get the new policy file that lets it run. There's nothing technically preventing it from running on the phone you currently have. Just needs an updated policy file." Hooray for non-agility.

      1. I think you *could* in principle solve that by having the Store sign the executables (if it doesn't do so already) and include relevant policy information, i.e., "this application references InitializeCriticalSection but is not permitted to use it". Or the Windows Store could ship the policy update itself rather than waiting on the usual update process.

        But that sounds to me like a lot more trouble than it would be worth, at least for this use case.

        (Now if the application could actually run inside a real sandbox, one with a robust runtime security boundary, that might be a different story. But I assume it was just too complicated to retrofit such a boundary into Windows.)

        1. Store apps do run at low IL, so there is an actual security boundary around the process. The issue is with in-process APIs like InitializeCriticalSection. There is no security boundary being crossed here, but the function is blocked from Store apps for policy reasons. (In this case, because we want to deprecate the old API in favor of the new one. Other reasons could be "Because this API can hang the UI if used incorrectly" or "Because this API is not available on Phone or Xbox".)

          1. Huh. I guess I had taken for granted that the policy was entirely motivated by security. That makes more sense, then, though I note my first not-quite-serious technical proposal would still work. (The second one wouldn't, because updating the policy could improperly invalidate old apps if they were released before a particular API was disallowed.)

            I've never been quite sure whether IL was officially a security boundary or not. On the one hand, I've seen reports that known UAC bypasses (which must presumably also be bypassing IL) have been rejected by MS security as won't-fix. On the other, I seem to recall at least one security update released to address an IL bypass. Perhaps it depends whether the problem is visible through kernel-coloured glasses or not? :-)

          2. Joshua says:

            @Harry Johnson [apparently we have a nest limit]

            I've seen them too. They are for things like website server code with less than full trust or silverlight apps busting out of the sandbox.

          3. @Joshua, I'm thinking more of the "get from an restricted command line to an elevated one" scenarios. Not sure if we can post links here, but see for example the article "Bypassing UAC with PowerShell" from "Lab of a Penetration Tester", easily found in Google.

            Basically it's just a matter of abusing auto-elevating COM objects and executables. From a kernel perspective, everything is working as designed, but the upshot is that UAC, in the default configuration, is insecure by design.

            ... as far as I know, though, all such attacks can be blocked by setting UAC to "Always Notify". And I assume Windows Store policy prevents Store apps from taking advantage of them.

      2. Pierre B. says:

        That's a Windows Store problem: does it support multiple version of an app, are app download based on the version of the Windows Phone running, are policies changed willy-nilly. There is also no fundamental reason why the policies need to be tied to the OS and not updatable directly from the store. I mean, I expect Microsoft to trust its store can securely msnage the signing of its policies and their distribution? I'd expect policies changes to be rare anyway.

        1. That's sort of the whole point. The Store is in charge of its policies. The operating system doesn't care how the app got onto your device.

    2. exchange development blog team says:

      So why did the Win16 loader provide a NULL pointer? The app has asked for a pointer to X, the loader knows it can't satisfy the request (meaning that the app will crash if it tries to use it), but instead of not loading the app it returns a pretend pointer that'll cause a crash if dereferenced. It's like going into a restaurant and ordering a nice piece of fish and the waiter brings you a flaming phone book instead.

      1. Drak says:

        @Dave: possiblybecause you could be using a newer version of a DLL (say win 98 vs win 95) and you could then take advantage of new functions in said DLL. In the Win95 version you could check for NULL on such a new function and call your own slower workaround function instead.

        At least that's what I think it would be for.

      2. Neil says:

        Actually the loader didn't; as per https://blogs.msdn.microsoft.com/oldnewthing/20060717-13/?p=30503 it wrote the address of a function that displayed the "Call to Undefined Dynalink" fatal error dialog.

        The above function itself was actually exported, so you could in theory compare your function pointer to it to see if it was defined or not.

    3. MarkKB says:

      "Games want to save different gamers high-score and progress in different file. Obviously, NTFS cannot support having multiple files, that would be a low-level component enforcing a high-level requirement!"

      Perhaps I'm a bit daft (always a possibility) but this argument makes no sense to me. A game is well within its rights to store its data however it likes. It can, for example, store its high scores and progress in the same file in shell:savedgames. It doesn't even have to do that - it could append its data to the end of its executable, much like how self-extracting packages work. Or it could store the data in a binary blob in the registry. Or store it as a hidden stream.

      From my point of view, this is the very definition of a high-level component enforcing a high-level requirement - by choosing which APIs it calls in which manner, the program (or reliant library) has enforced upon itself the requirement of multiple files (or not, as the case may be.) The existence of multiple files in no way necessitates the usage of it.

      (Besides, the feature of files in a general-purpose operating system is, by my understanding, the very model of a low-level feature - programs need some way of determining what libraries or data to load in order for them to operate correctly - some sort of sectioning is needed to determine where to start loading and where to stop. "Files" was the chosen metaphor to describe and represent this sectioning, and the NTFS file system format determines the method of sectioning for Windows computers, but you could use different names and different formats and still get the same results. Files only exist in software which interprets the binary blob contained in the hardware and then exposes the results to other programs so that they may query for the appropriate data - which they must be able to do in some manner, otherwise it would be impossible for them to function.)

      1. MarkKB says:

        Urgh, I don't think I actually summarised my point there. What I'm trying to say, simply put, is

        a) you're using a situation where a developer has a choice at the high level in the matter to contrast one where that decision is subsumed into the low-level, and

        b) file systems are essential for general-purpose operating systems, whereas weak-linking is not.

  3. —'... you have to tell the loader, “Oh wait, this program didn’t get installed via the Windows Store.” ...'

    I was under the impression that the forbidden API is forbidden for all Metro-style apps (both universal and non-universal ones) regardless of whether they come from Windows Store or are sideloaded. Was I wrong?

    1. SimonRev says:

      I haven't looked into it much since Windows 8 first came out, but at the time, I am fairly sure most of the API restrictions were policy based. Basically if you hacked your header files you could use the forbidden APIs, (although the store would validate your application and reject it if it did indeed use forbidden APIs). From that it follows that sideloaded applications could bypass those restrictions. Under Windows 8 sideloading wasn't very easy so there weren't a lot of apps that chose to go that route.

      I suppose this actually opens up some interesting possibilities. I always wanted a live tile that showed the battery status of my laptop, but no one made such a Windows 8 app (at least a good one, there were some bad ones) since the battery query API was forbidden for some reason.

      1. Brian says:

        It seems to me that I remember someone explaining the reason for the lack of a "check the battery level" api. There was a fear that programmers would poll that API incessantly, causing excessive battery drain (if you remember, the original WinPhone 7 platform only showed the battery level on demand - I believe for that very reason).
        I believe that API is forbidden in an effort to inhibit irony.

        1. SimonRev says:

          Ironic perhaps, but I found it incredibly aggravating on my 8" Windows 8 tablet that the only way to see the battery percentage was to go to the desktop and try to tap the little battery icon to get the tool tip to appear.

  4. Myria says:

    Our solution is to not use the Windows Store. Our application uses runtime code generation, which is fundamentally not allowed by the Windows Store.

    1. Pseudonym says:

      Now there's an excellent example of low-level components enforcing a high-level policy!

    2. Darran Rowe says:

      Except it is.
      https://msdn.microsoft.com/en-us/library/windows/desktop/mt169845(v=vs.85).aspx for example and it makes reference to the codeGeneration capability. This was added with Windows 10.

  5. Rosyna says:

    "Second, how would you enforce this policy? The Windows App Certification Kit acceptance test would see that there is an attempt to import the Initialize­Critical­Section function"

    This appears to be creating a problem that doesn't exist? The initial issue was Initialize­Critical­SectionEx existed on newer platforms but not older ones. Weak linking would address that issue as it's exactly how it's addressed in other platforms.

    "Furthermore, the issue isn’t that the Initialize­Critical­Section function doesn’t exist. The function exists just fine. It’s just that universal Windows apps in the Windows Store are not allowed to call it"

    Doesn't the Windows Store check the functions imported by an app submitted to the store? That's the way forbidden APIs are usually prevented from being used in other app stores. This usually uses the same App Store infrastructure that runs basic heuristics on submitted apps as a front line to make sure malware is not submitted.

    If this kind of submission checking is used, then if a currently permitted or forbidden API changes status, the only thing that needs to change is the server-side list the Windows Store uses to validate submissions.

    1. This problem exists: The InitializeCriticalSection function exists in Windows 10 (hence would be a strong link), but it is disallowed in Store apps. Therefore, a Store app that links to it will be rejected by Store validation.

      1. Rosyna says:

        I'm not sure I understand.

        Weak linking isn't designed to be used in the case of InitializeCriticalSection, since that function is guaranteed to be available at runtime on all versions of Windows concerned. Weak linking is designed to solve the problem of InitializeCriticalSectionEx not being available on Windows XP when an executable declares (this is important) it can run on Windows XP. Without weak linking, you'd need shim libraries or dynamic runtime loading use GetProcAddress (as you state).

        Weak linking isn't suitable to be used in a situation of an API being banned (by policy) if the API actually exists. GetProcAddress isn't suitable for that use either.

        If an executable declares it can run only on versions of an OS that are guaranteed to have a certain API, then that API is never weak linked, only extremely strong linking is used (for lack of a better term). This is to prevent the issue of function addresses randomly being NULL due to configuration errors/problems that I believe you were referring to in the Win16 days.

        1. If your DLL is running on Windows XP, then you can't use InitializeCriticalSectionEx, and if it has to also run on Windows Store, you can't use InitializeCriticalSection either. So how are you going to initialize your critical section? (And if you don't need one, then you don't have a problem in the first place.)

          1. smf says:

            Conditional compilation and ship two dll's is probably the only solution.

            dllimport in .net ironically does allow weak linking. You can even link to different named functions in different dll's using delegates, so the same dll can be used on windows & windows ce.

            When your use case doesn't fit in with Microsoft's idea of software development then you may find yourself swimming against the current.

        2. > Weak linking isn’t designed to be used in the case of InitializeCriticalSection
          Then why did you suggest it as a possible solution?

    2. Note that you don't need a shim DLL for every Windows version. You need one for a desktop app and one for a universal app, and that's all. If you're building an application, you don't even need that, because you already know whether you're a desktop app or a universal app - this is only an issue for people building libraries. It needn't really be a shim DLL either, you could simply build two different versions of your library. I'm not sure why the original customer didn't want to do that.

      The correct Mac analogy here, I think, is the shift from PPC to x86, or from MacOS 9 to MacOS X, rather than simply supporting multiple versions of MacOS X.

      1. ... or, more to the point, the shift from MacOS X to iOS. D'oh.

  6. henke37 says:

    For the record, Visual Studio lets you pretend that static linking is the same as using LoadLibrary+GetProcAddress. It will do the job for you by the use of generates stubs. It's your responsibility to either not call functions that doesn't exist or provide an error handler (yeah, right).

Comments are closed.

Skip to main content