If it’s not yours, then don’t mess with it without permission from the owner


It's surprising how many principles of real life also apply to computer programming. For example, one of the rules of thumb for real life is that is that if something doesn't belong to you, then you shouldn't mess with it unless you have permission from the owner. If you want to ride Jimmy's bike, then you need to have Jimmy's permission. Even if Jimmy leaves his bicycle unlocked in his driveway, that doesn't mean that it's available for anyone to take or borrow.

In computer programming, the code that creates an object (or on whose behalf the object is created) controls what is done with the object, and if you're not that component, then it's only right to get the permission of that component before you start messing with that it thought was its private property.

Application compatibility is, in large part, dealing with programs which violate this rule of civilized society, programs which directly manipulate the contents of list views they did not create, use reflection to access private members of classes, that sort of thing. But I won't use that as the motivating example this time, because you're all sick and tired of that.

Instead, let's look at the low-fragmentation heap. The question is, "Under what conditions can I convert a heap to a low-fragmentation heap?"

Well, if you called Heap­Create, then that heap is yours and you decide what the rules are. If you want that heap to be a low-fragmentation heap, then more power to you.

If you didn't call Heap­Create then that heap doesn't belong to you; you're just a guest. But of course the owner of the heap can grant permission to you, at which point you are free to do whatever it was the owner said you could do. If Jimmy says, "You can borrow my bike if it's just sitting in the driveway," then you can borrow his bicycle if it is just sitting in the driveway. But if it's in the garage, then you can't borrow it. And even if it's sitting in the driveway, you can't sell it. You can only borrow it.

Okay, let's look at heaps again. If you are an executable, then the process heap was created on your behalf. (This is not obvious, but that's the guidance I've received from the people who work with this sort of thing.) Therefore, if you want, you can call Get­Process­Heap and convert that heap to a low-fragmentation heap. It's the heap for your process, so if you want it to be a low-fragmentation heap, the heap folks say that's okay with them.

On the other hand, if you're writing a DLL, then the process heap does not belong to you, nor was it created on your behalf. It belongs to the executable that loaded your DLL, and it is that executable which decides what type of heap it wants. If you would prefer that your DLL use a low-fragmentation heap, you can include that in the guidance in your DLL's documentation, but be aware that the process heap is shared with all DLLs in the process, so the hosting application may not be able to comply with your guidance if it is also using another DLL whose guidance documentation says that it should not be used with a low-fragmentation heap. If a low-fragmentation heap is really important to your DLL, then you can create your own heap with Heap­Create and set it into low-fragmentation mode. When you create a heap with Heap­Create, it's your heap, and you get to decide what the rules are.

If you use the C runtime library default heap, then that heap is under the control of the C runtime library, and you don't have the rights to change its parameters. However, the C runtime library is one of the examples where you're allowed to use an object that's not yours if you have permission from the owner: The _get_heap_handle function was specifically created so that you could convert the heap to a low-fragmentation heap. But now that you've unwrapped one layer of ownership, there is still the matter of which of the C runtime's clients is the decision-maker with regard to how that heap is to be configured?

Remember that a DLL is a guest in the host process. You don't go changing the carpets in someone's house just because you're visiting.

If you linked the C runtime library statically, then you are the only client of that heap, and you are therefore free to convert it to a low-fragmentation heap. (If you bring your own towels to someone's house, then you are free to abuse them in any manner you choose.) On the other hand, if you linked the C runtime library dynamically, then you're using the shared C runtime heap, and the authority to determine the mode of that heap belongs to the hosting executable.

Comments (19)
  1. Falcon says:

    "You don’t go changing the carpets in someone’s house just because you’re visiting."

    Why does that sentence make me think of Marie Barone from Everybody Loves Raymond?

  2. Karellen says:

    "If you are an executable, then the process heap was created on your behalf."

    "If you use the C runtime library default heap, then that heap is under the control of the C runtime library,"

    "Remember that a DLL is a guest in the host process."

    So, the C runtime library, which is a DLL, and therefore a guest in any process which dynamically links with it, creates its own default heap, which is different from the processes default heap?

    Actually, that does make kind of sense. Which is why malloc()/free() (which presumably works with the C runtime library heap) is separate from CoTaskMemAlloc()/CoTaskMemFree() (which I imagine works with the process heap)

    Yes? No? Seems weird though…

  3. Alexandre Grigoriev says:

    @Karellen,

    C runtime library DLL is not a guest, it’s a mother-in-law… Other guests even may bring their own MIL…

  4. arnshea says:

    @Karellen, I see this a lot.  In language it’s called overregularization.  In this case it’s an over-application of the principle that DLLs are guests in the process’ house to the C runtime library DLL.

    The special-ness of the C runtime library in this context seems obvious but some people need it spelled out.  I suspect the AppCompat team owes the bulk of its reason for existence to these kinds of people. #rant off

  5. Leo Davidson says:

    @Karellen, A DLL (or whatever) is allowed to create its own heap and then give it (or APIs which call it) to other people to use. That’s all the CRT is doing (unless it is using the process heap, which it is allowed but not guaranteed to do as well).

    The important things are that you don’t mess with someone else’s heap’s settings without knowing you are allowed to, and that you don’t allocate memory from one heap and try to free it from another.

  6. Tom says:

    This title is suspiciously relevant to current events.  Coincidence, or could this be Raymond’s way of weighing in on the drama surrounding a certain gadget that recently met an unfortunate end? :)

    [Seeing as this was written and scheduled 18 months ago, I prefer to describe its appearance as clairvoyant. -Raymond]
  7. Billy O'Neal says:

    @Tom: To what current event are you referring?

  8. Paul says:

    @ Billy:

    "If you want to use Steve’s iPod then you need to have Steve’s permission. Even if Steve leaves his iPod unattended in a bar that doesn’t mean that it’s available for anyone to take or borrow."

  9. Karellen says:

    @arnshea: So… the C runtime is different from other DLLs, but it’s still creating its own heap separate from the process heap (that is what Raymond said, right?) which it owns instead of you, but you can get permission to mess with if you want, which you could do for any heap created by any other DLL if it provided an API for that purpose?

    [There’s nothing special about the C runtime. It is a guest in the host process. It creates a heap, which under the general rules of polite society therefore belongs to the C runtime. The C runtime _get_heap_handle function is explicitly documented as granting the caller permission to use the handle to change the heap to a low fragmentation heap. Thus, the convention in the subject line is adhered to: “Don’t mess with it without permission from the owner.” I can’t believe I had to write that. -Raymond]
  10. person says:

    Is it ok to view porn on someone else’s computer without asking? What if I really really need to?

  11. 640k says:

    Why should the app care about low-fragmentation heap? Can’t the OS fix/defrag this in the background itself with some smart memory mapping stuff. Please remove this “feature” from windows 2010.

    [Do as much “memory mapping stuff” as you want; that won’t make the value of 500−280 any smaller, which is the source of fragmentation. I guess you could make everbody use memory handles and explicitly lock the memory before using it. Then people will wonder why you’re reinventing Win16. Though I’m impressed by the exhortation to fix a problem by something as vague as “smart memory mapping stuff.” It’s like the Monty Python episode where they explain how to rid the world of all known diseases. -Raymond]
  12. Arnt says:

    Hey Raymond,

    thanks for keeping your articles going, even through regular doubts of the ‘is this still worth doing’ kind.

    I love seeing these low level things I never knew about. A low fragmentation heap, I’ve been developing software for a long time and never heard of it… And all the discussion here gets me thinking, why do dll’s not just always create their own heap, what is the advantage of having the same/a different heap, …

    Good stuff, thank you!

  13. GregM says:

    Arnt, "why do dll’s not just always create their own heap, what is the advantage of having the same/a different heap, …"

    Because they share memory blocks with the rest of the process.

  14. GregM says:

    [There’s nothing special about the C runtime.  …  I can’t believe I had to write that. -Raymond]

    Raymond, I think you and Karellen are in violent agreement here,  My reading was that Karellen was pointing out to arnshea that the C Runtime was no different than any other DLL, exactly as you described.

  15. 640k says:

    > [Do as much “memory mapping stuff” as you want; that won’t make the value of 500−280 any smaller, which is the source of fragmentation. I guess you could make everbody use memory handles and explicitly lock the memory before using it. Then people will wonder why you’re reinventing Win16. Though I’m impressed by the exhortation to fix a problem by something as vague as “smart memory mapping stuff.” It’s like the Monty Python episode where they explain how to rid the world of all known diseases. -Raymond]

    I thought that locking of memory was still a requirement in win32 because when I tried to remove it in user space software i wrote, some API (drivers?) stopped working because they assumed that you locked the memory.

    Btw, descriptor tables + selectors which translates all addresses several times could probably be used to move/defrag memory without the need for locking memory.

    [May I suggest that you think through your suggestions before making them instead of just tossing in some buzzwords and saying “I’m sure you can find some way to use this to solve all your problems.” -Raymond]
  16. Leo Davidson says:

    @640k, Having to lock memory in Win32 is vanishingly rare.

    Memory allocated using GMEM_MOVEABLE or LMEM_MOVEABLE still need to be locked/unlocked but getting/setting data on the clipboard is about the only place you’d typically run into that.

    And memory fragmentation is a far, far, faaaar lesser issue than the complexity of having to use descriptors/selectors and locking/unlocking for everything. I think there are better ways to reduce (if not solve) fragmentation with better trade-offs which don’t involve making every memory access ridiculously complex and error prone.

  17. Csaboka says:

    @640k: I think you are trying to reinvent managed memory (possibly without the automatic garbage collection). Virtually all GC systems compact the heap automatically, so you don’t need to worry about heap fragmentation in a managed memory environment.

    If you choose to use manual memory management, you must be willing to handle all the complexities that come with that. You can’t just blame the OS because it doesn’t do your job for you.

  18. Dean Harding says:

    I think 640k’s posts are meant to be jokes… I mean, suggesting we go back to the 16-bit memory model, "please fix this for Windows 2010", etc.

    Maybe I’m being generous?

  19. David Walker says:

    @640k:  Why would you have to "translate all addresses several times"?  Isn’t once enough?

Comments are closed.

Skip to main content