Why are process and thread IDs multiples of four?


On Windows NT-based operating systems, process and thread IDs happen always to be a multiple of four. Is this just a coincidence?

Yes, it’s just a coincidence, and you shouldn’t rely on it since it is not part of the programming contract. For example, Windows 95 process and thread IDs were not always multiples of four. (By comparison, the reason that kernel handles are always a multiple of four is part of the specification and will be guaranteed for the foreseeable future.)

Process and thread IDs are multiples of four as a side-effect of code re-use. The same code the allocates kernel handles is also used to allocate process and thread IDs. Since kernel handles are a multiple of four, so too are process and thread IDs. This is an implementation detail, so don’t write code that relies on it. I’m just telling you to satify your curiosity.

Comments (51)
  1. Scott says:

    But if somebody has already written code that relies on that fact, then wouldn’t you end up being tasked with maintaining it for backwards-compatibility?

    [Hence the need for the big warning text in this article. -Raymond]
  2. Xepol says:

    Sounds suspiciously like a memory aligned pointer masquarading as a magic number.  

    Since it is faster to just work with the pointer than to look a pointer up from an actual arbitrary handle, it isn’t a huge surprise when the pointer is misused like that – even if it is deperately bad idea with all sorts of negative consequences.

  3. Adam says:

    I bet you dollars to doughnuts that somebody will read this, think "oh, that’s interesting," ignore the warning, and write code that depends on it.

  4. Keithius says:

    Thanks for satisfying my curiosity! (No, seriously, I really appreciate it – I love this sort of stuff!)

  5. Yuhong Bao says:

    >Sounds suspiciously like a memory aligned pointer masquarading as a magic number.  

    In fact, Win9x does just that for it’s PIDs and TIDs. They are pointers to internal data structures XORed to obfuscate them. Early builds of Win95 did not obfuscated them, the obfuscation was in response to books like Unauthorized Windows 95.

    [Please do not use the comments section of my site to make it easier for people to rely on undocumented behavior. If you want to do that, do it on your own blog. -Raymond]
  6. anonymous says:

    and write code that depends on it.

    This has already been done, and is called ntoskrnl.exe. Yes, by modifiying the (undocumented) EPROCESS structure one can easily create an unkillable process. Gladly this requires loading a kernel-mode driver and therefore admin privileges.

  7. KristofU says:

    “[Please do not use the comments section of my site to make it easier for people to rely on undocumented behavior. If you want to do that, do it on your own blog. -Raymond]”

    Sorry Raymond, but any which way you twist this, this is a sad reply.

    You can do it, but he can’t?

    Well yes he can, but not here?

    C’mon he’s even talking about Windows 95.

    [I explicitly note that the behavior is not supported and is provided solely to satisfy the reader’s curiosity. Yuhong Bao is encouraging people to do undocumented things. I hope you see the difference. -Raymond]
  8. Warren says:

    KristofU, it’s a reasonable request — his house, his rules.  If Mr. Chen said, "don’t smoke in my house", then you wouldn’t smoke in his house, regardless of whether you smoke in your own his, or he smokes in his house.  It doesn’t matter.  You wouldn’t want someone being disrespectful of you and your decisions in your own house, would you?  Offer Mr. Chen the same.

  9. Nar says:

    >Yuhong Bao is encouraging people to do undocumented things

    Uh, what? What comment are you reading? There’s no encouragement there. There’s no discouragement, either, but that’s not the same thing. Mostly just an indifference to courage ;)

    [My mistake. I didn’t follow the pages that were linked to. But the whole discussion is off topic anyway. The topic is “Why are the values multiples of four? It’s an artifact of code re-use.” It’s not “How do I de-obfuscate Windows 95 process IDs so I can access internal structures?” or “Let me link to APIs that were newly-documented in Windows XP SP1.” Yuhong Bao has a rather well-established record of going off topic. -Raymond]
  10. Nick Mason says:

    How can this be classified as a "misuse" of a pointer?  Using it to your advantage, more like.  Its use might have simplified or sped up some things, but surely, the whole point of an ID or handle is that it uniquely identifies something. So long as the ID is within the specified range and is unique, that’s all that matters.

    Anyone who noticed that the ID is always a multiple of four and wrote code that specifically relied on this behavior is a fool; they deserve all they get.  Next time I fill up my car at some random station, I’ll blindly go for the pump that’s second on the right – after all, it’s always the diesel one.

  11. DriverDude says:

    "Anyone who noticed that the ID is always a multiple of four and wrote code that specifically relied on this behavior is a fool; they deserve all they get."

    Unfortunately, as Raymond has pointed out repeatedly, the programmer or vendor rarely pays for their foolishness; it is the users who run into incompatibilities when they upgrade and Microsoft who (sometimes) has to fix somebody else’s bug.

    That is why I prefer full disclosure instead of (or in addition to) invisible application-compatibility hacks.

  12. Bryan says:

    "Next time I fill up my car at some random station, I’ll blindly go for the pump that’s second on the right – after all, it’s always the diesel one."

    This is an excellent analogy the next time someone tries to use undocumented behavior for no real reason.

    I’m not sure what you’d ever do with knowing that PIDs & TIDs are multiples of 4 by coincidence or otherwise.  I guess I can’t think of a scenario you would misuse this information.  Obviously there is one though.

  13. Yuhong Bao says:

    "Windows 95 process and thread IDs were not always multiples of four."

    That was what reminded me that it was actually a XORed pointer.

  14. Francisco Arevalo says:

    Yuhong Bao’s comment was as informative as your post. And that analogy was an excellent way to see what blindly using an undocumented random detail.

    Had anyone noticed the "multiples-of-four" thing before you read it here?

    I use "tasklist" everyday and didn’t even notice they were only even numbers. Let alone multiples of four.

  15. Xepol says:

    Evan -> I had hoped that OS developers had learned their lessons from OSes in the early 90ies and late 80ies that could not handle memory expansion beyond a certain point because they had decided they could use the top x bits of a pointer for "other things".  

    That said, you could be right.  On the other hand, if handles are memory aligned pointers, you get the storage space for 2 bits for free and can easily map them out with an AND mask.  I guess it’s easy enough to find out- just check how 64bit windows treats its handles.  If you find they are all aligned on 8 byte boundaries and you suddenly get a extra spare bit in the bottom…

  16. Yuhong Bao says:

    "I had hoped that OS developers had learned their lessons from OSes in the early 90ies and late 80ies that could not handle memory expansion beyond a certain point because they had decided they could use the top x bits of a pointer for "other things"."

    On Mac OS versions before System 7, for example, the memory manager uses the top 8 bits for flags.

    It took a while to fix these apps that was not 32-bit clean because they assumed that the upper 8 bit had flags.

    In the meantime the Mac II by default had an AMU or HMMU that could translate 24-bit addresses to 32-bit addresses. Mac II could be upgraded with a 68851 PMMU for OSes like A/UX, and in this case it used the PMMU to do this translation. Most later Macs, thanks to the 68030 processors, had the PMMU built-in and thus the AMU was not needed.

  17. Igor Levicki says:

    Just kidding :)

    Seriously, who needs those extra two bits of information?

    I cannot imagine how this can save anything (in terms of performance) considering the need for OR-ing and AND-ing in order to insert and extract that information as opposed to using additional one byte variable.

  18. Evan says:

    @Xepol: "Sounds suspiciously like a memory aligned pointer masquarading as a magic number."

    Nah, it’s probably that someone wanted to be able to store two bits of information and didn’t want to use another variable.

    From the excerpt of ntdef.h in Raymond’s other article *that he linked to there*:

    // Low order two bits of a handle are ignored

    // by the system and available

    // for use by application code as tag bits.

  19. dolph says:

    Empirically, it seemed that PIDs in NT 4 were multiples of 2 rather than 4.  An example of this is the PID of the system process, which (again empirically) seemed always to be 2.

  20. nksingh says:

    @Xepol:

    I don’t think it’s ever a problem to use the bottom bits of a pointer for flag.  All you’re doing in that case is forcing alignment on future consumers of your API, which isn’t usually all that terrible… especially when it’s the natural alignment for the data type in question.

  21. KJK::Hyperion says:

    I doubt this is ever going to change, anyway. Interix (the POSIX subsystem) uses odd pids to tell actual pids (which are all even) from virtual pids that should be translated by its internal process table first (because in Windows, exec() is much more efficiently implemented by creating a whole new process, not to mention security issues with support for setuid executables), and to my knowledge has done this since forever

  22. Igor Levicki says:

    If you have:

    int pid = 111;

    void *table;

    void *p = table[pid];

    Then in assembler it could look like this:

    mov eax, dword ptr [pid]

    lea esi, table

    mov edi, dword ptr [esi + eax * 4]

    But if pid was multiple of four:

    mov eax, dword ptr [pid]

    lea esi, table

    mov edi, [esi + eax]

    It simplifies address calculation and gives shorter instruction encoding. How much it affects the performance is something I would really like to know.

  23. mongo says:

    @dolph:

    Empirically, it seemed that PIDs in NT 4 were multiples of 2 rather than 4.  An example of this is the PID of the system process, which (again empirically) seemed always to be 2.

    This beautifully illustrates Raymond’s point. At some point in the evolution of the NT kernel, the PID/TID allocation strategy changed. Anything dependent on the exact (undocumented) nature of this strategy could have experienced problems.

    And of course, it could change again in the future.

  24. anonymous says:

    The good thing is that out of userland the last two bits are always ignored, including the process creation. That is, whatever you try, you cannot abuse any inconsistency.

  25. Neil says:

    Well, I guess the XOR obfuscation constant is ……01 because all my Windows 95 process IDs end in those two bits.

    16-bit Windows didn’t obfuscate any of its handles, which allowed cheeky programs to get away with various "shortcuts", such as

    * doubly-indirect dereferencing of a local movable handle

    * turning a global handle into a selector (in protected mode only, of course)

    For one private app I even did the dirty and peeked into USER’s data segment (obtained by getting the HINSTANCE of the desktop) to find out whether a given window had had FlashWindow called on it an odd number of times (without actually calling FlashWindow again, which obviously suffers from side-effects).

    I really would like to think Igor is just kidding, even though his next word is "Seriously".

  26. SuperKoko says:

    "

    I’m not sure what you’d ever do with knowing that PIDs & TIDs are multiples of 4 by coincidence or otherwise.  I guess I can’t think of a scenario you would misuse this information.  Obviously there is one though.

    "

    Just follow the link in Raymond’s article, and you’ll get a sample of use for HANDLEs (rather than process Ids):

    http://blogs.msdn.com/oldnewthing/archive/2005/01/21/358109.aspx

    Igor wrote:

    "

    void *p = table[pid];

    "

    A process Id may be something as big as 4000000000 (especially on Windows 98). Indexing tables with pid is insane.

    "

    It simplifies address calculation and gives shorter instruction encoding.

    "

    How?

    Did you read the IA-32 architecture manual?

    [esi + eax] as well as [esi + eax*4] both require two bytes (ModR/M + SIB). I don’t see why one would be slower than the other.

    Neil wrote:

    "

    Well, I guess the XOR obfuscation constant is ……01 because all my Windows 95 process IDs end in those two bits.

    "

    I don’t remember the whole story, but I think that Windows 98 or Windows 98 SE use an improved XOR obfuscation with a random number generated at boot time.

  27. Igor Levicki says:

    >>A process Id may be something as big as 4000000000<<

    I believe that is impossible:

    MSDN: “The number of threads a process can create is limited by the available virtual memory. By default, every thread has one megabyte of stack space. Therefore, you can create at most 2,028 threads.”

    So if you can’t create more than 2,028 threads, and each process needs at least one thread that means you are topped at 2,028 processes as well.

    2028 * 4 = 8112 which is 80 bytes less than two 4KB pages of RAM.

    >>Did you read the IA-32 architecture manual?<<

    Of course, the other would be slower because AGU has to do the multiplication of eax by the scale factor (4) to generate the load address. As I already said, I doubt it has any impact on modern CPUs.

    [You read the conclusion without understanding the logic that led to it. There is no limit on the number of threads; the limit is process address space. -Raymond]
  28. Igor Levicki says:

    >>So if you can’t create more than 2,028 threads, and each process needs at least one thread that means you are topped at 2,028 processes as well.<<

    Oops… that should have read:

    So if you can’t create more than 2,028 threads in one process (beause of 2GB per process limit) and each process needs at least one thread, that means you are capped by the amount of physical RAM available for stack.

    Given that the smallest stack allocation is 4KB and assuming 32-bit address space:

    4,294,967,296 / 4,096 = 1,048,576 PIDs

    Since they have to be a multiple of 4:

    1,048,576 / 4 = 262,144 PIDs

    Having 262,144 processes would consume 1GB of RAM just for the initial stack commit assuming that all processes are single-threaded. If they commited 1MB of stack each you would need 256 GB of memory.

    So, good luck avoiding PID reuse and reaching 4,000,000,000 of processes.

    [Who said stacks couldn’t be paged out? Or that all stacks had to fit into a 32-bit address space? Or that PIDs are always allocated as “lowest available”? Or that a process has to have a thread? (It doesn’t.) -Raymond]
  29. Xepol says:

    To those who don’t understand why handing pointers to programmers is a bad thing need to read this blog more.

    Any time you expose an actual undocumented internal detail, someone will abuse it to work with said undocumented internal detail, causing long term forward compatibility issues.

    The only 2 possible solutions are to change things in the future and let the apps crash or to not expose the internal details with a pointer and use a true "magic number" as the handle instead.

    Too often it is easy to be lazy in design because it is "convenient", but all too often, it will bit you in the butt sooner than later.

  30. Triangle says:

    "The only 2 possible solutions are to change things in the future and let the apps crash or to not expose the internal details with a pointer and use a true ‘magic number’ as the handle instead."

    Or you could patch the offending program, which seems to be working passably for Microsoft.

    Drawbacks:

    • Encoding and decoding handle values is expensive performancewise.

    • If, like most programmers, you encounter a bug in your program, you can deference the pointer to access the members of the structure, which can provide amazingly useful information for finding the bug.

  31. Grumpy says:

    Igor Levicki dribbles:

    ] Seriously, who needs those extra two bits of information?

    ] I cannot imagine how this can save anything (in terms of performance) considering the need for OR-ing and AND-ing in order to insert and extract that information as opposed to using additional one byte variable.

    Just because you are unimaginative, and have probably never written any substantial software, doesn’t mean that the rest of us are that boring.

  32. Igor Levicki says:

    >>Grumpy can just point to…<<

    Link is broken.

    >>describes one application that uses…<<

    It is one thing to describe application, and completely another to provide some performance data for that application and prove how much has been saved/gained by using low two bits of a PID to store data. Sorry, but I want some numbers from Grumpy, not a description. He said I was boring so I now have to keep up to his expectations ;-)

    >>I still don’t see how your argument proves that a PID whose numerical value is greater than 0x40000000 is impossible<<

    There must be some language barrier at work here.

    You say that PID can be larger than 1,073,741,824? Of course it can, but there can’t be more than 1,073,741,824 PIDs. And assuming each process consumes at least 4KB of physical RAM that number goes down considerably in practice (I guesstimated it to be ~262,144).

    Of course you can argue that even those 4KB can be swapped out and that you can indeed have 1,073,741,824 processes running in Windows, but I sincerely doubt they would process anything.

    [Finding the correct link is left as an exercise, since it’s right here on this page. And you were the one who responded to “A process Id may be something as big as 4000000000″ with “I believe that is impossible.” Now you’re saying that when you wrote “impossible” you meant something else? I give up. -Raymond]
  33. Igor Levicki says:

    Just so I don’t forget…

    Grumpy said:

    >>Igor Levicki dribbles:<<

    >>Just because you are unimaginative, and have probably never written any substantial software, doesn’t mean that the rest of us are that boring.<<

    I asked that question in hope that someone who used such a dirty trick will provide some real-world example or performance data so I can expand my knowledge.

    Your use of argumentum ad hominem instead of facts just proved that you are not that someone (read: you are not an expert).

    [May I recommend that in the future when you want to ask a question, you ask a question instead of stating the opposite as a fact and seeing if anybody corrects you. (And the issue is rarely performance. It’s borrowing space in someone else’s namespace.) -Raymond]
  34. Yuhong Bao says:

    "Of course you can argue that even those 4KB can be swapped out and that you can indeed have 1,073,741,824 processes running in Windows, but I sincerely doubt they would process anything."

    Indeed, if you are creating that many processes, something is seriously wrong. Here the limiting factors would be the amount of memory. And the amount of memory a process would take would be much larger then the amount of memory a thread takes.

  35. Yuhong Bao says:

    Can you add something about undocumented function here:

    http://blogs.msdn.com/oldnewthing/archive/2004/02/21/77681.aspx

    [You should well know my position about undocumented functions if you’ve read this blog for any significant length of time at all. -Raymond]
  36. Igor Levicki says:

    Grumpy, is that the best you’ve got?

    Are you saying that you have actually written an aplication which creates processes and threads so that it can use lowest two bits to store some data?

    >>Who said stacks couldn’t be paged out?<<

    Are you saying that COMMIT doesn’t really mean COMMIT or what?

    I understand that the system will eventually start paging stacks too (together with the rest of the minimum process working set) when the physical memory gets exhausted, but to me your reply sounds as if you are implying that Windows has been designed to allow infinite number of processes and threads when some practical limit *must* exist. I have only attempted to prove that such a limit is well below 4,000,000,000, not that my assumptions are exact.

    [Grumpy can just point to this earlier comment that describes one application that uses the bottom bit to store some data. As for paged-out stacks: I was responding to your previous comment that stack size was constrained by RAM. RAM is not the same as COMMIT. I still don’t see how your argument proves that a PID whose numerical value is greater than 0x40000000 is impossible. In fact, your claim is directly contradicted by Windows 95. -Raymond]
  37. Grumpy says:

    Igor quibbles:

    ] Your use of argumentum ad hominem instead of facts just proved that you are not that someone (read: you are not an expert).

    It’s not an ad-hominem attack.  An ad-hominem attack would be: "you’re a talentless whiner who has never contributed anything worthwhile to the human race, much less software."

    What I said is far more focused, and true: You have probably never dealt with any design interesting enough to make you think, hmmm, what could I do with two extra bits there?  And yes, I have, years ago.

  38. Alex says:

    <em>Mongo: Empirically, it seemed that PIDs in NT 4 were multiples of 2 rather than 4.  An example of this is the PID of the system process, which (again empirically) seemed always to be 2.</em>

    In Windows XP the PID of the System process is 4, so now you can generalize and say that the PID is a multiple of the PID of the System process.

  39. Igor Levicki says:

    >>It is unclear to me how the maximum number of PIDs has anything to do with this article<<

    I already admitted that I misunderstood “A process Id may be something as big as 4,000,000,000″ as “there can be 4,000,000,000 processes”. Should I apologize too?

    Since my curiosity has not been satisfied with this article could you please clarify under which circumstances PID can reach such an insane number?

    [Run Windows 95. PIDs on Windows 95 were regularly in the 4 billion range. -Raymond]
  40. Igor Levicki says:

    >>As for the “stating as fact” I wasn’t referring so much to that comment…<<<

    Then you shouldn’t put it there because it looked like you are defending Grumpy’s attitude which I hope wasn’t your intent regardless of how much you hate me.

    >>…as a subsequent one that claims as fact various characteristics of RAM, stacks, processes, threads, and the AGU – all of which were wrong.<<

    I already said above that all calculations were guesstimates, not facts.

    As for the AGU, if I remember correctly Pentium 4 AGU lacks the barrel shifter making indexed table lookups slower. I reserve the right to be wrong on this.

    [The issue wasn’t with your guesstimates. Sure, the numbers may not be precise. But you manipulated them based on faulty assumptions. E.g., you divided per-process address space by stack size to arrive at the maximum number of PIDs. This is a meaningless computation. -Raymond]
  41. Igor Levicki says:

    >>Run Windows 95.<<

    1. I can’t, my computer is too new for that.

    2. I was asking (and all my guesstimates were) about present, not the past.

    [Please read the title of this blog again. -Raymond]
  42. Igor Levicki says:

    >>This is a meaningless computation.<<

    It is meaningless in the context of searching for the highest PID number because it is based on false assumption(s), but not in the context of searching for the maximum number of processes which I admit may not be related to this topic, but for me it was in that moment. You have my apology for assuming that much and going off-topic.

    >>Please read the title of this blog again.<<

    Why not look at the first sentence of this article?

    “On Windows NT-based operating systems…”

    I thought that is what we were discussing here, not Windows 95.

    [It is still a meaningless computation for determining the maximum number of processes. And I’m not going to say “You don’t have to worry about PIDs bigger than 4 billion” because PIDs can be any 32-bit value. -Raymond]
  43. ANONOYMOUS says:

    > A process Id may be something as big as 4000000000.

    Well, except that the maximum PID is 2**16-1.

    Now 2**16/4*4 = 64 KB = 16 pages for a PID index table.

  44. Igor Levicki says:

    >>Now you’re saying that when you wrote “impossible” you meant something else?<<

    As a matter of fact I did. I meant MAXIMUM NUMBER OF PIDS, while he meant MAXIMUM PID NUMBER.

    >>ask a question instead of stating the opposite as a fact<<

    Well I thought that:

    “Seriously, who needs those extra two bits of information?”

    Can be considered a question. Moreover, I also thought that saying:

    “I cannot imagine how this can save anything (in terms of performance)…”

    Wasn’t me stating any facts because if I wanted to state some facts, then *I* would post some performance numbers or provide a link to the relevant information.

    Actually, it was just me wondering how that hack can do anything usefull from a *current* perspective (read: today on multi GB and multi GHz computer). In other words — it was not a statement much less factual.

    >>It’s not an ad-hominem attack…<<

    And what would be the difference between this and your previous claim about my qualifications except that first time you used somewhat “politically correct” speech?

    I’ll tell you — there is absolutely NO difference.

    Both of your claims are argumentum ad hominem because both of them do not contain any relevant facts, and both are equally insulting to me.

    >>You have probably never dealt with any design interesting enough to make you think…<<

    Unproven assumption.

    >>And yes, I have, years ago.<<

    Unproven claim of an anonymous person hiding behind the pseudonym “Grumpy”.

    So Grumpy, it turns out that your replies say more about your own credibility than they say about mine.

    [It is unclear to me how the maximum number of PIDs has anything to do with this article. As for the “stating as fact” I wasn’t referring so much to that comment as a subsequent one that claims as fact various characteristics of RAM, stacks, processes, threads, and the AGU – all of which were wrong. -Raymond]
  45. Igor Levicki says:

    Ok, then I will reverse-engineer ntoskrnl.exe to find out the actual limit.

  46. enough already says:

    Igor,

    Are you kidding? So you are going to spend your time reverse engineering ntoskrnl.exe so you can prove something to Raymond, yourself? I mean geez is it really that serious? Personally I can think of a ton of other things I would rather spend my time doing like making money doing real work, relaxing outside, spending time with my wife and kid even come to mind. Sorry but getting into a pissing match with Raymond just doesn’t rank up there as high on the list. Especially when the outcome regardless of whether I am right or wrong really just makes me look juvenile. Although I may be wrong and this is just my humble assumption Raymond is here to help and provide interesting anecdotes into Windows development. He has stated many many many times that he is not the windows authority and that nothing he says is official or necessarily researched or factual. Yet you on nearly an everyday basis take it upon yourself to prove him wrong. Igor I am going to break it down to you as simply as I can. You’re exchanges now come off like a teenager who just has to be right. Knock it off, while it was once funny to read your nonsense, it’s just getting old now.

Comments are closed.