Processes, commit, RAM, threads, and how high can you go?

Back in 2008, Igor Levicki made a boatload of incorrect assumptions in an attempt to calculate the highest a process ID can go on Windows NT. Let's look at them one at a time.

So if you can't create more than 2,028 threads in one process (because of 2GB per process limit) and each process needs at least one thread, that means you are capped by the amount of physical RAM available for stack.

One assumption is that each process needs at least one thread. Really? What about a process that has exited? (Some people call these zombie processes.) There are no threads remaining in this process, but the process object hangs around until all handles are closed.

Next, the claim is that you are capped by the amount of physical RAM available for stack. This assumes that stacks are non-pageable, which is an awfully strange assumption. User-mode stacks are most certainly pageable. In fact, everything in user-mode is pageable unless you take special steps to make it not pageable.

Given that the smallest stack allocation is 4KB and assuming 32-bit address space:

4,294,967,296 / 4,096 = 1,048,576 PIDs

This assumes that all the stacks live in the same address space, but user mode stacks from different processes most certainly do not; that's the whole point of separate address spaces! (Okay, kernel stacks live in the same address space, but the discussion about "initial stack commit" later makes it clear he's talking about user-mode stacks.)

Since they have to be a multiple of 4:

1,048,576 / 4 = 262,144 PIDs

It's not clear why we are dividing by four here. Yes, process IDs are a multiple of four (implementation detail, not contractual, do not rely on it), but that doesn't mean that three quarters of the stacks are no longer any good. It just means that we can't use more than 4,294,967,296/4 of them since we'll run out of names after 1,073,741,824 of them. In other words, this is not a division but rather a min operation. And we already dropped below 1 billion when we counted kernel stacks, so this min step has no effect.

It's like saying, "This street is 80 meters long. The minimum building line is 4 meters, which means that you can have at most 20 houses on this side of the street. But house numbers on this side of the street must be even, so the maximum number of houses is half that, or 10." No, the requirement that house numbers be even doesn't cut the number of houses in half; it just means you have to be more careful how you assign the numbers.

Having 262,144 processes would consume 1GB of RAM just for the initial stack commit assuming that all processes are single-threaded. If they commited 1MB of stack each you would need 256 GB of memory.

Commit does not consume RAM. Commit is merely a promise from the memory manager that the RAM will there when you need it, but the memory manager doesn't have to produce it immediately (and certainly doesn't have to keep the RAM reserved for you until you free it). Indeed, that's the whole point of virtual memory, to decouple commit from RAM! (If commit consumed RAM, then what's the page file for?)

This calculation also assumes that process IDs are allocated "smallest available first", but it's clear that it's not as simple as that: Fire up Task Manager and look at the highest process ID. (I've got one as high as 4040.) If process IDs are allocated smallest-available-first, then a process ID of 4040 implies that at some point there were 1010 processes in the system simultaneously! Unlikely.

Here's a much simpler demonstration that process IDs are not allocated smallest-available-first: Fire up Task Manager, tell it to Show processes from all users, go to the Processes tab, and enable the PID column if you haven't already. Now launch Calc. Look for Calc in the process list and observe that it was not assigned the lowest available PID. If your system is like mine, you have PID zero assigned to the System Idle Process (not really a process but it gets a number anyway), and PID 4 assigned to the System process (again, not really a process but it gets a number anyway), and then you have a pretty big gap before the next process ID (for me, it's 372). And yet Calc was given a process ID in the 2000's. Proof by counterexample that the system does not assign PIDs smallest-available-first.

So if they aren't assigned smallest-available-first, what's to prevent one from having a process ID of 4000000000?

(Advanced readers may note that kernel stacks do all share a single address space, but even in that case, a thread that doesn't exist doesn't have a stack. And it's clear that Igor was referring to user-mode stacks since he talked about 1MB stack commits, a value which applies to user mode and not kernel mode.)

Just for fun, I tried to see how high I could get my process ID.

#include <windows.h>
int __cdecl _tmain(int argc, TCHAR **argv)
 DWORD dwPid = 0;
 GetModuleFileName(NULL, szSelf, MAX_PATH);
 int i;
 for (i = 0; i < 10000; i++) {
  STARTUPINFO si = { 0 };
  if (!CreateProcess(szSelf, TEXT("Bogus"),
        &si, &pi)) break;
  TerminateProcess(pi.hProcess, 0);
  // intentionally leak the process handle so the
  // process object is not destroyed
  // CloseHandle(pi.hProcess); // leak
  if (dwPid < pi.dwProcessId) dwPid = pi.dwProcessId;
 _tprintf(_TEXT("\nCreated %d processes, ")
          _TEXT("highest pid seen was %d\n"), i, dwPid);
 _fgetts(szSelf, MAX_PATH, stdin);
 return 0;

In order to get the program to complete before I got bored, I ran it on a Windows 2000 virtual machine with 128MB of memory. It finally conked out at 5245 processes with a PID high water mark of 21776. Along the way, it managed to consume 2328KB of non-paged pool, 36KB of paged pool, and 36,092KB of commit. If you divide this by the number of processes, you'll see that a terminated process consumes about 450 bytes of non-paged pool, a negligible amount of paged pool, and 6KB of commit. (The commit is probably left over page tables and other detritus.) I suspect commit is the limiting factor in the number of processes.

I ran the same program on a Windows 7 machine with 1GB of RAM, and it managed to create all 10,000 processes with a high process ID of 44264. I cranked the loop limit up to 65535, and it still comfortably created 65535 processes with a high process Id of 266,232, easily exceeding the limit of 262,144 that Igor calculated.

I later learned that the Windows NT folks do try to keep the numerical values of process ID from getting too big. Earlier this century, the kernel team experimented with letting the numbers get really huge, in order to reduce the rate at which process IDs get reused, but they had to go back to small numbers, not for any technical reasons, but because people complained that the large process IDs looked ugly in Task Manager. (One customer even asked if something was wrong with his computer.)

That's not saying that the kernel folks won't go back and try the experiment again someday. After all, they managed to get rid of the dispatcher lock. Who knows what other crazy things will change next? (And once they get process IDs to go above 65535—like they were in Windows 95, by the way—or if they decided to make process IDs no longer multiples of 4 in order to keep process IDs low, this guy's program will stop working, and it'll be Microsoft's fault.)

Comments (35)
  1. Leo Davidson says:

    "this guy's program will stop working, and it'll be Microsoft's fault"

    He can't say he wasn't warned.

    I remember that thread well. The particularly bad thing is relying on undocumented behaviour when there are easier, more obvious methods to achieve the exact same thing in a kosher way, some of which were mentioned in the thread.

    But what can ya do? :)

  2. Danny says:

    Gotta love that Hugo guy:

    "This is called thinking out of the box incidentally, for those of an overly

    conservative mindset."

    Out of that annoyingly constraining box of writing robust software, it seems.

  3. Joshua says:

    The pointed thing is the guy's claim that no large system can be developed depending on only the written documentation.

    Trouble is, he's right.

    Even on UNIX, there's the POSIX definitions and the oral tradition. Microsoft implemented POSIX in a way that ignored the oral tradition to a point that few UNIX programs could run in the environment. They were rightly flamed for it, and now we have Cygwin which comes much closer to what it should have been. Cygwin uses crazy undocumented hacks because the NT kernel cannot support the oral traditions.

    More on the point is his claim is for all operating systems, including embedded realtime operating systems. It has bitten UNIX itself. time_t is a signed number, hence the 2038 bug.

    Microsoft is stuck with this kind of behavior. They are not alone.

  4. Why do you have to wait for your program to complete?  Just have it print out the "highest PID so far" every time it changes and let it run indefinitely.

  5. Ben Voigt [Visual C++ MVP] says:

    @Maurits: Possibly because the I/O is going to slow down the progress of the program by several orders of magnitude.  However, you could check for a new "highest PID" on every spawn, but only set a dirty flag and only actually print out anything every 1000 spawns or so.

  6. jcs says:

    If you read that Hugo guy's comments more closely, you'll see that he is likely writing code for the financial industry.

    Now does it surprise you that he lampoons the "overly conservative mindset" of those who write robust software?

  7. Crescens2k says:

    Well, when I think about this kind of discussion I always remember the Pushing the Limits of Windows on Mark Russinovich's blog. For anyone curious about what these limits are on Windows, I think this series is a really good read. You can find the first one here (…/3092070.aspx) and the rest are linked in that entry.

  8. James Schend says:

    Joshua: Not to start a flamewar, but if Microsoft wrote their POSIX code to the spec, and POSIX programs couldn't run because of some unpublished "oral traditions"… how is that Microsoft's fault? It seems to me that the authors of the POSIX standards, or possibly the writers of that software, are at fault.

    This is not an isolated case; the same thing happens to the IE team all the time while implementing W3C standards. Other browsers (being open source) just "borrow" each others' implementations, but the IE team writes it from scratch– then the IE team is blamed when incompatibilities inevitably arise. The W3C is seen as the perfect voice of reason, even though their vague standards were the source of the confusion in the first place.

    I guess the lesson here is "life just isn't fair." Heh.

  9. Skyborne says:

    "Just for fun, I tried to see how high I could get my process ID."

    It was difficult not to burst out in evil laughter at this point.  I should try this on Linux sometime.  (A machine at hand happened to show ps at 32755, so I ran it a few times until it rolled over from 32767 to 300.  But I don't know if that represents a limit or a convenient point to start recycling if available.)

  10. From my observations process ids and threads share the same namespace (or number space). That's why the first process id is in the hundreds. Most of the space is taken by system threads.

  11. Klimax says:

    I wonder, does cygwin still abuse hacks or did they fixed them? (in 2005 they hacked their way for del of current directory)

    Anyway if you use undocumented features/API/hacks/… then either you overthink it,do horrible things or horribly or don't know any better/was told so on some sites like StackOverflow. Neither should be acceptable.

    One should never ever rely on undocumented things – so far I never saw a reason why to. (Maybe one should read books like Windows System Programming.)

    But then I generally use WxWidgets or DirectX. (and so far no hacks seen in Wx)

  12. Alex says:

    Great article except for the counterexample "… and then you have a pretty big gap before the next process ID (for me, it's 372). And yet Calc was given a process ID in the 2000's. Proof by counterexample that the system does not assign PIDs smallest-available-first."

    Not a counterexample, as there can be an easy explanation for that gap. Take my Linux machine for example. After boot, a few background daemons have low PIDs like 200 etc., however, the first programs I start get PIDs in the 1000s or 2000s as well. Yet process IDs do get assigned sequentially. So this counterexample isn't one. You should have started two calcs right after another and compared their PIDs as a counterexample, so that the time period between both start was so low that it is unlikely that the "missing" PIDs in between just got used by other programs, which have already exited.

    [The fact that Calc was not given a process ID below 372, even though we knew that those IDs were available, shows that process IDs are not assigned lowest-available-first. -Raymond]
  13. Crescens2k says:


    That of course is assuming Windows works like linux, which is a bad assumption.

    But anyway, if you want that kind of proof. Try starting two instances of calc in quick succession (I did it on the command line using the up key, but I'm sure you can get the second instance even faster by using a script), it gives two pretty different PIDs. For me it was 2016 and 4292.

  14. Justin Olbrantz (Quantam) says:

    A while back <a href="…/audio-driver-incident.html">I noted</a> via a buggy app that Windows had no problem with a process having 4 million registry handles, in case anybody cares. I actually had no idea that Windows supported that many simultaneous handles.

  15. Crescens2k says:


    Windows has an upper limit of over 16.5 million handles per process (exact figure is 16,777,216, the source being the Pushing the Limits of Windows article I posted earlier).

  16. Joshua says:

    UNIX oral tradition cases in point:

    1) The shell is /bin/sh. NT4 POSIX didn't have /bin. Because #! (which is in the standard), moving /bin was ill-advised.

    2) Deleting your own binary works. While POSIX allowed -ETXTBUSY, it wasn't supposed to be issued unless the underlying filesystem didn't support hardlinks.

    3) Deleting a file with an open handle must succeed on the primary filesystem and allow another file to be created with the same name. This broke thousands of applications.

    As far as I know, Cygwin got their hacks down to 4.

    1) NtCreateXxxxx security api (name omitted so Raymond doesn't complain) is used by ssh. This is kind-of avoidable now but the documented method has so many caveats that the old undocumented method is favored (see oral tradition case 2).

    2) fork() does crazy things to address space. Not a problem unless somebody else is using AppInitDlls. Don't dll-inject Cygwin processes either. The hack's only particular dependency is the system dlls are loaded at the same address space for all process of the same bitness. With the Vista documentation, this fact is finally documented in the ABI.

    3) The various methods of unlocking current directory (oral tradition case 3)

    4) Delete file in use uses a similar hack to NFS.

    Actually the problem of a malicious implementation was considered by the POSIX writers and dubbed weirdnix. The spec authors were horrified when NT produced a functioning weirdnix.

  17. Alex says:

    @Raymond: as always, you are right :) Missed that nuance.

    @Crescens2k: No I did not assume that anywhere, I just compared the two. I simply totally stupidly missed the fact that if numbers 1, 4 and 372 are taken, then 2 would be the lowest available number (unless available is defined as "not recently used").

  18. GL1zdA says:

    Actually the multiple of 4 is specific to the Windows version for Intel – on both Alpha and PowerPC (have no MIPS/Itanium to test it) the NT 4.0 Task Manager shows odd PID numbers, so they definitely are not a multiple of 4.

  19. Klimax says:

    Joshua: 3) Why they don't switch current directoy to known existing one like "temp/sub" where sub is created by cygwin at the beginning with random name? Is there some problem?

    4)This one is though one. (currently user of unlocker  mainly to see who is still holding that bloddy lock)

    Malicious implementation? Weirdnix? What is that?

  20. James Schend says:

    Joshua: If they anticipated that their spec could be interpreted "incorrectly" (to make 'weirdnix'), why didn't they fix the spec? …and you're still blaming Microsoft? My brain boggles.

  21. Crescens2k says:


    The problem is that Win32 works differently from posix in a few cases and these sticking points are in major implementation differences. So does Win32 really need to be fixed up when it is working in the way it was intended?

    Cygwin is basically an emulation layer on top of Win32, so it should be expected to work in the bounds of the operating system it is running upon. The reverse would hold true too, would you think a function would be added to the *nix kernel just to help Wine run better?

  22. ErikF says:

    @dancol: There are several reasons why Windows separated out its POSIX and OS/2 support into subsystems; this is one of those reasons. Pick up any of the "Inside Windows NT" editions (section "Providing Multiple Environments" in the first edition by Helen Custer [Microsoft Press, 1993]; I don't have the other editions on hand) and the rationale is pretty clear.

    I appreciate the work that the designers of the kernel did to anticipate changes to hardware architecture in the 1990's (for example, until reading the above book, I didn't realize that the VM was architectured to support 64-bit addresses from the beginning!) As long as I follow the fairly simple rule of treating opaque values as just that — opaque –, I should never have to worry about strange assumptions that will bite me later.

  23. Ivan says:

    Danny: Better still, that comment comes at the end of a message which can be summarised as, "I stress-tested my application and found that it failed, but that doesn't count because it was deliberate stress testing."  *Flamboyantly* un-conservative!

  24. @Crescens2k, functionality is added all the time. We didn't always have condition variables either.

    @erikf, I've read the latest edition of Windows Internals. It's an excellent survey of the operation of the NT kernel, but it doesn't address a lot of the historical justification of that design. While subsystem separation is elegant in theory, it doesn't address many real-world concerns because subsystems are inherently isolated, and additionally because non-win32 subsystems are often neglected (see the sad case of poll(2) support in psxss). Frankly, as a practical matter, the SKU restrictions for POSIX limit adoption as well. Adding a few capabilities to the win32 subsystem doesn't impair its current users and makes a major class of user scenarios work much better, and the idea shouldn't rejected out of hand.

  25. Yuhong Bao says:

    "They were rightly flamed for it, and now we have Cygwin which comes much closer to what it should have been"

    Don't forget Interix (now an optional feature in Vista/7 Enterprise/Ultimate) which extends the POSIX subsystem to provide a better Unix environment.

  26. Engywuck says:

    Crescens2k: There's an even faster way: just use && operator in between :)

    calc && calc && calc && calc && calc

    gives me the PIDs 2272, 4276, 6092, 6100, 6108

    so some sort of "re-use" seems to happen.

    When I added another ten instances of calc (without closing the previous 5) it gave me 4520, 5020, 5152, 5304, 5308, 5412, 5416, 5548, 5696 (all in between the "original 5") so the difference of "8" between the last three just happens to be "by accident"

    Or, worded differently: don't make assumptions about the PID you'll be getting when starting another instance.

    In linux I've seen often enough that PIDs tend to be "increase only" until some sort of wraparound (don't know if the wrap is variable), here it seems to be different. Both subject to change, though.

  27. Worf says:

    IIRC. in ancient versions of Linux, there was a max PID value of 65536. ISTR that this really wasn't an issue since a practical system could host only 10,000 processes or so. These days, I don't know what the limit is. I'm guessing it's a 32-bit int now, but wraps under 65536 when it can to avoid large PIDs.

    Or maybe legacy keeps it at 65536 processes (threads share the same PID).

    Easy way is to disable resource limits and forkbomb the machine. The reason things fail when I did this 10+ years ago was "No more PIDs". Of course, Linux back then (don't know now) struggled when the loadavg hit the low 10s. Not sure if this is still the case (I've heard FreeBSD could handle the better part of a 100 loadavg without skipping audio and still being somewhat responsive).

    Wish Windows had a loadavg metric.

  28. Danny Moules says:

    @James Schend: That's simply not true. Their interpretations of the specification weren't 'creatively distinct'. They were wrong. Wrong in so many places, so much of the time. Just plain didn't work in huge places because somebody didn't read the specification or understand it even when there was no ambiguity to be had. It was ineptitude and it's hard to defend.

  29. Danny Moules says:

    @James Schend: That's simply not true. Their interpretations of the specification weren't 'creatively distinct'. They were wrong. Wrong in so many places, so much of the time. Just plain didn't work in huge places because somebody didn't read the specification or understand it even when there was no ambiguity to be had.

  30. Klimax, that was one option considered; actually, you don't even need a random directory: you can use pipefs instead. There's a long thread from mid-2010 on cygwin-devel about the current directory hack; you can go and read it yourself. There's no easy solution; if the win32 API contained a SetCurrentDirectoryByHandle function, this hack wouldn't be necessary at all. *sigh* Ten lines of code in kernel32 could save a ton of pain for a popular, important user of the API.

    A supported win32 fork would save a lot more pain, but it'd also involve more work. (Not all *that* much more work though: the kernel already supports fork. It's just win32 that'd need to be fixed up.)

  31. Neil says:

    XP Interix doesn't let you delete your own binary. This bit me when I tried to do the equivalent of

    /bin/ln /bin/rm /tmp

    /bin/rm /tmp/rm

  32. Crescens2k says:


    The fact that there was reuse wasn't debated. It was about whether the lowest free PID was reused first. As you can see for yourself it regulary grabs from the 2000-5000 range, but it doesn't fill in the gaps in what it grabs, and the lowest PID isn't the lowest available. Occasionally it gets a PID < 2000, very rarely it gets a PID < 1000 and this is done on a system where besides the calc processes being created there is nothing else going on.

  33. Cesar says:

    @Skyborne, @Worf:

    $ cat /proc/sys/kernel/pid_max


    Yes, the default limit is 32768 on Linux. Yes, it can be increased (just echo the new value to that file, or use the sysctl command, or edit /etc/systcl.conf which is read by the sysctl command on boot).

  34. Random832 says:

    @ Joshua "The spec authors were horrified when NT produced a functioning weirdnix." Why weren't those things [and whatever else NT supposedly got wrong] in the standard? It seems like the entire point of POSIX is to create a subset [the real unix stuff is in XSI/UP/etc] of unix stuff that can be implemented on a substantially non-unix-like OS.

    As for /bin/sh, there are plenty of unix OSes that don't have their posix-conforming shell [and other tools] in /bin – /usr/xpg4 anyone? Systems are supposed to prescribe a $PATH for conforming scripts to use, and you're never supposed to depend on the absolute path of something.

    Re #! see…/bwg2000-004.html [no version I could find actually mandates support for this, though the 'sh' documentation has a section 'on systems that support executable scripts' and recommends that the paths be rewritten by installers]

    (Re (2) the [ETXTBSY] case was actually explicitly allowed on systems that support hardlinks, if there aren't any actual hardlinks to that file. Don't know what NT's behavior is here, but [and re (3)] see also "[EBUSY] The file named by the path argument cannot be unlinked because it is being used by the system or another process and the implementation considers this an error." – rename and rmdir also allow this)

    I don't think that doing something different that the standard has text devoted to explicitly allowing to be different really counts as a 'weirdnix' type situation – it's meant for unintentional oversights. "The spirit of POSIX" is not "your entire system has to work exactly like Bell Labs UNIX circa 1979 down to every detail". Do you have any other/clearer examples for the NT POSIX stuff? (Citation also needed on anyone involved in creating POSIX having reported being horrified by the NT implementation, for that matter)

  35. Dave Sawyer says:

    Raymond, thanks for making my blood boil with your *this guy* link.

    Hugo's blustering over why he needs to wedge PIDs into a 16 bit array (2 dimensions dontcha know 'cause a square takes less space than a line) instead of using a map or other trivial working solution is just too familiar.  The pleading of others to write good code are to no avail.  Maybe he can say "my code is frequently featured in articles on leading websites… such as

Comments are closed.