Concurrency, part 7 – Why would you ever want to use concurrency in your application?

So I've spent a great deal of time talking about concurrency issues, but one thing I've avoided mentioning until now is when do you worry about concurrency.

The first (and most common) time that concurrency matters occurs when your code lives in a DLL.  If your code is in a DLL, then you've got to worry about concurrency (unless you have some external contract that guarantees your code is only called on a single thread (like being a COM apartment model object)).  This is an inescapable rule, if you're in a DLL, you have no control over how your code is called, and you must assume a worst case scenario.  And there are times in a DLL when it's necessary to do work on separate threads - as I mentioned before, you can't use COM objects in a DLL, which means that all COM calls in a DLL have to be done on another thread (unless your code is living in a COM DLL, in which case you can safely assume that your caller's called CoInitialize for you).

But if you're writing an application, you still might want to use multiple threads.  The biggest reason is for convenience.  There are times when it's just useful to be able to kick off a chunk of work onto another thread.  This is especially true for applications that interact with the user.  Even though those applications spend 99% of their time idle, it's critically important that the application be immediately responsive to the user.  So any tasks that could conceivably take a long time (like opening a file) should be performed on a thread that's not interacting with the user.  That's why my copy of FrontPage has 7 threads running.  In fact, on my machine, the only application with only one thread is cmd.exe - all the other processes have at least two threads.  Outlook has 37 threads on my machine right now :).

And, of course, the third reason for writing for concurrency is for performance.  If your machine has only one CPU core, then adding multiple threads won't actually improve your performance (actually they'll hurt your performance due to the time spent waiting for the system to switch from one thread to another), but on a machine with more than one processor, if you have multiple CPU-bound operations that can be overlapped, then running them on separate threads can dramatically improve your scalability.

But if you take the latter tack (adding multiple threads to resolve performance bottlenecks), it's critical to realize that you're potentially walking into a minefield.  All the stuff I've talked about so far is pretty straightforward, and applies to all sorts of applications.  But when you start trying to adopt your code for high performance computing, it opens up a whole new world of potential bottlenecks and issues.

And that's what I'll spend some time talking about next - a rough primer on concurrency for scalability, and some of the issues associated with it.

Comments (14)

  1. Anonymous says:

    Multiple threads on a single CPU system can greatly improve performance if the tasks can be split up and the threads are not CPU bound and making them multithreaded improves CPU usage. This assumes that adding more threads won’t increase the stress on other subsystems that are already swamped. (i.e. adding more disk IO threads in a program that is already disk IO bound.)

    Oh course you can get into other problems such as training where one thread has to wait on another thread thus in effect causing all threads to be serialized in such a way as to only have one thread ready/current at any one time.

    Concurrency is a bear.

  2. Anonymous says:

    To me there are more reasons than performance that add to the wish to thread stuff.

    Threading makes it simpler to follow the code many times, since it becomes more top down. Given that your application has many dijunct asynchrounous code paths. Doing the same work in a single thread application tends to force you into a FSM kind of application, where you tend to lose type checking and it also makes the code much more difficult to follow.

    So even if you application only runs "one"(think FSM, one path would be ending as you asynchrounously wait for the next event) thread at a time it might be beneficial to go down the threaded path, as the code becomes top down and failure handling becomes trivial. The API between the "states" could be queues, which then will be easy to flow control properly

    The beauty of threads is that you get another persistant state during execution, that is easily maintained and understood.

    I hope that made some sense

  3. Anonymous says:

    Absolutely true Niclas. Your example (creating a new thread to do work) simply uses principle #1. As another example, it’s relatively hard (not impossible, but relatively hard) to do file I/O asynchronously (even though it’s usually faster) because of all the state you need to manage. But if you dispatch to a worker thread, you can keep your state on the stack without having to worry about concurrency issues – you ensure that only one thread can access the data because its on the stack.

  4. Anonymous says:

    Concurrency and scalable? From Microsoft??

    When is Windows ever used for clusters, except possibly deep in the bowels on Microsoft?

    For a start, anyone hearing the words ‘Microsoft’ and ‘High Performance Computing’ in close proximity better have good health insurance, given the severe abdominal pains that would ensue from having laughed so hard.

    Second, supercomputing requires tinkering at the lowest level (compilers, kernel…). Windows isn’t really suited to this, can’t think why? I somewhat doubt that the people running superclusters are content with the rather meagre scripting abilities of Win32! (And only God or billg could contemplate the license fees…)

    Scalable architectures run *nix for a reason. In fact, rumor has it that Windows builds thenselves are compiled on BSD.

  5. Anonymous says:


    Microsoft cares HIGHLY about concurrency and scalability. That’s why we’ve got to such lengths to get the AD to scale evenly to (I believe) 12+ CPUs (making apps scale to more than somewhere around 3 CPUs is HARD, and gets exponentially harder the more CPUs you want to support).

    Clusters aren’t about concurrency and scalability, not really. They’re about load balancing and fault tolerance.

    Clusters can also be done to host distributed computing projects, but that’s a different type of problem. For Microsoft’s clustering solution, see: – I know some of the people working on the project, and they absolutely know what they’re doing.

    And the rumour that Windows builds are compiled on BSD is a complete lie. Windows (and MS-DOS) builds have NEVER been on BSD.

    Back in 1984, MS-DOS builds were done on Sun workstations running Xenix, and Microsoft’s languages were done on a Decsystem-20 running TOPS-20, but since 1987 or so, all Microsoft development’s been done on a PC based platform (OS/2 originally, then Windows NT).

  6. Anonymous says:

    A typical use of threads by me is communications (RS232, TCP/IP, etc.) The communications take place in a worker thread and the GUI thread handles the display.

    I seem to remember OS/2 having ONE message queue for the whole machine. It was stressed to never, never, block that thread with any IO. So we did put everthing into another thread: File IO, communcations, etc. It was rather combersome to do.

    With windows, everbody seemd to be doing it, so quick file IO takes place in the main thread. We put up the hourglass in case it takes "awhile".

    Another issue is threads and windows controls, expecially with MFC.

    Funny you should mention Outlook – its one of my peeves lately. It blocks the GUI when it can’t communicate with my exchange server. I use a VPN and if the VPN goes down, Outlook pops up the "Retrieving data.." box and the rest of the program blocks. Maybe its a modality issue. Mine has 18 threads running right now.

  7. Anonymous says:

    The "outlook locks up with exchange" problem’s been fixed since Outlook XP, and it should be impossible to see it with Outlook 2003.

    And yup, OS/2 had one message queue, it was one of the huge improvements put in for Win32.

    I’m surprised you can do serial I/O synchronously – I thought you couldn’t get above about 9600 baud with synchronous I/O.

    The Threads and window controls issue is directly related to the per-thread window issue, btw.

  8. Anonymous says:

    It’s good of you to point out that even a simple "File > Open" ought to take place in another thread, since it might involve sluggish networking, name resolution, etc.

    Speaking as a software developer, the reason why we don’t do things correctly, and treat OpenFile() as if it were a fast operation, is that spawning things off into another thread automatically doubles the complexity of the user interface. We now have to have a UI state that says "Yes, thank you for your request to open a file, one day we’ll get back to you but for the moment, just wait".

    All in all, an hourglass and a temporarily unresponsive application seems a friendlier solution!

  9. Anonymous says:

    Hmmm – I’m about to give a talk on this at the next new england code camp

    As far as reasons go….

    1) All the obvious reasons, i.e. not locking up the ui, not getting slowed down by i/o bound operations, etc

    2) The observation on the production of more readable code by Niclas is cogent –

    3) The observations by Artemis are not so cogent – I’d actually be of a mind to compliment msft on the multithreading in .net, save for the fact that I CAN’T SET FREAKING AFFINITY WHICH IS SOMETIMES A REAL PITA GUYS!!! Other than that, nice job

    4) …… drum roll……

    Larry, you missed the biggest one – Herb Sutters ‘The Free Lunch is Over’

    In a nutshell, the next generations of processors will have multiple cores, not just hyperthreading

    For a single threaded application, this means

    a) if you have a multithreaded competitor, they’re going to stomp you into the ground

    b) if you aren’t multithreaded, you aren’t exercising all the resources the user paid good money for

    c) and worst yet (Herb didn’t mention this)

    Since syncronizing the two cores takes time and effort, your single threaded app running on a new multicore cpu may in fact RUN MORE SLOWLY!!!!

  10. Anonymous says:

    Larry, my Outlook XP regulary freezes its UI for around a minute every single time it connects to the Exchange server. It asks for my credentials right away, then it pops up a timeout window and when I click Retry it freezes for a very long time. It eventually displays my folders and home page and freezes again for about 10 to 20 seconds, simetimes timing out loading my tasks/reminders whatever. After this horrible startup it works fine, with ocassional exchange timeouts when resolving names (which btw causes the message to be saved to my Sent Items without recipients even though it sends the e-mail out fine).

  11. Anonymous says:


    If outlook’s popping up the logon UI, that means that there’s an issue with your domain controllers – the client tried to connect to the server, and the server rejected the logon.

    I was very clear in differentiating between Outlook XP and O2K3 – Outlook XP moved almost all the MAPI processing to another thread, but some of the processing was still on the foreground thread. For O2K3,all the MAPI work’s done on background threads.

  12. Anonymous says:

    O2K3 freezes after it displays the home page/inbox. There’s a few second delay (5 seconds or so) when it’s responding and then it hangs for tens of seconds and then comes back. As I said, my guess is that it’s loading my tasks and meetings, because sometimes it comes back after a very long wait saying that it was unable to load my tasks and/or free/busy data (I don’t remember the exact wording of the error message). Once that is over it works great, with the exception of resolving names (Alt+K), that is still done on the UI thread, or at least it blocks the UI thread.

  13. Anonymous says:

    Mark Mulin:

    Larry pointed to the "Free lunch is over" article in the first article in this serie (, and earlier here

    Not 100% of Win32 is available from the .NET v1.x classes. But, have you checked out System.Diagnostics.ProcessThread.ProcessorAffinity property? A nice article is [1] which shows the mapping from Win32 APIs to the .NET classes.


Skip to main content