Why is my message queue full of WM_TIMER messages?

Dmitry wondered how it's possible for a lot of auto-generated messages to pile up in the message queue. I remarked, "That's a good question, and I didn't provide all the information necessary to answer it. Answering it will take more than two sentences, so I will toss it onto the topic queue."

One of my colleagues wrote to me and said, "Hey, could you bump up the priority of that topic? I happen to have a bug where COM calls are failing because the message queue is full. I wrote some diagnostic code to drain the message queue to see what was in it, and it was full of unprocessed WM_TIMER messages. There were 53 timers running at 16ms each, and the UI thread stopped processing messages for 9 seconds."

Back when I explained how asynchronous input worked, I didn't talk about where auto-generated messages came from.

If a request for a message is about to say, "Nope, no matching messages," the window manager makes one last check: "Is there an auto-generated message that could satisfy this request?" If so, then it generates the message, and hey look, there's a message!

The catch is that auto-generated messages are grouped together. For example, if you ask for any kind of mouse message, and there is an auto-generated WM_MOUSE­MOVE available, then the window manager will generate a WM_MOUSE­MOVE and then check if that matches the filter you provided. The auto-generation is done this way so that message ordering is preserved within a group. You wouldn't want a mouse-up to be generated before the corresponding mouse-down.

The message groups can be see in functions like Get­Queue­Status and the PM_QS_* flags to Peek­Message.

Okay, now we're getting closer to seeing how auto-generated messages can pile up: If you are filtering for a message, and there is an auto-generated message from the same group, but which doesn't match your filter, then the window manager will auto-generate the message, and then go back and re-run the "Find a message" code, which sees the auto-generated message but says, "Nope, I'm not interested in that message."

Another piece of the puzzle is understanding the timer group. There are two messages in the timer group. One is your friend and mine, WM_TIMER. The other is an undocumented internal message known as WM_SYS­TIMER. This is an alternate universe of timers used by the system to manage system things, like the animated concentric circles in the Mouse Sonar feature, deciding when to time out the system tooltips (like the one that appears when you hover over the × button), driving autorepeat when you click on the scroll bar, and blinking the caret in an edit control.

The final piece of the puzzle is the COM modal message loop. This is the message loop used by COM when you call a method on an STA that needs to be marshaled. COM notifies the destination thread that it needs to run some code, and then it enters a modal message loop waiting for the destination thread to reply, "Okay, I'm done. Here's the answer."

The COM message loop is a complicated beast, most likely the result of over twenty years of evolution rather than having been designed that way from the beginning. One of the things that it does is peek WM_SYS­TIMER messages. Another thing that it does is dispatch timer messages, provided you passed the COWAIT_DISPATCH_WINDOW_MESSAGES flag.

Okay, here comes the wild ride.

COM wants to process WM_SYS­TIMER messages, but not WM_TIMER messages. It therefore does a Peek­Message(&msg, nullptr, WM_SYS­TIMER, WM_SYS­TIMER, PM_REMOVE). If there is a WM_SYS­TIMER message due, then the window manager generates the WM_SYS­TIMER message on the fly, puts it in the queue, and the Peek­Message function returns it. That's the good case.

Another good case is that there is neither a WM_SYS­TIMER message nor a WM_TIMER message due. In that case, the window manager generates nothing, and the Peek­Message function returns "Sorry, I didn't find anything."

The bad case is where there is no WM_SYS­TIMER message due, but there is a WM_TIMER message due. In that case, the window manager generates the WM_TIMER message on the fly and puts it in the queue. But the Peek­Message function ignores that message because it's interested only in WM_SYS­TIMER messages.

Result: A WM_TIMER message got generated and dumped into the queue.

Every time a WM_TIMER comes due, another WM_TIMER message gets generated and added to the queue. Eventually, your queue fills up with WM_TIMER messages.

My colleague replied, "Thanks for the explanation. Of course, it's COM, the Bermuda triangle of Win32!"

Comments (18)
  1. pm100 says:

    so whats the solution?

    1. Darran Rowe says:

      Be careful when doing message queue operations with filters, and make sure that your message loop is written in a way to handle this.

      1. pc says:

        But if I’m reading this correctly (and I may not be), the core of COM has a message loop that *doesn’t* handle this. So you’re just stuck if you need to use COM?

        1. Darran Rowe says:

          Yes, but for a GUI application, you should always have a message loop that isn’t filtered for your window anyway.
          The thing that you must remember is that the COM message loop only runs for marshalling, so when the COM object has finished the marshalling of data between compartments then it will exit the loop. So the reason why it isn’t seen as much of a problem is that the main window loop will start handling the messages as soon as it starts running again.
          The issue only becomes a real problem if there are lots of timers triggering at once because the COM serialisation is keeping the message queue busy.

        2. Darran Rowe says:

          Oh, and one solution if you are having problems because of the COM Bermuda Triangle is to use other threads if you can, but this gets quite problematic.

        3. Darran Rowe says:

          OK, after thinking about it a bit and realising my daftness, I’ll try this again. Sometimes I wish I could delete comments.
          There is only one major cause for this, and that is a filtered message loop is running to long.
          But there are actually two ways for a filtered message loop running to long to actually happen. The first is you are only writing filtered message loops in your code. The obvious solution is to make sure that one, preferably the one used to handle the window messages, is completely unfiltered. So that is GetMessage(&msg, nullptr, 0, 0);.
          The second is to keep the unfiltered message loop from running long enough to allow the message queue to saturate. I admit, I kinda forgot about this for a while, hence the daftness. In this case, if you can, try to move the COM related code onto a separate thread. The message loop will then have to run on that other thread and so your main thread. But this can get complicated. But the important thing here is that since you have two threads with two message loops and two message queues, there shouldn’t be any risk of one interfering with the other. You can also communicate back and forth with thread/windows messages too.

          1. Darran Rowe says:

            “The message loop will then have to run on that other thread and so your main thread.”
            Meh, I was going well this time too. I thought about that sentence, decided to split it up, and then forgot to delete a part of it. This should be:
            “The message loop will then have to run on that other thread.”

  2. Trevligt med ett inlägg även på midsommarafton!

  3. Yukkuri says:

    Kiiiiiiiiiiiiiiiiiiiinda sounds like one should think very carefully before filtering at all…

  4. Antonio Rodríguez says:

    Question: What is the maximum number of timers I can use?
    Answer: If you are asking this question, you are using too many!

    Using modal message loops for marshaling may or may not be a good design, but firing 53 timers at a 16 ms. (60 Hz.) frequency is crazy. My psychical powers tell me they are related to some kind of UI update (60 Hz. is often considered the optimal speed for animations), and you can easily use a single timer to trigger 53 animations instead of having to send a single message for every item you need to update.

  5. Waleri says:

    I’ve always wondered, why this message filtering is needed anyway…

  6. gdalsnes says:

    Isn’t it an equally bad case where there is a WM_SYS­TIMER due and a WM_TIMER due? So it doesn’t really matter if there is a WM_SYS­TIMER due or not? But I guess this could all be solved with a new flag to disable message group generation.

  7. cheong00 says:

    Would it be better if we change those 53 timers into Waitable Timer / Timer Queue instead to evade the problem? Is there any drawback?

  8. 640k says:

    Solution: Remove COM from windows.

    1. Darran Rowe says:

      The problem is, a lot of APIs like DirectX would have to be re-implemented, and that would break a lot of things in Windows. Is that really worth the cost here?

    2. Engywuck says:

      if you want to remove COM you must know a way to replace it. But don’t forget such small details like network-transparency, apartments, language-agnosticism, etc..
      Oh, and of course you should provide a way to be backwards-compatible with (nearly) all programs written with or relying on COM in the last 25 years.

      1. Joshua says:

        Better solution: restore the old restricted COM used by Win95 so that you don’t load the whole corrupt edifice for common dialog boxes and don’t use it at all elsewhere.

    3. Nick says:

      I laughed out loud

Comments are closed.

Skip to main content