Why does SetThreadPriority sometimes take a really long time?

A customer had a video application that gathers information as it runs. Every so often, they need to take all this information, process it, and save it to a database. They perform this processing every fifteen seconds, but they found that whenever the processing occurred, it created enough of a CPU hiccup that the application would drop frames whenever the updates were being processed.

The customer solved the problem by calling Set­Thread­Priority with the THREAD_MODE_BACKGROUND_BEGIN priority level at the start of the update, and then THREAD_MODE_BACKGROUND_END when the update was complete. This reduces the impact of the update on foreground activity, and it seemed to do a good job of getting rid of the glitches.

And then later, they were investigating an unrelated problem, and as part of their investigation, they noticed that the very first time to enter background mode sometimes took over two seconds to complete. This initial update happened to take place as part of the application's startup, which meant that the long delay was directly affecting the application's startup time.

Further investigation revealed that this extra-long delay occurred only if the application was launched shortly after the machine was rebooted. If the system had been running for a while before the application was run, then the delay was not observed.

What's going on here?

What's going on here is that the system is doing exactly what you asked it to do. You said, "Please reduce the priority of this thread to minimize its impact on the rest of the system." And shortly after a system boots up, there's a lot of activity. Since you said that the thread should have minimal impact on the rest of the system, the thread gets stalled behind all those other activities.

"It's okay. Don't worry about me. I'll be fine."

"Hey, why aren't you paying attention to me?"

The thread said that it wants to run at very low priority, and the main consequence of running at very low priority is that it gets very low priority. The application didn't notice this most of the time because the system usually had enough spare capacity that it could deal even with very low priority things relatively promptly. But once the system gets busy, those very low priority things may end up having to wait a long time.

One the customer understood that this was a problem of their own creation, there were a few possible solutions.

One solution was to suppress the call to lower the thread's priority if the application is still starting up. During startup, update the database at normal priority instead of low priority.

What the customer chose to do was simply to suppress database updates completely when the application is starting up. The application will update the databases in the background after everything else is off and running.

Comments (15)
  1. Mason Wheeler says:

    Filed under: Works As Coded

  2. ZLB says:

    Another interesting thing in this neighborhood is Thread Boosting. SetThreadPriorityBoost().

    When a thread completes a wait for something, its gets a tempory kick to it priority. Usually, this is a good thing. Threads sleep until there is work to do then they get a boost so they can complete their work and go back to sleep promptly.

    However, it is possible that this can interupt other time-critical threads by starving them.

    It sounds like in this case, maybe this could be the initial problem (if the dropped frames lined up with the background task starting) and disabling ThreadBoosting for that thread would stop the hiccup that could cause dropped frames…

  3. Karellen says:

    Strange. Given that the application had unacceptable behaviour (dropping frames) if the foreground activity was interrupted for any reason at all, I wonder why they didn’t raise the priority of the foreground thread (“realtime”?) rather than lowering the priority of the background thread.

    In that case, the background thread would still have had higher priority than the system startup tasks.

    1. ErikF says:

      If the foreground thread ran in realtime, the chances of the background thread running *at all* would be very low: that’s the point of realtime priority!

    2. Darran Rowe says:

      The problem then would be that the process would have higher priority than the startup tasks. In order to solve a small delay in start up, under high CPU load, you starve everything except one process of CPU.
      There is a good reason why things default to normal priority and the advice is to only increase the priority if you really need to.

      1. Karellen says:

        But if the process does not have a higher priority than startup tasks, it will surely drop frames, which – as the customer has pointed out as being the core issue here – is unacceptable.

        Yes, my suggestion to bump it up to “realtime” was going too far (as ErikF also pointed out, thankyou) but there’s room above “normal” priority but below “realtime” which could be explored to get the balance needed. Without dropping the background thread to background priority.

        1. Darran Rowe says:

          The thing is, the easier solution is to just wait until the startup tasks had finished then the problem went away. Then the thread would have higher than normal priority, which is higher than most system components, for the rest of its life. All of this to just get rid of a short period of time after startup where the application would not be able to get as much CPU as it would need.
          Messing with priority is generally not a good thing to do unless you really know what you are doing. It is way to easy to increase priority on a thread and cause problems without realising. Also, due to the fact that this customer’s program can start dropping frames when the CPU is under load shows that it can potentially use a large portion of the CPU. Putting anything that uses a lot of CPU time at a higher priority than Explorer is always a disaster waiting to happen. This is one reason why I really think that Windows should stop user threads from setting their priorities to higher than normal.

        2. ErikF says:

          A possible partial solution that incorporates your idea would be to have the main application set a CPU affinity and then boost its priority on those CPUs; I’m making an assumption that there are cores to spare and I/O isn’t bottlenecking everything. That way the rest of the system can keep going and (hopefully) you get constant performance on the main app.

          Actually, if startup is hitting the system that hard, I’d probably start looking at exactly what services and applications the program is trying to start up. If you’re running a dedicated system, maybe you can get rid of some of the cruft or at least put it on delay-load.

  4. alegr1 says:

    THREAD_MODE_BACKGROUND was a wrong solution for the problem. They should have just set the thread priority to THREAD_PRIORITY_BELOW_NORMAL.
    Then there is a new “lock-free” scheduler, which is perhaps better for a 256 core box, but for a typical consumer 4 processor system.

    1. Zan Lynx' says:

      I think that your comment got cut off somewhere. Yes, in both Windows and Linux the scalable lock-free schedulers are not as good on small numbers of cores. However, the era of small core counts is coming to an end.

      Games using DX12 or Vulkan use up to six cores for rendering and the game engines support threaded job queues for all of the other work. Next generation games for 2017 and 2018 can use eight to ten cores (or hyperthreads).

      Servers can of course use infinite cores.

      There’s really no place for schedulers that are limited to four in today’s environment.

      1. alegr1 says:

        >I think that your comment got cut off somewhere
        That’s Microsoft’s wonderful home-baked discussion engine.
        The new lock-free scheduler doesn’t have ability to requeue a thread to an available processor, if it’s already queued for execution to a busy processor. This is why one or two runaway threads can affect responsiveness of the whole 4 core consumer system.

        1. Klimax says:

          Reminder and correction: Most of MSDN and TechNet blogs are by now migrated to WordPress-based CMS.

  5. Roger says:

    Wow! This sounds like the first time a customer has made the correct decision after receiving information, for once! Not only did they not blindly bump their main processing priority higher (cuz they’re a very special snowflake), but they also just suppressed a inconsequential DB update during app startup.

  6. Killer{R} says:

    With nowadays programmable HPET and high CPUs frequencies it would be possible instead of reducing frequency of time slices granted to lower-priority threads – reduce time slices duration. Some analogy from electronics engeenering world: Replace PFM with PWM. This will increase a bit overhead but should improve responsiveness. So this should be good for UI threads that gets preempted too heavy.

  7. Ray Koopa says:

    “It’s okay. Don’t worry about me. I’ll be fine.”

    “Hey, why aren’t you paying attention to me?”

    Sounds like my girlfriend.

Comments are closed.

Skip to main content