I hope you weren’t using those undocumented critical section fields


I hope you weren't using those undocumented critical section fields, because in Windows Server 2003 Service Pack 1, they've changed.

Mike Dodd tells an interesting story of a vendor who used reserved fields and then complained when the system started using them!

Comments (25)
  1. asdf says:

    That response is classic. Maybe you can convince the winapi team to start using unnamed bit-fields instead of dummy variables to make it inconvienient to modify without editing the header or via a memcpy:

    struct _RTL_CRITICAL_SECTION_DEBUG {

    WORD Type;

    WORD CreatorBackTraceIndex;

    RTL_CRITICAL_SECTION *CriticalSection;

    LIST_ENTRY ProcessLocksList;

    DWORD EntryCount;

    DWORD ContentionCount;

    DWORD : sizeof(DWORD);

    DWORD : sizeof(DWORD);

    };

  2. hi. says:

    Why does the service pack change the structure?

  3. Alan says:

    I use them, but only when I’m debugging a deadlock. If you have a thread blocking on a critical section, then ESI (IIRC) points to the critical section data block, and the third DWORD (again IIRC) tells you the ID of the thread owning that critical section.

    Then you look at why the owning thread is blocked. If it is also waiting on a critical section, repeat the process. Once you find a loop, you’ve got your deadlock.

    So I hope the new structure still has the ID of the owning thread in it somewhere obvious.

  4. asdf says:

    Wait, nevermind about that one. Unnamed bit-fields would be a bad idea for ABI reasons to protect against the case of the user that compiled against an unnamed bit-field header and made a copy of the struct since it’s not guaranteed to copy padding.

  5. Spyware Hater says:

    I’m sure Gator would love to know the gory details of your undocumented APIs when you buy them. That way you can serve even more obnoxious ads.

  6. James Risto says:

    This raises an interesting question. I read blogs that talk about when something is "wrong", but maintained for compatibility. How is this decided? The app’s popularity? For this thread’s example, what if it broke a big-name app?

  7. Pavel Lebedinsky says:

    Owning thread ID and things like lock count are still there in the new structure, they’re just encoded in a different way. The recommended way to dump critical sections is to use WinDbg commands like !cs and !critsec. You can also look directly at the fields, but correctly interpreting them is now a bit more difficult than it used to be. If you want to know the details, install the latest debuggers from http://www.microsoft.com/whdc/devtools/debugging/default.mspx and check out the "Displaying a Critical Section" topic in debuggers.chm.

    The reason for this change was to improve performance. The old implementation was mostly fair (FIFO) which lead to lock convoys. The new one isn’t strictly fair anymore but should perform better when there’s a lot of contention.

  8. Mike says:

    "The new one isn’t strictly fair anymore but should perform better when there’s a lot of contention."

    Should?! Haven’t you performed simulations and real-world testing on everything from 486 (if the OS can run on it, I don’t know) to 32-way or larger systems running both IA32 and AMD64 before such a change?

    I hope you were just being overly humble.

  9. Pavel Lebedinsky says:

    Also, a high contention count does not necessarily mean a convoy is formed. So some applications might get better performance, while others might not. (Well-designed multithreaded apps generally don’t have a lot of contention on their locks, so they will probably see no difference).

    As for adding APIs to access internals of the CRITICAL_SECTION structure, I personally think that the benefits of doing this are very dubious. The usual arguments against exposing implementation details apply (people would start using the functions inappropriately, even if they are clearly documented as debugging APIs, changing the implementation in future versions would become more difficult, etc).

    The number of people who write their own debuggers is relatively small, and they already have to deal with a lot of version-dependent, semi-documented structures (TEB, heap or loader structures, etc). Having to add some platform specific code for working with critical section internals does not seem too bad in contrast.

  10. Pavel Lebedinsky says:

    This seems to be an interesting break from

    > the past, where we have read numerous

    > stories about how Microsoft has bent over

    > backwards to keep compatibility with

    > software using even more undocumented fields

    > in way more undocumented parts of Windows.

    There’s a similar story here, by the way. If you read the windbg docs I mentioned above, you’ll notice that in the SP1 implementation, the LockCount bits representing the number of waiters are stored inverted. This was done to minimize the risk of breaking apps that depend on the fact that an unlocked critical section has a LockCount of -1.

  11. Mike says:

    This seems to be an interesting break from the past, where we have read numerous stories about how Microsoft has bent over backwards to keep compatibility with software using even more undocumented fields in way more undocumented parts of Windows.

    Though, what Pavel wrote triggered something for me:

    Had MS really (as in *really*) considered developers, they would have created a few API’s to query e.g. "Current owner thread ID" and "Recursion count", no? While technically part of the Win32 API, those functions would have obviously resided in a highly Windows-version specific DLL used only for debugging purposes.

    Suggesting one should use a Windows-version + service-pack specific Microsoft debugger just to get this information, perhaps especially if YOU are writing a debugger, is just absurd.

    Raymond, do you know if they now have added accessor functions (I fear not, but I must ask)?

  12. oldnewthing says:

    Obviously changes to something as important as critical sections went through rigorous testing, but you can’t rule out the possibility that a program somewhere uses critical sections in a highly unusual and pathological way that could be negatively affected. (Maybe it *relies on* convoying for some insane reason.)

    It amazes me how people tend to focus on one throwaway word.

  13. Mike Kelly says:

    > Had MS really (as in *really*) considered developers, they would have…

    Kind of a funny comment, given that most of Microsoft IS composed of developers who write software for Windows, not Windows API developers. What do you think the SQL Server, Visual Studio, Office, and Exchange teams are but developers of Windows software?

    So obviously Microsoft thinks A LOT about Windows developers. Perhaps we could attribute less to bad motives than to the inevitable tradeoffs in developing software.

  14. Malinda says:

    Hello,

    Your website is so informative and I am hoping that you can help with my situation. I have an old Compaq Armada laptop and it will not recognize my touchpad. When I boot up it comes up with a message that there is no mouse being detected. Any suggestions? Thanks for your time. Malinda

  15. Jason says:

    It amazes me how people tend to focus on one throwaway word.

    Then don’t use the throwaway word. You’re dealing with programmers here — people who are guaranteed to take what you say at face-value. When you say, "this may work better" you shouldn’t be surprised when people respond with, "what the *&^$ do you mean ‘may’?!"

    This isn’t a newspaper column; you don’t need to conserve words. If the change will work better for every program that wasn’t written "in a highly unusual and pathological way" then say so in the entry. It’ll save a lot of frustration in the long run.

  16. oldnewthing says:

    "If the change will work better for every program that wasn’t written "in a highly unusual and pathological way" then say so in the entry."

    Yeesh, this is a blog not formal documentation or a specification or a contract. Can’t I just write casually? Do I have to pore over every word for hours considering all possible interpretations and defending against them?

  17. Tim Smith says:

    Come on people, lets just make working on these blogs a little bit more fun for these people. That is a great way of making sure the stop bloging.

    I for one would love to know about the internals of the CS change and how it helps performance. (off to google land).

  18. SuperKoko says:

    Why does the winnt.h header contain a structure with fields whose name is explicit?

    Why not naming them dwReserved1, dwReserved2, dwReserved3, etc?

    So, developers should clearly know that using these fields is highly non-portable.

    And also, why not specifying in MSDN library that such thing or such thing is unspecified and must not be used?

  19. memet says:

    Come on people, lets just make working on these blogs a little bit more fun for these people. That is a great way of making sure the stop bloging.

    In response to Tim Smith, I’d like to add: yeah Raymond, don’t forget there’s the silent masses reading your stuff avidly without contributing negative comments. =)

  20. Dean Harding says:

    SuperKoko: You mean like this:

    "A critical section object cannot be moved or copied. The process must also not modify the object, but must treat it as logically opaque. Use only the critical section functions to manage critical section objects."

    From the documentation for InitializeCriticalSection (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/initializecriticalsection.asp)

  21. Jason says:

    Raymond, you can write whatever you want however you want. But if you write something confusing, don’t get testy when people get confused.

  22. "This isn’t a newspaper column; you don’t need to conserve words. If the change will work better for every program that wasn’t written "in a highly unusual and pathological way" then say so in the entry. It’ll save a lot of frustration in the long run. "

    Blogs are much more related to newspaper columns then specifications, happily. You should go to http://msdn.microsoft.com/library and read the specs there. If you’re unhappy with something then click the "What did you think of this topic?" link, or send an e-mail to sdkfdbk@microsoft.com. And read the disclaimer at the right-hand side of this page too ;)

    Raymond, like ‘memet’ wrote there are a lot of people who appreciate your writing here, and it would be a very sad day if you stopped writing because of people like Jason…

  23. oldnewthing says:

    > if you write something confusing, don’t get testy when people get confused.

    What was so confusing about "The new one isn’t strictly fair anymore but should perform better when there’s a lot of contention"?

    > This isn’t a newspaper column; you don’t need to conserve words.

    I’m not actively "conserving words" but I’m also not going to spend two hours on every sentence saying "Gosh, I wonder how people might misinterpret this, maybe I should go into mind-numbing detail just in case somebody decides to take the worst possible interpretation". Which seems to be what happens. I’m trying to adopt a conversational style, not a nitpicky pedantic one.

  24. Dean Harding says:

    But if you write something confusing, don’t get testy when people get confused.

    Wait a minute, wait a minute. Raymond didn’t even write the sentence people are getting upset about!! Now I don’t know if Pavel actually works for Microsoft or not, but whether he does or not, why would people expect him to list out every possible configuration that gets improved performance and every possible combination which does not?

    To me, saying "should" in that sentence implies that most of the time you do, but sometimes you don’t. It doesn’t imply that the developers don’t know, just that there’s not enough space in a comment to a blog to list all combinations of configurations out!

  25. Jason says:

    Apologies for the brevity; my UPS decided that mid-reply would be a good time to flake out, and I’m not typing that all again.

    Yes, Raymond didn’t even write the sentence that caused the problem. I did a poor job of acknowledging that the first time and by the time I came back I’d forgotten who wrote what.

    I inadvertently changed "should" to "may" in my original reply. In my common usage the two are basically the same: "this change should work [but I haven’t tested it yet]." Mike seemed to read it the same way.

    Raymond, I understand not wanting to write your blog like a spec sheet, but as has been pointed out what you wrote wan’t even the problem. I assume from the guy’s knowledge that he’s just another MS employee, so my request (however terse it was, it was… kinda-sorta a request) to be more careful with wording applys more to him.

    As my argument has basically been disassembled and bitch-slapped up and down the block, I must now bow out. I have a word find to do in the other entry that’s more my speed…

Comments are closed.