In the conversion to 64-bit Windows, why were some parameters not upgraded to SIZE_T?


James wonders why many functions kept DWORD for parameter lengths instead of upgrading to SIZE_T or DWORD_PTR.

When updating the interfaces for 64-bit Windows, there were a few guiding principles. Here are two of them.

  • Don't change an interface unless you really need to.
  • Do you really need to?

Changing an interface causes all sorts of problems when porting. For example, if you change the parameters to a COM interface, then you introduce a breaking change in everybody who implements it. Consider this hypothetical interface:

// namedobject.idl
interface INamedObject : IUnknown
{
    HRESULT GetName([out, string, sizeof(cchBuf)] LPWSTR pszBuf,
                    [in] DWORD cchBuf);
};

And here's a hypothetical implementation:

// contoso.cpp
class CContosoBasicNamedObject : public INamedObject
{
    ...
    HRESULT GetName(LPWSTR pszBuf, DWORD cchBuf)
    {
        return StringCchPrintfW(pszBuf, cchBuf, L"Contoso");
    }
    ...
};

Okay, now it's time to 64-bit-ize this puppy. So you do the natural thing: Grow the DWORD parameter to DWORD_PTR. Since DWORD_PTR maps to DWORD on 32-bit systems, this is a backward-compatible change.

// namedobject.idl
interface INamedObject : IUnknown
{
    HRESULT GetName([out, string, sizeof(cchBuf)] LPWSTR pszBuf,
                    [in] DWORD_PTR cchBuf);
};

Then you recompile the entire operating system and find that the compiler complains, "Cannot instantiate abstract class: CContosoBasicNamedObject." Oh, right, that's because the INamed­Object::Get­Name method in the implementation no longer matches the method in the base class, so the method in the base class is not overridden. Fortunately, you have access to the source code for contoso.cpp, and you can apply the appropriate fix:

// contoso.cpp
class CBasicNamedObject : public INamedObject
{
    ...
    HRESULT GetName(LPWSTR pszBuf, DWORD_PTR cchBuf)
    {
        return StringCchPrintfW(pszBuf, cchBuf, L"Basic");
    }
    ...
};

Yay, everything works again. A breaking change led to a compiler error, which led you to the fix. The only consequence (so far) is that the number of "things in code being ported from 32-bit Windows to 64-bit Windows needs to watch out for" has been incremented by one. Of course, too much of this incrementing, and the list of things becomes so long that developers are going to throw up their hands and say "Porting is too much work, screw it." Don't forget, the number of breaking API changes in the conversion from 16-bit to 32-bit Windows was only 117.

You think you fixed the problem, but you didn't. Because there's another class elsewhere in the Contoso project.

class CSecureNamedObject : public CBasicNamedObject
{
    ...
    HRESULT GetName(LPWSTR pszBuf, DWORD cchBuf)
    {
        if (IsAccessAllowed())
        {
            return CBasicNamedObject::GetName(pszBuf, cchBuf);
        }
        else
        {
            return E_ACCESSDENIED:
        }
    }
}

The compiler did not raise an error on CSecure­Named­Object because that class is not abstract. The INamed­Object::Get­Name method from the INamed­Object interface is implemented by CBasic­Named­Object. All abstract methods have been implemented, so no "instantiating abstract class" error.

On the other hand, the CSecure­Named­Object method wanted to override the base method, but since its parameter list didn't match, it ended up creating a separate method rather than an override. (The override pseudo-keyword not yet having been standardized.) As a result, when somebody calls the INamed­Object::Get­Name method on your CSecure­Named­Object, they don't get the one with the security check, but rather the one from CBasic­Named­Object. Result: Security check bypassed.

These are the worst types of breaking changes: The ones where the compiler doesn't tell you that something is wrong. Your code compiles, it even basically runs, but it doesn't run correctly. Now, sure, the example I gave would have been uncovered in security testing, but I chose that just for drama. Go ahead and substitute something much more subtle. Like say, invalidating the entire desktop when you pass NULL to Invalidate­Rect.

Okay, so let's look back at those principles. Do we really need to change this interface? The only case where expanding to SIZE_T would make a difference is if an object had a name longer than 2 billion characters. Is that a realistic end-user scenario? Not really. Therefore, don't change it.

Remember, you want to make it easier for people to port their program to 64-bit Windows, not harder. The goal is make customers happy, not create the world's most architecturally pure operating system. And customers aren't happy when the operating system can't run their programs (because every time the vendor try to port it, they keep stumbling over random subtle behavior changes that break their program).

Comments (29)
  1. Anonymous says:

    Ah, the old Python adage "practicality beats purity".  If you build the tallest ivory tower of utmost purity, nobody will want to live in it.

  2. xpclient says:

    Thank God the x86 to x64 porting of Windows happened while Jim Allchin was still there at the company! The current team would have made all sorts of compromises like they did for ARM.

  3. Anonymous says:

    My compiler generates a pretty strong warning when I try that setup (hides virtual base member).

    Of course on that particular case, meh it's just not worth it.

  4. Anonymous says:

    @xpclient:  You liar, Windows 8 provides a no-compromise computing experience.  I read it in a marketing release so it must be true.

  5. Anonymous says:

    I agree but it's a slippery slope guys. Look at PHP ;)

  6. Anonymous says:

    This looks like a C++ specific problem.  I didn't actually know that there was a not-yet-standardized "override" pseudo-keyword, but I know where the idea comes from: that's one of the many little details that makes Delphi a better language.

    In Delphi, when a derived class declares a method with the same name as a virtual method in a base class, it has to match the signature and mark it as "override" to make it an override.  Otherwise you get a compiler warning.  If you do mark a method as override and it doesn't match the signature, as in the example here, it's a compiler error. And there's another directive, "reintroduce," that makes it explicit that you don't mean to override the virtual method and gets rid of the compiler warning.  Between "override" and "reintroduce", it's impossible to accidentally make this kind of mistake unnoticed.  (Especially if you also treat warnings as errors.)

    [There are non-C++ examples, too. For example, changing the function signature silently breaks every p/invoke. -Raymond]
  7. One notable old PHD from CMU, an expert in Windows kernel and user mode, has bitterly complained that Win64 didn't expand ReadFile/WriteFile length to 64 bits. Seriously.

  8. GregM says:

    Mason, Visual C++ has similar warnings about differing virtual functions, but I believe that they are disabled by default unless you use the highest level of warnings (which I highly recommend).

  9. Anonymous says:

    Why did you use DWORD_PTR here instead of SIZE_T? I thought that type was used for when you wanted to non-lossy cast between pointers and integers in the DWORD range?

    [To avoid introducing a breaking change to the 32-bit side. -Raymond]
  10. Anonymous says:

    One notable old PHD from CMU, an expert in Windows kernel and user mode, has bitterly complained that Win64 didn't expand ReadFile/WriteFile length to 64 bits. Seriously.

    Wow. Do you really want a >2GB single IO request?

  11. Anonymous says:

    wrt >2GB I/O requests..

    I would like them, yes. The entire point of going 64-bit is to address more memory. When you are designing standards for 64-bit computing, the last thing that you want to do is use 32-bit thinking to limit it. Not to mention the fact that modern consumer grade SSD's push half a gigabyte per second these days, that high end SSD's use proprietary interfaces to push several gigabytes per second over PCIe, that new computers are often now sold with 8= GB of memory…

    What desirable purpose does it serve to require specialized code to read and write a dataset whose size happens to breach an artificial barrier imposed by a 20 year old paradigm? There is no justification for a 31-bit size restriction on a 64-bit architecture, only fallacious rationalizations such as "do you really want a >2GB single IO request?" .. YES, I WOULD LIKE TO INFORM THE OS OF EXACTLY WHAT I AM DOING.

    [It doesn't matter whether you extend the parameter to 64 bits or not, because the maximum size of a single write is 32MB. (Remember that I/O buffers must be locked, and locking 4GB of memory is not very friendly to the memory manager.) -Raymond]
  12. Anonymous says:

    @Mason Wheeler

    Unfortunately, and it breaks my heart to say so, Delphi is not better in this perspective then any other language and that is because Delphi had/forced to recognize the existence of C/C++ languages (hence we got those nice register;cdecl; reserved words) while C/C++ acted just like Windows (Linux? who is Linux? I am the only OS and the given God gift to PC's). So, from this perspective, the entire Delphi implementation of COM / Win32s / interfaces etc was made purely as a port to match binary compilation like a C/C++ compiler would do. And that's why Delphi tastes so bad when you start to need those .h files to have a Delphi implementation of an already C/C++ library.

    Now, back to our example, do this in Delphi, port Ray's example one on one to a Delphi equivalent and then see that Delphi will suffer from the same "kick" like Ray said in the end regarding CSecureNamedObject.

    Perhaps I should feel better because when you deal with interfaces at least you already know what to expect if you decide to switch from C/C++ to Delphi as your main language for living, only to discover later that Delphi is a breeze when it comes to "my boss wants this app done in 2 weeks while I know form my experience it will took at least 2 months". Sorry Ray, but it is the truth, Delphi is so much better then C/C++. If Microsoft would decide to make W9 in Delphi, it will find out that it need only half the time, so maybe sticking with C/C++ to create it is better from economics perspective – you feed more people. Decisions decisions…

  13. Danny, you realize that C# was designed by the creator of Delphi and was influenced a lot of its design, especially in the libraries?

  14. [because the maximum size of a single write is 32MB]

    Strictly speaking, it's not true anymore in Vista+ (2GB-4K) and Win7+ (4GB-4K). Length argument in IoAllocateMdl is still ULONG.

    But this is a game of diminishing returns. Most storage controllers have a limit for a single IO size (usually around 1MB, more or less). A single I/O larger than that has to be chopped to multiple requests anyway. It doesn't make sense to push a whole giant buffer at once.

  15. [A single I/O larger than that has to be chopped to multiple requests anyway.]

    Although this is done by classpnp.sys, transparently for the application.

  16. Anonymous says:

    [… (Remember that I/O buffers must be locked,….) -Raymond]

    I would assume that only the lowest-level device driver needs to lock and unlock pages, and only for the duration of the onging respective I/O requests.

    Is Windows really locking all the pages of a ReadFile() call at once, even if (for example) a USB 1.1 memory stick needs seconds to minutes to read 10 Mbyte of data request by the ReadFile call?

    [If you're going to issue a single I/O to the device, then it had all better be locked! It's hard to DMA into the pagefile. -Raymond]
  17. Anonymous says:

    @Axel: Exactly.  Things have diverged a fair amount since then, but if you look at C# and .NET 1.0, it's essentially "Delphi rewritten to look like Java."

  18. Anonymous says:

    Look, a 2 TB IO request may not make sense now, but in just a few years we'll all be pointing and laughing at the people who never said 640K ought to be enough for anybody, so we'd better start rewriting things right now.

    @Joseph: a non-fallacious rationalization is the simple observation that we're not "designing standards for 64-bit computing". In itself, that's not such a good market to be in anyway (ask Intel about Itanium some time). What we *are* doing is getting 64-bit out there in a way that makes the most pragmatic sense. Sometimes that may err on the side of too much conservatism, but that tends to be less harmful than erring on the other side. Of course, everybody has their favorite examples of some doofus getting it obviously wrong.

    The blanket statement "there is no justification for a 31-bit size restriction on a 64-bit architecture" is faintly ridiculous as a comment on a post where *just that* is demonstrated.

  19. Anonymous says:

    What's going to happen to Raymond's blog?  It's clear from Windows 8 that Microsoft considers desktop applications deprecated, so why bother to have backward compatibility in the long run?

  20. Anonymous says:

    @JamesJohnston – To be fair, Delphi has provided overloaded versions of the conversion functions for many versions (not sure which one was first, before Delphi 2006. Yes it should have been earlier, preferably Delphi 2 when Win32 support was added). The original versions were written before multithreading was common – Delphi 1 was for Windows 3.

  21. Mike Dimmick says:

    @xpclient: Windows RT has a full enough implementation of Win32 and the classic GDI model that the majority of Office 2013 Home & Student can be ported, and that all the same desktop control panels are present. It's just that ordinary developers are not being permitted to even sideload apps, which is a very different thing. I would be *astonished* if Windows RT were not simply a recompile of the common source code, with just a few pieces written in assembly.

    Macros are missing from Office RT because – assuming things are the same as they were when MacBU were trying to migrate from Carbon to Cocoa, see http://www.schwieb.com/…/saying-goodbye-to-visual-basic – the VBA bytecode interpreter is written in tens of thousands of lines of x86 assembly.

    My suspicion is that the only reason that the desktop remains in Windows RT is because the Office team did not want to take a dependency on WinRT while it was still in its formative stages, and similarly that the control panels deemed less important were not prioritised for porting to WinRT. New core features like Storage Spaces probably didn't want to take a dependency on WinRT at this stage either. The Windows division have learned from the Longhorn debacle. Shipping is a feature.

    Certainly Windows RT is intended to only ever be used on tablets – allowing any arbitrary code to be recompiled for it would not meet the design goals of very low power consumption and low heat output.

  22. @Mason Wheeler:  "if you look at C# and .NET 1.0, it's essentially 'Delphi rewritten to look like Java.'"

    I'd clarify that by saying it's essentially Delphi rewritten to look like Java, but with loads of improvements to the runtime libraries.  Having worked with both .NET Framework and the VCL, I've lost count of the number of times in the VCL I've thought "wow, this is like .NET, but worse."

    For example: if you want to convert a string to a date/time from different formats, you have to modify *global variables* that control date/time formatting.  (You wanted to multi-thread your program and use VCL date/time functions?  Ha!!)  The TDateTime constructor that accepts a string parameter won't accept any parameter that specifies formatting information.  The only way I've found is to modify global variables…

    Comedy ensues when you have programs storing dates as strings in files that are shared across international borders and the programmer's didn't think to fuss with global variables to make it culture-neutral.  At least .NET has an overload to make you think about it, and makes it brain-dead easy to use a culture-neutral format.

  23. Anonymous says:

    [It doesn't matter whether you extend the parameter to 64 bits or not, because the maximum size of a single write is 32MB. (Remember that I/O buffers must be locked, and locking 4GB of memory is not very friendly to the memory manager.) -Raymond]

    Then why support sizes larger than 32MB, like 2GB? This is just another rationalization.

    We are talking about a legacy 32-bit API that supported 2GB I/O requests at a time when only 12MB to 16MB of system memory (NT 3.1) was expected. You simply do not get to talk about some irrelevant 32MB restriction as if it has any teeth… because it didnt then, so why would it now?

    The reasoning is "We don't have to change anything this way" .. the rationalization is all this other stuff that makes excuses for it.

    [I don't know where you got the idea that 32-bit Windows supported 2GB I/O requests. Like I said, the maximum was 32MB. -Raymond]
  24. Is Windows really locking all the pages of a ReadFile() call at once

    Yes, this is about the first thing (along with the handle dereference) done by the I/O manager, for non-buffered requests. Because most of the further processing is done outside of the original process context.

    Then why support sizes larger than 32MB, like 2GB?

    This is more likely for special applications, such as RDMA.

  25. dbacher says:

    There is also an additional cost on all basic type changes. If a type changes bits, then you have to repack it from the other subsystem.  If you change a type like SIZE_T, you also break cross task marshalling.  In Delphi and other Pascal derivatives, there is more formalization, but having a structure different sizes still will complicate any IPC, and so these sorts of definition changes are bad there as well.

    From an app architecture point of view, your better off ignoring the CPU, and sizing types based off of intended usage.  Used to be I'd advocate using 32bit values for most everything, but 64 is just too wide.  You waste way too much space.

  26. [I doubt that this lock-the-whole-range-at-once approach is the best way of doing I/O]

    This would be an unnecessary optimization for obscenely large I/O requests, causing unnecessary complication of the storage stack. Otherwise if you lock pages as you go, you can't do asynchronous I/O. Welcome to early single-threaded UNIX.

  27. Anonymous says:

    [If you're going to issue a single I/O to the device, then it had all better be locked! It's hard to DMA into the pagefile. -Raymond]

    Ok, thats how it is implemented today: The low-level drivers will not request the pages as needed to be present in memory to lock them. But I doubt that this lock-the-whole-range-at-once approach is the best way of doing I/O. The discussed examples of starting very large* I/O operations demonstrate that this strategy will needlessly lock pages for absurd long durations.

    *  "Large" in terms of time it will take to complete, which will be many hours for a 2 TB read operation even on a harddisk, let alone slower devices like CD-ROM/Blu-Ray, USB-Sticks or network access.

Comments are closed.