AF_UNIX comes to Windows


Introduction: 

Beginning in Insider Build 17063, you’ll be able to use the unix socket (AF_UNIX) address family on Windows to communicate between Win32 processes. Unix sockets allow inter-process communication (IPC) between processes on the same machine. 

Overview: 

Support for the unix socket has existed both in BSD and Linux for the longest time, but, not on Windows. On Windows, there were some alternatives for local IPC, such as named pipes. But, calling conventions are different between the named pipes and sockets, making writing low-maintenance cross-platform applications difficult. For example, one such place where these two constructs differ (other than the API) is terminating the connection. BSD Socket API provides a bidirectional close semantics using `shutdown`. There is no direct equivalent of that in named pipes. Such differences make it difficult to port unix socket applications from Linux to Windows and vice versa; up until now! 

Build 17063 brings native support for the unix socket to Windows. Starting this build, two Win32 processes can use the AF_UNIX address family over Winsock API (which is very similar to the BSD socket API) to communicate with each other. Currently, the support only exists for the stream (SOCK_STREAM) socket type, which is a connection-oriented protocol for one-to-one communication. Support for the datagram (SOCK_DGRAM) can be considered in future depending on the adoption, feedback and scenarios. 

Addressing scheme - sockaddr_un: 

The ‘sockaddr_un’ structure is used for defining the address of a unix socket. In the Windows implementation of the unix socket, we have kept the name, definition and semantics of the unix socket address the same as that in Linux, to make cross-platform development easier. 

There are three different addressing formats for unix sockets. ‘pathname’, ‘abstract’ and ‘unnamed’ sockets. ‘pathname’ sockets are bound to the filesystem, where in the ‘sun_path’ member of the struct is used to specify a null-terminated UTF-8 file system path. You can expect the same from the Windows implementation, where ‘sun_path’ specifies a Win32 UTF-8 file system path. 

The second category is the ‘abstract’ socket address where the first character in ‘sun_path’ is a null byte. Windows implementation of AF_UNIX socket can also accept abstract addresses. The one difference noteworthy here is that the Windows unix socket implementation currently does not support the autobind feature whereby an abstract address is auto-generated by the implementation on behalf of the user. 

Lastly, ‘unnamed’ sockets, where the socket is bound to a pathname with no name. This is also supported on Windows unix socket implementation. Although, one of the socket API’s socketpair that generates unnamed sockets, itself is not supported in Winsock 2.0. 

Security: 

Unix sockets provide a mechanism for secure communication. Communication over unix sockets can be secured by controlling the file (or directory) permissions on the pathname sockets (or the parent directory). For example, the bind socket API creates a ‘socket’ file with the given pathname. The creation of the new socket file will fail if the calling process does not has write permission on the directory where the file is being created. Similarly, for connecting to a stream socket, the connecting process should have write permission on the socket. The same level of security is available and enforced on the Windows unix socket implementation. See the man page on AF_UNIX for more details on the security. 

Implementation: 

Majority of the functionality for the Windows AF_UNIX is implemented in the Windows kernel in a driver: afunix.sys. The Windows kernel networking stack provides a very pluggable and extensible model, that makes it easy to support new network providers. The socket file itself that is created as part of the bind call is a custom NTFS reparse point. 

Unsupported\unavailable: 

Summarizing from the above, the following Linux unix socket features are either currently unavailable or unsupported in the Windows unix socket implementation. 

  • AF_UNIX datagram (SOCK_DGRAM) or sequence packet (SOCK_SEQPACKET) socket type. 
  • Ancillary data: Linux's unix socket implementation supports passing ancillary data such as passing file descriptors (`SCM_RIGHTS`) or credentials (‘SCM_CREDENTIALS`) over the socket. There is no support for ancillary data in the Windows unix socket implementation. 
  • Autobind feature (see the section on ‘sockaddr_un’ for details). 
  • socketpair: socketpair socket API is not supported in Winsock 2.0. 

How can I write a Windows AF_UNIX app? 

  1. Download the Windows Insiders SDK for the Windows build 17061 -- available here.  
  2. Check whether your Windows build has support for unix socket by running “sc query afunix” from a Windows admin command prompt. 
  3. #include <afunix.h> in your Windows application and write a Windows unix socket winsock application as you would write any other unix socket application, but, using Winsock API’s. 

Note: As mentioned above in the ‘security’ section, when a socket binds a socket to a valid pathname address, a socket file is created within the filesystem. On Linux, the application is expected to unlink (see the notes section in the man page for AF_UNIX) before any other socket can be bound to the same address. The same applies to Windows unix sockets, except that, DeleteFile (or any other file delete API) should be used to delete the socket file prior to calling bind with the same path. 

What’s next? 

We are listening. Please try out the Windows unix socket provider and let us know what works, what doesn’t work, what you like, what you don’t like or what improvements would you like.  For now, the best way to provide feedback is either via Feedback Hub under apps -> Hyper-V, the WSL GitHub issue tracker, or as a comment on this blog.

And, if you are wondering, there is already support for unix socket within Windows Subsystem for Linux (WSL), how does that work with the Windows unix socket implementation? Well, currently, it doesn’t, but stay tuned! 

- Sunil Muthuswamy and the WSL Team

Comments (30)

  1. Murtaza says:

    If only this was available 5-10 years earlier!

    1. Chris Katko says:

      1988, IEEE creates the POSIX standard.
      2018, a mere (some would say blink of an eye!) thirty years later, Microsoft finally implements an industry standard!

      With progress like this, it’s no wonder it took them this long to make cmd.exe adjustable size, add tabs, virtual desktops, and all the other things people on *NIX machines have been using for decades.

      So now we’ll only have to wait about 20-25 years for C# to support WPF on Linux. But let’s be honest. Microsoft only does ANYTHING “cross platform” if it benefits Microsoft. They don’t give a crap about the community, ecosystem, or even Windows user’s experience. All that matters is some manager puts his (unneeded) two cents in to ruin Windows so he can get his bonus for the year. ::cough::Metro interface::cough::

      1. With respect

        1. Windows NT 3.51 shipped with a POSIX 1 compatible subsystem, making NT FIPS 151-2 compliant. The original POSIX subsystem was later replaced with Services For UNIX (SFU) which was built atop Interix (nee OpenNT) from the acquisition of Softway Systems Inc.

        2. Cmd is a shell, like PowerShell or Bash. The Windows Console is the terminal/console that is shared by all Windows command-line apps. We agree that Console has not been enhanced much … which is why we formed a team to own and modernize Console, which has undergone some substantial improvements, both visible and internal, with many more on the way and planned for future releases.

        3. WPF on non-Microsoft platforms is tricky because WPF is very tightly coupled to Windows platform, esp. graphics and input, and because WPF renders Windows UI semantics that would appear “foreign” on other platforms. Instead, we encourage users who want to build x-platform GUI apps to target Xamarin, and UWP on Windows: With a small amount of work, it’s possible to create an app that shares perhaps 80-90% of all code across all platforms, but render and behave “native” on each platform.

        4. Your other views are inaccurate and have no place in comments to this blog.

  2. matt says:

    Will AF_UNIX be available on earlier versions of Windows?

    1. There are no plans at this time to backport to earlier versions of Windows. What versions would be useful to you/what’s the scenario?

      1. abc says:

        There are still a large amount of customers running Windows 7/Server 2008 R2 in China. I think it helps if you make a redistributable package for new APIs if possible so that Chinese developers can quickly get rid of obsolete APIs.

      2. abc says:

        Redistributable for versions from Windows Server 2008 R2 to 2016, Windows 7 and 8.1. There are still a large amount(according to tongji.baidu.com/data/os) of these customers running these old OS’es.

  3. cbhacking says:

    What’s the path format for the sun_path field look like on Win32? Is it just standard Win32 paths (C:\foo\bar.ext)? Is it WSL-like (C:\foo\bar.ext -> /mnt/c/foo/bar.ext)? Does it allow UNC-style paths (\\\\foo\bar.ext), and if so can it be used over SMB (indeed, even if not, could it be used with a mapped network drive)? If an unmodified Unix-style path is provided (/foo/bar.ext), how is that interpreted (perhaps as relative to the root of the drive of the current working directory of the application…)?

    Also, is there any way to get info about the other end of the socket? Windows Named Pipes have very convenient functions for getting the client or server identity, but only *some* Unix-like systems have such an API (and Linux is notably not among them). Finally, how does the performance of this feature compare with other IPC mechanisms, such as a Windows named pipe, a loopback TCP/IP socket, a shared memory segment, ALPC (perhaps via MSRPC or similar), or so on? In other words, if we have highly performance-sensitive code that is being (or has been) ported to Windows, should we stick with (or move back to) Unix/local sockets, or use a more “Windows-y” IPC mechanism as was necessary in prior versions?

    1. Sunil Muthuswamy says:

      @cbhacking – good questions. For the path, it is just standard Win32 path, ex: `C:\foo\bar.txt`. It can also accept UNC paths. As for the performance, we haven’t done a perf analysis yet, but the performance should be comparable to the loopback TCP/IP. As for getting information about the `peer`, unfortunately, that is not available (other than `getpeername`), because the interface is bound to the socket API (`SCM_CREDENTIALS` ancillary message on Linux also gives you some info about the peer, but, that is not yet available as mentioned above in the blog post). What peer information would be useful to you?

  4. Scott Tsai says:

    Great work! I really wish this had been supported sooner 😉

    A few points on the not yet supported features:
    1. I use SOCK_SEQPACKET to avoid doing message framing and boundary parsing
    2. I use socketpair() for IPC between parent-child processes
    3. I use SCM_RIGHTS style FD passing for privilege separation
    For the HANDLE passing use case, some way to pass HANDLEs once an AF_UNIX connection is established would suffice, I wouldn’t need the exact same Linux/POSIX API.

    1. Thanks, Scott for the details on your scenario. `SCM_RIGHTS` is very much in our radar and we understand it’s importance as well. `SOCK_SEQPACKET` and `socketpair` are more involved as it requires changes in the network stack all the way up. But, we are definitely talking to the right set of teams internally here to make progress on all fronts.

    2. Also, if you have any feature requests for AF_UNIX, please add your request to the user voice page https://wpdev.uservoice.com/forums/110705-universal-windows-platform?category_id=22999

  5. K Benson says:

    I know it’s probably going to be awhile but is there an estimate as to when this might reach non-Insiders? I have two machines, one for Insider/other-risky-things(which could be wiped and re-installed) and my main development machine which is backed-up weekly and would still be a disaster to rebuild/reinstall. However, it’s the development machine that really needs the AF_Unix(and the WSL daemons).

    1. Sunil Muthuswamy says:

      It should be part of the next publicly available Windows 10 release, which can be followed publicly. Until then, please try out the scenarios and give us the feedback so that any bugs can be flushed out.

    2. Sunil Muthuswamy says:

      It will be available in the next public release of Windows 10, which can be followed online. Meanwihile, pleaes try out your scenarios and provide feedback so that any issues can be flushed out.

    3. I don’t have any dates to share, but note that:

      Insiders builds are ~weekly builds of the next version of Windows as it’s being built
      We’re aiming to release Windows ~twice a year -typically once in the spring and again in the fall/winter

    4. K Benson says:

      Thank you, Rich and Sunil. So twice a year would mean probably April for the next full update. Sunil, I would like to help, but as I said my development machine is to important for me to possibly mess up with Insider updates. The laptop is more of a game/ Internet exploration machine, which I can wipe and reload if needed. I did do Insider from June to October but the pace (and size) of updates was a little too much for me. I kept having to take my laptop to the local library to update it as my home connection is metered. Anyway, thank you for the information and good luck with the new Intel issue.

  6. WSLUser says:

    Any updates for WSL interop with this? Also, any idea of when SOCK_DGRAM or SOCK_SEQPACKET among others listed as Unsupported/Unavailable will be available?

    1. Thanks for your query. An update on the WSL interop for AF_UNIX can be expected soon. As for `SOCK_DGRAM` and `SOCK_SEQPACKET`, that is not planned yet. If you have any feature requests for AF_UNIX, please add your request to the user voice page along with a description of your scenario:
      https://wpdev.uservoice.com/forums/110705-universal-windows-platform?category_id=22999

  7. Jon Lennox says:

    What’s the preferred way for an application to determine at runtime whether it’s running on a version of Windows with AF_UNIX support?

    1. I would suggest using the the return code from socket(AF_UNIX,…) to determine whether the address family is supported or not. It should return `WSAEAFNOSUPPORT` or something along those lines.

  8. Shadrach Hart says:

    Great work, but could the AF_UNIX Tool be applicable to upgraded Windows 10 Machines? As I am forced to think that it might shift of inter phase.

    1. I am not sure I fully follow your comment about `inter phase`, but, yes `AF_UNIX` will be available in generally released builds starting next major release in 2018.

  9. Does this mean that we can make a FIFO in the file system and programs can use it for communication using normal file APIs?

    1. FIFO is not available in Windows today. There are alternate mechanisms such as named pipes that you can use.

  10. 劉斌 says:

    it’s not necessary, MSMQ、COM/COM+ etc. these better than simple socket,more scalable。

  11. aliotan says:

    hello
    your blog is very helpful and very promising. You wrote that some features are not currently available yet – section “Unsupported\unavailable” – would you share information why and when we would have them (i.e Ancillary…)

    thanks

    1. Thanks. Glad you enjoyed it. Some features such as interop with WSL is coming “soon”. Other features will have to be prioritized depending on the user feedback. If there is a feature that you would like, please provide feedback at user voice page and make sure to capture your scenario there.

      https://wpdev.uservoice.com/forums/110705-universal-windows-platform?category_id=22999

  12. BobVul says:

    Since there’s a few comments about SOCK_SEQPACKET and SOCK_DGRAM here, and I couldn’t find an existing uservoice, I’ve created one here: https://wpdev.uservoice.com/forums/110705-universal-windows-platform/suggestions/33247168-add-sock-seqpacket-and-sock-dgram-to-af-unix

Skip to main content