Advances in Windows Vista TCP/IP


The Windows Vista TCP/IP stack has made tremendous improvements in its efficiency, taking full advantage of hardware advances (e.g. gigabit networking). As explained by Murari in a previous posting (Advances in Windows TCP/IP Networking), there are a number of bottlenecks that affect TCP throughput. Here, I will give some examples of how we’ve addressed these bottlenecks in the Windows Vista TCP/IP stack.


 


TCP auto-tuning: At any given time, the amount that TCP can send is governed by three factors: the congestion window, the receive window and the number of bytes available to send. Without using TCP window scaling (which is disabled by default in previous versions of Windows), the maximum receive window a receiver can advertise is 64K bytes. Since the congestion window is usually greater than 64K bytes in high-bandwidth/high-latency networks, the receive window is often the limiting factor if the application is submitting enough data.


 


In previous versions of Windows, users can work around this problem by setting the TcpWindowSize registry key value. However, TcpWindowSize is a global setting applied to all connections, and it’s often hard for users to know the appropriate window size to set.


 


To address this issue in Windows Vista, we implemented TCP auto-tuning. It enables TCP window scaling by default and automatically tunes the TCP receive window size based on the bandwidth delay product (BDP) and the rate at which the application reads data from the connection. With TCP auto-tuning, we have seen 1000% (10x) throughput improvements in internal testing over underutilized wide-area network links.


 


Receive-Side Scaling: Networking stacks face a number of challenges in scaling their receive processing across processors on multi-processor systems.  For instance, on previous versions of Windows all packets indicated in a single interrupt service routine (ISR) are typically processed in a single deferred procedure call (DPC) queued to a specific processor to avoid packet reordering. Until the outstanding DPC completes, no more receive indication interrupts can be triggered. As a result, only one processor can be used at any given time for processing received packets for a single network adapter.


 


Receive-side scaling (RSS) is our solution for this issue in the new networking stack: it enables parallelized processing of received packets on multiple processors, while avoiding packet reordering. It achieves parallelism by allowing ISRs to queue DPCs on multiple processors, enabling packet processing on multiple processors at the same time. It avoids packet reordering by separating packets into flows, and using a single processor for processing all the packets for a given flow. Packets are separated into flows by computing a hash value based on specific fields in each packet, and the resulting hash values are used to select a processor for processing the flow. Using TCP as an example, this approach ensures that all packets belonging to a given TCP connection will be queued to the same processor, in the same order that they were received by the network adapter.


 


TCP offload: Previous Windows releases already support network task offload for stateless per-packet operations (e.g. LSO, checksum offload etc). In Windows Vista, in addition to the offloads supported on previous Windows releases, we’ve also introduced support for TCP chimney offload. TCP chimney offload enables Windows to offload all TCP processing for a connection to a network adapter. Offloads are initiated on a per-connection basis, based on heuristics. Compared to task offload, TCP chimney offload further reduces networking-related CPU overhead, enabling better overall system performance by freeing up the CPU for other tasks.


 


We have also responded to customer feedback by making the Windows Vista TCP/IP stack much smarter and more adaptive in a number of scenarios.  One such improvement we’ve made is to enable TCP black-hole detection by default in Windows Vista.


 


Historically, problems due to the presence of black-hole routers have been among the highest product support call generators for the previous Windows networking stacks. To understand why, it’s important to know that TCP/IP relies on ICMP packet-too-big error messages to discover the maximum transmission unit (MTU) for any given connection’s path, so that it can reduce the size of the packets that it sends if they’re too large. If a router along the path does not send back ICMP error messages, or if a firewall drops ICMP error messages, TCP will never find out that its packets are too big. As a result, it will retransmit the packets repeatedly with the same size, up to its maximum number of retransmissions and, when it gets no responses, it will terminate the connection.


 


Black hole router detection is a mechanism used in this scenario to automatically reduce the size of the packets sent for a connection, based on the current status of the connection, in the absence of feedback from ICMP packet too big error messages. This mechanism was disabled by default in previous versions of Windows, because previous approaches would often yield too many false positives, lowering the packet size unnecessarily and reducing performance.  In Windows Vista, our improvements have reduced the likelihood of false positives and, consequently, minimized the adverse performance impact, enabling us to turn on black hole detection by default in the upcoming Beta 2 release.


 


There are many, many more innovations that we’ve made in the network stack, far more than I can write about in this one posting. Stay tuned for more…


 


Xinyan Zan


Software Development Engineer, TCP/IP Networking

Comments (37)

  1. Andrew Meldrum says:

    Pity it dowsn’t work. Adobe, Disney and about 1/2 the internet isn’t accessable with Firefox. I wonder if M$ have tied BlackHole detection into the IE libraries??

  2. wndpteam says:

    Andrew: I’m sorry that we missed your earlier comment, can you be more specific? Black hole detection in Vista is a kernel tcp/ip stack feature, it shouldn’t matter whether the browser is IE or firefox.

    If firefox is having trouble browsing to specific sites on Vista that work on XP, there might be some other application compatability problem that we haven’t detected yet. If you could post more or send an email (http://blogs.msdn.com/wndp/contact.aspx) with specific sites that are having trouble with and on which vista build you are seeing issues, we can follow up with the app compat person who helps with firefox issues.

    — Ari

  3. I’ve been in converstations with consultants on some large global companies on determining which deployment

  4. Наткнулся на замечательное руководство по оптимизации windows vista.

  5. Al says:

    This is all well and good, but it doesn’t mention that some client-side routers do not like these settings, and there is no way to disable it other than deep digging in the shell.

    Right now I’m faced with the problem of 90% of my TCP packets being returned from my router for being out of sequence (during high bandwidth sessions), duplicate ACK, or other problems that should never occur. I have an ethereal (aka wireshark) log that I’m looking at right now, and at a random point during a download, I have 7 ‘green’ successful packets (5 of which are TCP segment of a reassembled PDU), and 24 "black" bad packets, which are about 50/50 out of order + dup ACK. This means 2 out of these 31 packets are successful traffic. Cool, huh?

  6. wndpteam says:

    hey Al. If you’d like to submit a packet trace and tell us what router you are running, we’ll take a look. Use the Email link at the top of the page to get communication going.

  7. Zach says:

    The reason for your dropped packet is that Vista has the ability to increase the TCP window size to a 32-bit value.  Since this is a relatively new option.  Most firewalls, routers, etc. are not capable of dealing with these larger window sizes.  I believe patches were made dealing with this issue.

  8. Mike says:

    Could TCP Autotuning be the cause of a problem I keep getting with Windows Mail?

    Quite often I get an error code 0x800CCC19 and my server times out without sending mail. The only way I can fix this is to shut Windows Vista & then restart. Very annoying.

    This seems to happen irrespective of Firewall settings.

    I have read somewhere that disabling TCP Autotuning will fix this?

  9. Jrz says:

    C:>netsh interface tcp set global autotuninglevel=disabled

    Ok.

    C:>netsh interface tcp show global

    Querying active state…

    TCP Global Parameters

    ———————————————-

    Receive-Side Scaling State          : enabled

    Chimney Offload State               : enabled

    Receive Window Auto-Tuning Level    : disabled

    Add-On Congestion Control Provider  : none

    ECN Capability                      : disabled

    RFC 1323 Timestamps                 : disabled

  10. Hi, I was wondering if anyone could email me and maybe tell me how to get ecn enabled on my computer I have windows vista Home premium. I have a linksys router befsr41 v4.3 I also have a cable modem connection through Comcast I believe it’s 768 package I only use the router as a hardware firewall that’s it. I’m not running multiple computers or servers or anything like that.C:>netsh interface tcp show global I ran this and recieve side scailing is enabled chimmney offload state is enabled. The auto tuning for tcp is disabled they recommended to do that. Add on congestion controller provider is ctcp. ecn capability is disabled. I can’t figure out how to enable it in my router that is through windows I do and windows take it but, I think both windows vista and my router have to support it. Also rfc 1323 timestamps is enabled. so any help you could give me would be awsomme

    thank you so much Steve

  11. So how do we set MaxMTU?

    An application that I’m running requires a MaxMTU of at least 1364 in order to function. How do I change that setting in Windows Vista?

  12. Jason says:

    hi,

    I’m running Vista home premium 1GB RAM connecting to the internet via wireless linksys router connected to Comcast Cable internet. All of the other computers connected to the same router are running either XP home, or XP pro, and have no internet/network issues. constantly through the day, i have to turn off, and back on my wireless adaptor on this laptop in order to reconnect. it is wreaking havoc on everything I do with the internet from chatting to playing online games to surfing the web. I’ve changed the autotune feature moments ago, and I am going to let you all know if it works for me… Thank you.

  13. Jason says:

    nope! It didn’t work. This computer is aggravating me… if I had $150 i would go buy windows XP, just so I could have this problem fixed… but I don’t… instead I spent $900 on a laptop and wasn’t given a choice of operating systems… I think Microsoft screwed up BIG time when they made Windows Vista.

  14. wndpteam says:

    Jason: If the other machines are not having issues, I would check a couple things:

    a) Are you using the same wireless security settings on Vista as you are on XP?

    b) Check for updates for both the firwarm for your router and for the drivers on the wireless card in your computer.

    Christopher: As far as I understand the default MaxMTU on a ethernet network is 1500. You can see the current setting via "netsh int ipv4 show interfaces"

  15. wndpteam says:

    The following enables ECN, but why do you want to set it?

    netsh interface tcp set global ecncapability=enabled

    Disabled, or default:

    netsh interface tcp set global ecncapability=default

    netsh interface tcp set global ecncapability=disabled

    and since this is the second question around this area, check out

    netsh interface tcp set global ?

    for help on how to set the various machine wide TCP settings.

  16. zman says:

    I just disabled rss and autotuning just so I could map a drive to Windows 2003 servers without waiting 20-30 seconds for each click to take complete.  The window would show the contents but an indicator bar in the folder name space in explorer would take 20-30 seconds to complete whatever it was doing before I could actually access the files.  Now everything is nice and snappy like Windows XP and Windows 2003.  What exactly is the autotuning feature attempting to do every time I open a folder?  Shouldn’t it autotune the initial session once and leave it alone?  The constant autotuning seems redundant and does not work in practice at least with my switches and seriously interfered with streaming audio and video even on my gigabit network.  If this is a limitation of my switch or a compatibility issue then it should be noted but I have never heard of it until today.

  17. Frosty says:

    Plz, plz plz plz plz plz will you tell me how to enable my rss….Every where i look I see…..

    Receive-Side Scaling State : enabled

    Chimney Offload State : enabled

    Receive Window Auto-Tuning Level : disabled

    Add-On Congestion Control Provider : none

    ECN Capability : disabled

    RFC 1323 Timestamps : disabled

    BUT MINE IS…..

    Receive-Side Scaling State : DISABLED

    Chimney Offload State : enabled

    Receive Window Auto-Tuning Level : disabled

    Add-On Congestion Control Provider : none

    ECN Capability : disabled

    RFC 1323 Timestamps : disabled ….

    Also whnever I open an internet page it pauses ofr about second first…

    Please help

    Thank you

    You want:

    netsh int tcp set global rss=enabled

    Regarding IE startup time, check out this troubleshooting page.

      — Ari

  18. Mike says:

    We are running 2 Vista machines & 1 still on XP (the XP for 5 years with no problems.)

    Also running 1 linux machine with similarly trouble free E-mail via the same servers.

    On the 2 Vista machines , I am continually getting Mail Server time-outs with error code 0X800CCC19. Have tried various fixes all to no avail. Have to restart the PC everytime it happens.

    Why does Microsoft continue to introduce expensive new products which simply refuse to work in the real world?

  19. Mike says:

    How do i change the the TCP recieve window when autotuning is off? TCP optimizer and dr.tcp dont work.

    Apparently, you don’t. The registry key applied to all interfaces (not a good thing), we believe that autotunning is a much better model. Is there a specific scenario you are trying to achieve? — Ari
  20. Hello,

    I simply have to agree with everyone who is saying that Vista is the worst operating system ever to come out of Microsoft.  The networking is particularly evil.  I am a Microsoft partner, a Netgear Partner, a Linsksy Partner and a Dell solution partner.  I have no choice but to tell ALL of my business clients to avoid Vista at all cost because IT IS NOT INTEROPERABLE with business networks.  In order to get fully up to speed I recently added 2 brand new Dell system with Vista Business.  One is a new laptop that is essential to my work.  The fact that Microsof INTENTIONALLY (so you say) made Vista Incompatible with IPSEC standards is a real killer.  By definition, ANYONE who needs a lap top for business MUST be able to use standard IPSEC VPN clients from behind firewalls running NAT. (Have you ever stayed at a hotel on a business trip?) is just three weeks I have determined that Vista will NOT support IPSEC via NAT, does not support many applications and drivers, and crashed with a blue scrren or startup error on MULTIPLE Windows Update downloads. I have already had to format and reinstall Vista on the Laptop, and now I an very hesitant to even install the new Desktop.  I am getting 10 licenses for Vista via my MAPS subscription, but if I install them I will KILL my business.  PLEASE< if there is any hope for fixing Vista networking, tell where to find it.

    cmccallan@mccallantech.net

    Regarding your IPSEC NAT-T Tunnel mode issue, I would suggest reading this FAQ. Specifically this part:

    Q. Why do other vendors claim that they support standards and that Microsoft VPN technologies are proprietary?

    A. Most vendors making this claim are using IPSec tunnel mode for remote access. Unfortunately, the IPSec RFCs do not describe the use of IPSec tunnel mode for remote client access. In particular, the RFCs provide no mechanisms for user authentication, IP address assignment, and name server address assignment.

    As a result, vendors implementing a remote access solution based on IPSec tunnel mode have been forced to extend the protocol. These extensions are not standard, and drafts that were introduced to the IETF to define a standard have been withdrawn. As a result, there is no standard for remote client access using IPSec tunnel mode. Consequently, many vendor implementations are not interoperable.

    In contrast, Microsoft has followed several standards precisely using L2TP (RFC 2662, a Proposed Standard) as the remote access protocol and IPSec (RFCs 4301-4303, Proposed Standards) as the encryption protocol, combined in a manner described in the L2TP/IPSec RFC 3193.

    You might be intrested in all of the questions in the “VPN Standards and Interoperability” section of the page.

    Regarding any instability you’ve had with vista, if you let me know what bucket IDs you’ve hit. I’ll be happy to do some quick research into what might be the underlying cause. To get the bucketIds, I’ve put together a video: Finding the Crash Bucket ID
    Finding the Crash Bucket ID

  21. Nick Fiekowsky says:

    What is the max WindowSize in Autotuning? And how do I persuade Windows Server 2008 to increase WindowSize more aggressively?

    My test scenarios. Run outside business hours, when links have little utilization. Windows Server 2008 x64 on 4 GB machine with offloading to NIC. Both involve downloading 15 MByte PDF document from well-tuned Linux servers.

    a) US from Japan. 205 ms RTT on a 10 Mbps link that will burst to 20 Mbps. Achieved 10 Mbps rate with 128 KB window. Well-tuned Linux and XP with 1 MB window reach and sustain 20 Mbps.

    b) US to Singapore 280 ms RTT on 10 Mbps link that will burst to 0 Mbps. Started with 128 KB window. After 9 L-O-N-G seconds paused and went to 256 KB window. Stayed there for the rest of the transfer, again peaking at 10 Mbps sustained rate. Out-of-the-box Leopard on 2 GB MacBook (not Pro), well-tuned Linux & XP all achieve 20 Mbps.

    We also have 1 GByte links with latency up to 10 ms. Can we ever persuade WS08 to scale up to 1 MByte window or larger? What is WS08 version of BICtcp or Westwood?

    Appreciate any help making this OS sing.

  22. Nick says:

    Hi – I am working on a server farm at the moment – running Win2003 Sp2 – We would like to increase the window size to optimise the server – server communications.

    We have 2 Web 1 Application and a 2 node SQL Cluster.  All Servers communicate over a 1Gb Full Duplex Private network.  They are used to host a MOSS2007 implementation – can you recommend the best window sizes for these boxes? – I have been struggling to find anything meaningful to make sure that our implementation real flies.  Kind Regards,

    Nick

    While I don’t have specific recomendations, you can look at KB224829 and sections of The Cable Guy for what you can tweak to find optimal settings. — Ari
  23. Nick Fiekowsky says:

    Server Farm Nick – I’ve done some testing with WS03, offer an "it depends" recommendation.

    Objective is to get improvement without exhausting non-swappable memory:

    – 256 kByte windowsize & globalmax windowsize.

    – Timestamps on for web servers, not necessary for app & DB servers (if you tolerate specialized tuning)

    – Selective acks on

    – Reduce TcpTimedWaitDelay to 30 seconds to free memory more quickly

    – Modern NICs offer a lot of offload options – see advanced properties. Use them. Also increase the number of Receive & Transmit Buffers.

    Notes:

    – Reboot required for TCP tuning

    – Apply equivalent regedits to client devices (except Vista, WS08 and Mac Leopard) for full benefit. Most people report their home broadband gets faster, too.

    – WS03 x64 with 4 GB memory or more will support 1 MB max windowsize. This is useful for really broad band (1 Gbps WAN, 20+ mbps across oceans), provided you can apply equivalent tuning on the other end.

    – Some afd (winsock) parameters are so small one would think they haven’t been adjusted since W2k called 64 MB memory a large server. Adjust at your own risk, and lab test before production deployment.

    Rant:

    – Ping time is only part of latency. Consider the time required for the receiver to process the info (clear receive buffer) and sender to tee up the next buffer-load of bits. iPerf doesn’t include this.

  24. Anthony says:

    Why when using Windows Mail my router losses connect to the internet? This only happens on my 2 Vista PC. Never had this issue on XP. After I do a check mail and I receive my mail, my router losses internet connectivity. I had to unplug the router to fix this. Any answers out there?

    emailto:aimperato@acn.net

  25. smartman says:

    IF you need a easy smart vpn sultion, tell your IT to check out OpenVPN. if you have ? about it, goto http://openvpn.net/ . its free and open source

  26. smartman says:

    I agree Vista needs alot of work, but not really. See Vista is here for windows server 2008. Not server 2003. Its to get everyone off of XP and into vista. Just like when Support ended for win98. It was making a shift to XP

  27. Maz says:

    I am trying to find an easy solution to setting up an ethernet (crossover cable ) between a vista SP1 PC and a Mac with OSX Leopard. Mac sees PC files (wont print share yet!) and I can transfer, but vista receives transfers but the Mac  will not show up in vista explorer no matter what I do! Any help appreciated.

    If transfers are actually happening, it sounds like the issue isn’t in networking, but prehaps in something else. I don’t personally use OSX, so I’m not sure what could be happening…. If I shake my magic eight ball, I get “investigate the file types and expected location of the share”.

    — Ari

  28. Vijay says:

    hi,

    Could you please let me know how you have the receive window size increased in the Vista machines. I know it has the feature to auto tune the RWIN and window scaling but no matter what options I use in netsh I get only about 1.7Mbps. Where as in windows 2000 and XP i can get 6Mbps and more because I am able to set the parameters in the registry. Now I see that Vista PC is always keeping the window size to 64K (with scaling 8K x 8). How can this behavior be changed. I need to have larger window size for my setup.

    Thanks

  29. Andrew says:

    We like many other people are trying to see how we can hardwire the TCP recieve window to something which we know if optimised for the network we are running. If we cannot be allowed to do this then we will just have to take this global application into Linux as we would have been too tightly controlled by microsoft in a way that stops us from working.

  30. Vivek says:

    Andrew:

    The application always has the choice to specify its own receive buffer via the SO_RCVBUF socket option. When this socket option is specified, autotuning will no longer affect the socket.

    Now if autotuning doesnt provide optimal performance for your network, I would be happy to take a look at it and see how it can be improved. Could you please obtain network captures (using ethereal or netmon) of the case showing poor throughput, perferably taken at both the sender and receiver?

    Thanks

    Vivek

  31. den says:

    Sorry for a bit off-topic, but maybe somebody here knows to what advance in Vista we owe that ICMP raw sockets do not recieve TTL expired or UNREACHABLE type ICMP replies, and how to workaround that one?

  32. John says:

    I have been looking everywhere but cannot find how to increase the time before a packet times out, I’m running a VPN over a Sattelite link so the latency is quite high, say 2-3000ms combine this with an occasional key exchange on the tunnels or a bad day with the signal reception and I have my links timing out. ?

  33. steve says:

    John, re your vpn connection,it can be the IPSEC keepalives.  check your client to see if you can disable the keepalives all-together.

    I know that the nortel and checkpoint clients support disabling the keepalives.  

  34. Cecil Ward says:

    Any pointers to guidance about the virtues of enabling / disabling TCP timestamps?

  35. chris isong says:

    Hi Xinyan,

    I was reading your blog with interest. would you be able to help.

    My vista reliability and performance monitor shows 0 percent network utilization and the bandwidth is 1MB. When I download a file the speed is 10 percent(101kbps) of the 1MB.

    I never get anywhere near the 1MB bandwidth.

    I believe it is a problem in vista as my ISP confirms the speed up to the point of my router is 1MB.

    I have looked at and tried solutions such as running the netsh command, disabling remote compression differential in vista, running vista in safe mode with networking but have had no improvement.

    Please can you suggest something.

    Thanks.

    chris.

  36. Mike Lewis says:

    Hello, what are the recommendations for a virtual machine running in hyper-v? Since the NIC is virtual and not physical, shouldn't we disable all of these features?