SWAP space in Linux VM’s on Windows Azure – Part 1

With the recent GA of IaaS on Windows Azure, many new scenarios are opening up. One such scenario was a recent project where I needed to deploy a multi-node Hadoop Cluster on Linux based machines in Azure. To get started, I used a CentOS 6.3 image from the Azure gallery to provision a medium sized VM and proceeded to deploy a single node core Hadoop. This seemed to work fine except that as I started testing slightly heavier workloads, I noticed that the VM would often freeze up or become unresponsive.

It is not difficult to guess that this had something to do with the limited resources of a medium sized VM – after all it has only 2 CPU cores and 3.5 GB of memory. But, while I would have expected the workloads to run slow, I was not expecting the whole VM to become unresponsive or start dropping connections. After discussing this issue with my friends and colleagues, we determined that the VM did not have Swap (i.e. what is called a page file on Windows) configured at all. Thus, its virtual memory system could not swap to disk when the memory pressure increased.

You can check how the system is doing memory wise by running the “free” command from the Linux shell prompt, and in particular, you can use “cat /proc/swaps” to see the state of the swap space – how much is configured and how much is in use. See the screenshot, below.

If the swap space is not configured at all, as is the default case with Linux VM’s provisioned in Azure IaaS, the “cat /proc/swaps” will return nothing, and likewise the “free” command will not show any activity in swap.

An interesting question is why doesn’t the VM provisioning using a Linux library image (i.e. from the Azure gallery) automatically configure swap space. Perhaps the thinking is that the user should decide on the size and location of the swap and do it post provisioning. However, this information is not documented anywhere, and it is quite possible that one continues to use the VM without the swap ever getting configured till processes begin to crash or the VM freezes up.

That said, once the problem has been diagnosed, the solution is a simple set steps to configure a file based swap on the resource disk; a medium sized IaaS VM in Azure comes with 135 GB of resource disk mounted as “/mnt/resource”. Given below is a walkthrough of the steps for configuring a file based swap space on the VM.

• Use the “fallocate” command to allocate a swap file of suitable size, say, 5GB on the resource disk. The syntax is: “fallocate -l 5g /mnt/resource/swap5g” where “swap5g” is the name of the file
• Change the permissions on the file using “chmod” command so that only the root user has read/write permissions on the swap file. The syntax is: “chmod 600 /mnt/resource/swap5g”
• Use the “mkswap” command to set up the file as swap area. The syntax is: “mkswap /mnt/resource/swap5g”
• Enable the use of the swap file using “swapon” command. The syntax is: “swapon /mnt/resource/swap5g”
• The swap is ready for use now, and the “cat /proc/swaps” command should confirm it now. Add an entry to the “/etc/fstab” file so that even if the VM recycles in Azure, the swap settings are retained. The syntax is: echo “/mnt/resource/swap5g none swap sw 0 0” >> /etc/fstab

Here is a transcript of the above commands executed in my VM.

Acknowledgement: Thanks to my colleague Amit Srivastava for help with troubleshooting and resolving the swap issue.