Why are the Hadoop services disabled on my HDInsight cluster

I came across this question while working with a few customers recently and thought I would share a few tips with others who may find it helpful. There are times when we may need to check the status of a Hadoop service or restart the service as part of troubleshooting an issue – for example, you may want to restart Oozie service or Hive Metastore service while troubleshooting an oozie issue or a Hive issue. To restart a Hadoop service on an HdInsight cluster on Windows, currently we need to Remote Desktop (aka RDP) to the Headnode of the cluster. Consider a scenario where you have enabled Remote Desktop to the cluster via 'Enable Remote' button on the Azure portal (https://manage.windowsazure.com) or other alternative ways available. You then RDP to the Headode of the HDInsight cluster and open Windows services console via start -> Run -> services.msc

And then you find that hadoop services are disabled, like below –

This is interesting, right? We would expect Hadoop services to be running on the HDInsight Headnode. Consider another scenario, you have run a Hive query from the Hive CLI on the HDInsight Headnode and as part of troubleshooting the query, you wanted to review the hive.log file. You go to %hive_home%\logs folder and find that logs folder is empty, like below –

Another possiblity is, the log file does exist under the logs folder but we are not seeing the log entries for our specific execution or test. All of the above scenarios (Hadoop services being disabled or some of the logs folder being empty or log entries being non-existent on the Headnode) have the same underlying reason, which is, when we have connected via RDP we have not landed on the currently active Headnode. Remember that, as described in the Azure HdInsight documentation here, for High availability implementation, each HDInsight cluster has two Headnodes – headnode0 and headnode1. Due to various reasons, Headnode failover may occur and either headnode0 or headnode1 can become the active Headnode at any given time. So, in order to find the running Hadoop services on the Headnode or to find the logs that exist on the Headnode, we need to log on to the currently active Headnode.

How do I find the current active headnode?

On the desktop of each HDInsight Headnode, you will see an icon called 'Hadoop service availability', as shown below -

Double clicking the 'Hadoop service availability' icon will show you the current active Headnode where the services are running. For example, we can see below that the active Headnode is headnode0.

Now open a Hadoop command prompt or any command prompt and run the hostname command to verify which node we are logged into – in the example below, we can see that we are on headnode1.

If the hostname and Hadoop service Availability page shows the same Headnode name, then we are already on the active headnode. In my case, current active headnode is headnode0 and RDP is connecting to headnode1.

How do I RDP to active Headnode?

If you run into the above scenario where the default RDP connection is not connecting you to HDInsight active Headnode, you can use a simple workaround like below –

Click on Connect button on the azure portal and on the prompt, save the RDP file (with .rdp extension) on your workstation, instead of opening it, like below –

Right click on the .rdp file you saved in the previois step and select 'open with' and then select 'choose default program' option, as shown below-

After you select Notepad or any other text editor the first time, it will be shown in your available options the next time onwards. Open the file with any text editor like Notepad, as shown below-

All we need to do is to switch the last digit (circled above) on the line 3 between 1 and 0 in order to switch between headnode1 and headnode0. For example, the above .rdp file logged me into headnode1. If I need my RDP connection to log me into headnode0, we need to change the file like this (only change is the last digit on the line 3)-

Now, save the .rdp file. To connect via this saved .RDP file, either double click on the saved icon or right click on the .rdp file and select 'Connect'

Once conneted, we can now verify that we are now on the active Headnode (in my example, headnode0) and hadoop services are running on this active Headnode, as shown below-

I hope this helps to clarify some confusion around this! Ideally RDP should connect us to active Headnode and we have requests logged to change the behavior – until that is implemented, you can use this simple workaround if you run into the scenarios described above.