How to install Splunk on HDINSIGHT with a custom action script

  • Article

 

Recently I worked with a customer that wanted to use Splunk Enterprise and Splunk Forwarder to monitor and manage their HDINSIGHT Storm cluster. You can learn more about Splunk at https://www.splunk.com/ . Splunk has a version called Splunk Light that you can download for free. There are some restrictions, so read the documentation and license agreement. Splunk Light offers real-time log search and analysis. In this post I will show you how to install Splunk Light on all nodes of an HDINSIGHT cluster.

HDINSIGHT has a feature called custom action scripts that allow you to customize an HDINSIGHT's cluster during provisioning. With custom action scripts, you can do things like install software, change Hadoop configuration files, set environment variables, and many other things. You can read more about HDINSIGHT's custom action scripts at https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-customize-cluster/ and https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-script-actions/

 

Let's get started.

We first need to download Splunk light for windows X64 msi from their web site at https://www.splunk.com/en_us/products/splunk-light.html. I'll be using splunklight-6.2.2-255606-x64-release.msi for the article, but download the latest version from Splunk.

Next we need to create public container in an Azure Storage account. The storage account and container must remain accessible throughout the lifetime of the cluster. HDINSIGHT can re-image nodes and when it does, the custom action script will be executed again. The custom action script also has to be idempotent. The Azure Storage account can be the HDINSIGHT default account or an additional storage account. For example I have created a scriptactions container in my storage account for this purpose (https://portalvhdszmhjyc3XXXXXX.blob.core.windows.net/scriptactions).

Now that we have our .msi file in a container in our storage account we will write and place our custom action script in the container. HDINSIGHT custom action scripts are PowerShell scripts. I have named mine splunk-installer-v1.ps1.The full script is below.

The script checks for the existence of a c:\apps\dist\temp_splunk folder. C:\apps and D:\ are safe to write data to. The re-image process will delete or re ACL files and folders in other locations. We will use this folder to copy our .msi file from the storage container to the HDINSIGHT node. If the folder does not exit we create it. The script then downloads the .msi to the c:\apps\dist\temp_splunk folder. It then executes the msi with msiexec with the /lv, AGREETOLICENSE =YES and /quiet switches. The /lv creates a verbose install log in case the install fails. We can search this log for "Return Value 3" for the reason of the fatal error. The AGREETOLICENSE =YES indicates that we agree to Splunk's License agreement, and the /quiet does a silent installation of the msi package. The next code block loops for up to five minutes checking for the Splunkd service. This is the service name and not the display name. We need to give the msi time to execute and install. This code block allow for that. Finally we clean up after ourselves by deleting the c:\apps\dist\temp_splunk folder. Go ahead and review the script. You can add exception handling but I wanted to keep things simple for the article.

 

 

You can execute the script either through Azure PowerShell cmdlets or .net code. You can use the Add-AzureHDInsightScriptAction cmdlet, https://msdn.microsoft.com/en-us/library/dn858088.aspx.

You can also use the azure portal and do a HDINSIGHT custom create. The last form will give you the option to add your custom action script. You can give the script action a name and which nodes to run the script on.

 

 

The cluster customization stage is the last stage before it becomes operational. If the cluster is created but the Splunk software is not installed under D:\program files\Splunk, you can review the install log at c:\applications\splunk-install.log. If the whole cluster creation failed due to your custom action script you can review the HDINSIGHT install logs. Each HDINSIGHT cluster provision writes a setuplog to Azure Table Storage. You can review this log to troubleshoot the cluster provisioning failure. The following blog post discusses the log in the Azure Table Storage and how to access them. https://blogs.msdn.com/b/brian_swan/archive/2014/01/06/accessing-hadoop-logs-in-hdinsight.aspx. This is your best option to determine why your custom action script failed.

You now can remote desktop into the node and use windows search and execute Splunk Light. The default user is admin and the password is changeme. You will be required to change the password. You can now start to add data to monitor and perform searches and log analysis. To find out more about using Splunk visit https://www.splunk.com/view/SP-CAAAG2R

 

 

I hope this show how you can customize you HDINSIGHT with Splunk or other software. Our development team has written examples of custom action scripts for Spark, R, Solr, and Giraph. These are good examples to review to learn more about HDINSIGHT's custom action scripts.

Install Spark - See Install and use Spark on HDInsight clusters.

Install R - See Install and use R on HDInsight clusters.

Install Solr - See Install and use Solr on HDInsight clusters.

Install Giraph - See Install and use Giraph on HDInsight clusters.

 

Happy Splunking!

Bill