SCP.Net with HDInsight Linux Storm clusters


SCP.Net is now available on HDInsight Linux clusters 3.4 and above.

Versions

storm-scp-version-matrix

Note: HDInsight Storm team recommends HDI 3.5 clusters for users looking to migrate their SCP.Net topologies from Windows to Linux.

 

Development of SCP.Net Topology

Pre-Steps

 

Azure Datalake Tools for Visual Studio

HDInsight tools for Visual Studio does not support submission of SCP.Net topologies to HDI Linux Storm clusters.

The latest Azure Datalake Tools for Visual Studio is needed to develop and submit SCP.Net topologies to HDI Linux Storm clusters.

The tools are available for Visual Studio 2013 and 2015.

 

Please note: Azure DataLake tools has compatibility issues with other/older extensions to Visual Studio.

One known issue is where no clusters are shown in the drop-down for topology submission.

If you encounter this issue, please uninstall all extensions to Visual studio, and re-install Azure Datalake tools.

 

Java

SCP.Net generates a zip file consisting of the topology DLLs and dependency jars.

It uses Java (if found in the PATH) or .net to generate the zip. Unfortunately, zip files generated with .net are not compatible with Linux clusters.

Java installation requirements:

  • Java should be installed on the machine (JDK1.7+) (example: C:\JDK1.7)
  • JAVA_HOME system variable should be set to the installation path. (C:\JDK1.7)
  • PATH system variable should include %JAVA_HOME%\bin

One way to verify if your Java setup is good is to:

  1. Launch a command window
  2. execute: java -version
    The above command should output java usage.
  3. Execute: jar
    The above command should output jar usage

 

Creating a SCP.Net topology

    • Open visual studio, and the create new project dialog should have storm templates.visualstudio2015_newproject

 

    • Select Storm Project from Azure Datalake Templatesvisualstudio2015_newstormproject

 

    • For HDI 3.5, the SCP.Net nuget package needs to be updated to 1.0.0.1 version (or the latest available).visualstudio2015_stormproject_managenugetpackagesvisualstudio2015_newstormproject_updatescpnet_package_version

 

  • After update the package.config should look as belowvisualstudio2015_stormproject_updatedpackageconfig

 

Submission of Topology

    • Right click on the project and select Submit to Storm on HDInsight … optionvisualstudio2015_stormproject_submission_menu

 

  • Choose the Storm Linux cluster from the drop down.
    visualstudio2015_stormproject_submission_dialog
     

    Java File Path

    If you have java jar dependencies, you can include their full paths as a ; string.
    or
    you can use \* to indicate all jars in a given directory.

Investigating Submission failures

 
Topology submissions can fail due to many reasons:

  • Required java dependencies are not included
  • Incompatible java jar dependencies. Example: Storm-eventhub-spouts-9.jar is incompatible with Storm 1.0.1. If you submit a jar with that dependency, topolopgy submission will fail.
  • Duplicate names for topologies

 

Logs

Topology submission operations are logged into /var/log/hdinsight-scpwebapi/hdinsight-scpwebapi.out on the active head node.
Users can look into the above file on head node to identify causes for submission failures. (In case where the output from the tool is not helpful).

Comments (0)

Skip to main content