SCP.Net is now available on HDInsight Linux clusters 3.4 and above.
Note: HDInsight Storm team recommends HDI 3.5 clusters for users looking to migrate their SCP.Net topologies from Windows to Linux.
Development of SCP.Net Topology
Azure Datalake Tools for Visual Studio
HDInsight tools for Visual Studio does not support submission of SCP.Net topologies to HDI Linux Storm clusters.
The latest Azure Datalake Tools for Visual Studio is needed to develop and submit SCP.Net topologies to HDI Linux Storm clusters.
The tools are available for Visual Studio 2013 and 2015.
Please note: Azure DataLake tools has compatibility issues with other/older extensions to Visual Studio.
One known issue is where no clusters are shown in the drop-down for topology submission.
If you encounter this issue, please uninstall all extensions to Visual studio, and re-install Azure Datalake tools.
SCP.Net generates a zip file consisting of the topology DLLs and dependency jars.
It uses Java (if found in the PATH) or .net to generate the zip. Unfortunately, zip files generated with .net are not compatible with Linux clusters.
Java installation requirements:
- Java should be installed on the machine (JDK1.7+) (example: C:\JDK1.7)
- JAVA_HOME system variable should be set to the installation path. (C:\JDK1.7)
- PATH system variable should include %JAVA_HOME%\bin
One way to verify if your Java setup is good is to:
- Launch a command window
- execute: java -version
The above command should output java usage.
- Execute: jar
The above command should output jar usage
Creating a SCP.Net topology
Submission of Topology
- Choose the Storm Linux cluster from the drop down.
Java File Path
If you have java jar dependencies, you can include their full paths as a ; string.
you can use
\*to indicate all jars in a given directory.
Investigating Submission failures
Topology submissions can fail due to many reasons:
- Required java dependencies are not included
- Incompatible java jar dependencies. Example: Storm-eventhub-spouts-9.jar is incompatible with Storm 1.0.1. If you submit a jar with that dependency, topolopgy submission will fail.
- Duplicate names for topologies
Topology submission operations are logged into
/var/log/hdinsight-scpwebapi/hdinsight-scpwebapi.out on the active head node.
Users can look into the above file on head node to identify causes for submission failures. (In case where the output from the tool is not helpful).