Managing Your HDInsight Cluster using PowerShell – Update

  • Article

Since writing my last post Managing Your HDInsight Cluster and .Net Job Submissions using PowerShell, there have been some useful modifications to the Azure PowerShell Tools.

The HDInsight cmdlets no longer exist as these have now been integrated into the latest release of the Windows Azure Powershell Tools. This integration means:

  • You don’t need to specify Subscription parameter
  • If needed, you can use AAD authentication to Azure instead of certificates

Also, the cmdlets are fully backwards compatible meaning you don’t need to change your current scripts. The Subscription parameter is now optional but if it specified then it is honoured.

As such the cluster creation script will now be:

  1. Param($Hosts = 4, [string] $Cluster = $(throw "Cluster Name Required."), [string] $StorageContainer = "hadooproot")
  2.  
  3. # Get the subscription information and set variables
  4. $subscriptionInfo = Get-AzureSubscription -Default
  5. $subName = $subscriptionInfo | %{ $_.SubscriptionName }
  6. $subId = $subscriptionInfo | %{ $_.SubscriptionId }
  7. $cert = $subscriptionInfo | %{ $_.Certificate }
  8. $storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }
  9.  
  10. Select-AzureSubscription -SubscriptionName $subName
  11.  
  12. $key = Get-AzureStorageKey $storeAccount | %{ $_.Primary }
  13.  
  14. $storageAccountInfo = Get-AzureStorageAccount $storeAccount
  15. $location = $storageAccountInfo | %{ $_.Location }
  16.  
  17. $hadoopUsername = "Hadoop"
  18. $clusterUsername = "Admin"    
  19. $clusterPassword = "myclusterpassword"
  20.  
  21. $secpasswd = ConvertTo-SecureString $clusterPassword -AsPlainText -Force
  22. $clusterCreds = New-Object System.Management.Automation.PSCredential($clusterUsername, $secpasswd)
  23.  
  24. $clusterName = $Cluster
  25. $numberNodes = $Hosts
  26. $containerDefault = $StorageContainer
  27. $blobStorage = "$storeAccount.blob.core.windows.net"
  28.  
  29. # tidyup the root to ensure empty
  30. # Remove-AzureStorageContainer Name $containerDefault -Force
  31. Write-Host "Deleting old storage container contents: $containerDefault" -f yellow
  32. $blobs = Get-AzureStorageBlob -Container $containerDefault
  33. foreach($blob in $blobs)
  34. {
  35.     Remove-AzureStorageBlob -Container $containerDefault -Blob ($blob.Name)
  36. }
  37.  
  38. # Create the cluster
  39. Write-Host "Creating '$numberNodes' Node Cluster named: $clusterName" -f yellow
  40. Write-Host "Storage Account '$storeAccount' and Container '$containerDefault'" -f yellow
  41. Write-Host "User '$clusterUsername' Password '$clusterPassword'" -f green
  42.  
  43. New-AzureHDInsightCluster -Certificate $cert -Name $clusterName -Location $location -DefaultStorageAccountName $blobStorage -DefaultStorageAccountKey $key -DefaultStorageContainerName $containerDefault -Credential $clusterCreds -ClusterSizeInNodes $numberNodes
  44.  
  45. Write-Host "Created '$numberNodes' Node Cluster: $clusterName" -f yellow

The only changes are the selection of the subscription and the removal of the Subscription option when creating the cluster. Of course all the other scripts can easily be modified along the same lines; such as the cluster deletion script:

  1. Param($Cluster = $(throw "Cluster Name Required."))
  2.  
  3. # Get the subscription information and set variables
  4. $subscriptionInfo = Get-AzureSubscription -Default
  5. $subName = $subscriptionInfo | %{ $_.SubscriptionName }
  6. $subId = $subscriptionInfo | %{ $_.SubscriptionId }
  7. $cert = $subscriptionInfo | %{ $_.Certificate }
  8. $storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }
  9.  
  10. Select-AzureSubscription -SubscriptionName $subName
  11.  
  12. $clusterName = $Cluster
  13.  
  14. # Delete the cluster
  15. Write-Host "Deleting Cluster named: $clusterName" -f yellow
  16.  
  17. Remove-AzureHDInsightCluster $clusterName -Subscription $subId -Certificate $cert
  18.  
  19. Write-Host "Deleted Cluster $clusterName" -f yellow

This cluster creation script also contains an additional section to ensure that the default storage container does not container any leftover files, from previous cluster creations:

  1. $blobs = Get-AzureStorageBlob -Container $containerDefault
  2. foreach($blob in $blobs)
  3. {
  4.     Remove-AzureStorageBlob -Container $containerDefault -Blob ($blob.Name)
  5. }

The rationale behind this is to ensure that any files that may be left over from previous clusters creations/deletions are removed.