Azure PowerShell 0.8.14 Released, fixes problems with pipelining HDInsight configuration cmdlets

  • Article

We recently pushed out the 0.8.14 release of Azure PowerShell. This release includes some updates to the following cmdlets to ensure that values passed in via the PowerShell pipeline, or via the -Config parameter, are maintained:

  • Set-AzureHDInsightDefaultStorage
  • Add-AzureHDInsightStorage
  • Add-AzureHDInsightMetastore

Previously if you had done something like:

 $myconfig = New-AzureHDInsightClusterConfig -ClusterSizeInNodes 2 -ClusterType HBase
$myconfig = Set-AzureHDInsightDefaultStorage -Config $myconfig `
                  -StorageContainerName "somecontainer" `
                  -StorageAccountName "somedefaultstorage.blob.core.windows.net" `
                  -StorageAccountKey "U09NRVRISU5HIEVOQ09ERUQgSU4gQkFTRTY0Lg=="
$myconfig = Add-AzureHDInsightStorage -Config $myconfig `
                  -StorageAccountName "someaddedstorage.blob.core.windows.net" `
                  -StorageAccountKey "U09NRVRISU5HIEVOQ09ERUQgSU4gQkFTRTY0Lg=="
$myconfig = Add-AzureHDInsightMetastore -Config $myconfig `
                  -Credential (Get-Credential) -DatabaseName "somedatabase" `
                  -MetastoreType HiveMetastore -SqlAzureServerName "someserver"
$myconfig | Format-Custom # This is where you would usually call New-AzureHDInsightCluster

or

 New-AzureHDInsightClusterConfig -ClusterSizeInNodes 2 -ClusterType HBase | `
Set-AzureHDInsightDefaultStorage -StorageContainerName "somecontainer" `
                  -StorageAccountName "somedefaultstorage.blob.core.windows.net" `
                  -StorageAccountKey "U09NRVRISU5HIEVOQ09ERUQgSU4gQkFTRTY0Lg==" | `
Add-AzureHDInsightStorage -StorageAccountName "someaddedstorage.blob.core.windows.net" `
                  -StorageAccountKey "U09NRVRISU5HIEVOQ09ERUQgSU4gQkFTRTY0Lg==" | `
Add-AzureHDInsightMetastore -Credential (Get-Credential) -DatabaseName "somedatabase" `
                  -MetastoreType HiveMetastore -SqlAzureServerName "someserver" | `
Format-Custom # This is where you would usually call New-AzureHDInsightCluster

You would have found that some elements, like the initial ClusterType of "HBase" would have been lost from the configuration. These values will now be maintained as you add elements to the configuration. This should also address some scenarios where people have found that they needed to set options in a particular order for them to be maintained.

Side Note: Passing a configuration object to Format-Custom before using it for New-AzureHDInsightCluster is a great way to troubleshoot whether the configuration object is set up as you expect.