Save Time With Azure and the Magic of PowerShell Jobs


I was working on a project where I needed to drop and create Azure SQL databases repeatedly as I tested and tried different scenarios. Of course you start out thinking, “I’ll just drop and create them in the portal, it’s easy.”
After a couple times clicking around in the portal page, you begin to understand that is a clumsy and laborious way to repeatedly do something in Azure. It took me several minutes to drop and create one database, let alone several.
I turned my attention to Azure PowerShell to automate my task and came up with this:

$serverName='mysqlserver'
$resourceGroup='mysqlgroup'
$dbNames=@('DB1','DB2','DB3')

$startTime=Get-Date

$dbNames | foreach {
   Remove-AzureRmSqlDatabase -DatabaseName $_ -ServerName $serverName -ResourceGroupName $resourceGroup
}

$dbNames | foreach {
   New-AzureRmSqlDatabase -DatabaseName $_ -ServerName $serverName -Edition Basic -ResourceGroupName $resourceGroup
}

Write-Host "Run time:" ((Get-Date) - $startTime).TotalSeconds "seconds"

This completed dropping and creating all my databases in around 125 seconds (a little over 2 minutes). Sometimes longer depending on WAN to Azure and Azure resources themselves.
The script cut my time and labor greatly, but as the project wore on, I began to think, “If I were coding this in C# or C++, I would do this in parallel, not synchronously. None of these drop/create tasks depend on each other in the slightest.”
I took a foray into PowerShell jobs and wrote this:

$serverName='mysqlserver'
$resourceGroup='mysqlgroup'
$dbNames=@('DB1','DB2','DB3')
$jobs=@()

$startTime=Get-Date

$dbNames | foreach {
   $jobs += Start-Job -ScriptBlock {
     Remove-AzureRmSqlDatabase -DatabaseName $_ -ServerName $serverName -ResourceGroupName $resourceGroup
   } -Name "Drop $_"
}

$jobs | Wait-Job
$jobs=@()

$dbNames | foreach {
   $jobs += Start-Job -ScriptBlock {
     New-AzureRmSqlDatabase -DatabaseName $_ -ServerName $serverName -Edition Basic -ResourceGroupName $resourceGroup
   } -Name "Create $_"
}

$jobs | Wait-Job
Write-Host "Run time:" ((Get-Date) - $startTime).TotalSeconds "seconds"

This completed dropping and creating all the same databases in 7 seconds!
It did all these highly latent Azure tasks in parallel.

The code in the asynchronous script creates an empty array to hold the PowerShell jobs that will be started. It then starts a separate job consisting of a remove (drop) command for each of the databases. The script does not need to wait to start the jobs, it just holds a reference to each job in the array as it starts them in parallel.
Once all the remove jobs are running, the script then waits for all for them to complete before continuing.
It then iterates through the databases starting jobs to create each of them and storing a reference to each job in the array.
Finally, it waits for all the creation jobs to complete.

When working with a highly latent system (especially like the cloud), take a hard look at what you can do in parallel and invest the time to leverage that capability. You will likely be shocked at the time savings (= money savings) you will gain. I know, it’s CS101, but easy to get lazy and forget just how dramatic the difference can be.

References
Remove-AzureRmSqlDatabase
New-AzureRmSqlDatabase
PowerShell Background Jobs
PowerShell Arrays

Comments (0)

Skip to main content