Large Tenant and PowerShell I/O timeout


 

So did you ever run a large office365 Tenant with million of mailboxes and you wanted to run a report about devices stat for example and it keep failing?

Did you ever face a timeout error when you run a long report “The I/O operation has been aborted because of either a thread exit or an application request”?

It’s not quite easy with the network latency and the exchange throttling policy which is hardly implemented at the office365 to have a long running session opened with exchange especially that some commands consume more time like Get-mobileDeviceStatistics

So how can we bypass this limitation?

Let’s start with a sample code to get the device statistics for all users in one tenant.

PowerShell

#Retrieve the list of mailboxes

$MbxList=(Get-Mailbox -resultsize unlimited).alias

#Loop in the list and retrieve the device’s information

$file = "e:\activesync\office365$((get-date).tostring("yyyyMMdd")).csv"

ForEach($mbx in $MbxList)

{

$deviceinfo = Get-mobileDeviceStatistics -mailbox $mbx | select @{name="mailbox"; expression={$mbx}},devicetype, devicemodel, devicefriendlyname, deviceos, lastsuccesssync

If ($deviceinfo -ne $null) { 

If (Test-Path $file){

                $mbx.alias + “,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file -Append

         }

Else{

$deviceinfo | Export-Csv $file -Encoding ASCII -notypeinformation

         }

$deviceinfo = $null

    }

}

 

With over 50k mailboxes we might hit the limitations and receive the famous error

Starting a command on the remote server failed with the following error message : The I/O operation has been aborted because of either a thread exit or an application request. For more

 

So how to bypass the limitations?

A first analysis will show that we are running two commands Get-mailbox and Get-mobileDeviceStatistics so why not we run each command in a separate session.

In stage 1 we starts with Get-Mailbox and send the results to a csv file on the disk and once the command finish we start stage 2 with a new PowerShell session and run the second command for Get-mobileDeviceStatistics .

Looks promising but again with 50k or more of mailboxes the Get-mobileDeviceStatistics will throw the same error as it last for a long time.

What if we have the list of mailboxes less than 1000 mailbox? cool that will work fine, but is it possible to load 1000 entries only from the csv file?

What’s really brilliant is that you can load some lines from any csv file using the ‘-skip’ option in combine with the ‘-First’ so we load from 0..1000 and close the file, open again and load from 1000…2000.

 

PowerShell

$Offset=0;

$PageSize=1000;

$mbxMax=(import-csv users.csv).count

$file = "office365$((get-date).tostring("yyyyMMdd")).csv"

do{   

#Load the list of users from a csv file limited with the pageSize 1000 starting from the line $Offset + 1

$mbxlist=@(import-csv users.csv|select-object -skip $Offset -First $PageSize)

"Start at offset $($Offset) till $($Offset+$PageSize)"

ForEach($mbx in $mbxlist)

       {

"Start Processing $($mbx.alias)"

$deviceinfo = Get-mobileDeviceStatistics -mailbox $mbx.alias | select @{name="mailbox"; expression={$_}},devicetype, devicemodel, devicefriendlyname, deviceos, lastsuccesssync

If ($deviceinfo -ne $null)

              { 

       If (Test-Path $file)

                     {

                        $mbx.alias + ”,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file –Append

           #You can use –Encoding Unicode if you deal with Unicode characters like Arabic

                     }

       Else

                     {

                $deviceinfo | Export-Csv $file -Encoding ASCII -notypeinformation

                     }

                $deviceinfo = $null

}

      }

"------------------------------"

#Increase the start point for the next chunk

$Offset+=$PageSize

} while($offset -lt $mbxMax)

 

$PageSize variable here defines the number of items will processed in each run

That will split the 50k mailboxes into multiple chunks and process each chunk in a separate ForEach loop.

But we still run all commands in the same session, what we really need to add is to run each chunk in a new session and aggregate all the results.

This is the time to write a small function for office365 remote connection so that we close the session and reopen a new one without any interaction.

 

PowerShell
Function New-O365ExchangeSession()

{

param(

[parameter(mandatory=$true)]

$365master)

#close any old remote session

Get-PSSession | Remove-PSSession -Confirm:$false

#start a new office 365 remote session

$365session = New-PSSession -ConfigurationName "Microsoft.Exchange" -ConnectionUri 'https://ps.outlook.com/powershell' -Credential $365master -Authentication Basic

-AllowRedirection

$office365 = Import-PSSession $365session

}

 

The last thing is to call the remote connection function inside the foreach loop so we run each loop in a separate remote session

 

Finally we reached the objective; Enjoy Winking smile

Final Code

Function New-O365ExchangeSession()

{

param(

[parameter(mandatory=$true)]

$365master)

#close any old remote session

Get-PSSession | Remove-PSSession -Confirm:$false

#start a new office 365 remote session

$365session = New-PSSession -ConfigurationName "Microsoft.Exchange" -ConnectionUri 'https://ps.outlook.com/powershell' -Credential $365master -Authentication Basic

-AllowRedirection

$office365 = Import-PSSession $365session

}

 
#Main

$Offset=0;

$PageSize=1000;

$mbxMax=(import-csv users.csv).count

$file = "office365$((get-date).tostring("yyyyMMdd")).csv"

$365master = get-credential ashour@mylab.onmicrosoft.com

New-O365ExchangeSession $365master # call the office365 remote connection function

do{   

#Load the list of users from a csv file limited with the pageSize 1000 starting from the line $Offset + 1

$mbxlist=@(import-csv users.csv|select-object -skip $Offset -First $PageSize)

"Start at offset $($Offset) till $($Offset+$PageSize)"

ForEach($mbx in $mbxlist)

       {

"Start Processing $($mbx.alias)"

$deviceinfo = Get-mobileDeviceStatistics -mailbox $mbx.alias | select @{name="mailbox"; expression={$_}},devicetype, devicemodel, devicefriendlyname, deviceos, lastsuccesssync

If ($deviceinfo -ne $null)

              { 

       If (Test-Path $file)

                     {

                        $mbx.alias + ”,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file –Append

           #You can use –Encoding Unicode if you deal with Unicode characters like Arabic

                     }

       Else

                     {

                $deviceinfo | Export-Csv $file -Encoding ASCII -notypeinformation

                     }

                $deviceinfo = $null

}

      }

"------------------------------"

#Increase the start point for the next chunk

$Offset+=$PageSize

#Call the office365 remote session function to close the current one and open a new session

New-O365ExchangeSession $365master

} while($offset -lt $mbxMax)

 

Disclaimer The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Comments (7)

  1. Sherif Seddik says:

    Thanks Ahmed for this wonderful script , I used it with a tenant that has more than 500 thousands users , and it works as it should !

  2. Anonymous says:

    Great

  3. MSExchanger says:

    Excellent as usual Ahmed.

    Would it be possible to post it on Exchange team blog ?

  4. PiotrWrona says:

    Hello,

    Well it still does not work for me.

    I'm getting the same error when it reaches 4500-6500 users and it's asking me for credentials again.

    Earlier when i didn't found you blog. I tried to use 'if' and close old session when users counter reached (-like "*000") and opening new one. But i still have same issues.

    I even tried with other settings in New-PSSessionOption but it didn't help as well.

    Any ideas what could be wrong ?

  5. Nick Gillott says:

    Nice script Ahmed – will save some admins a lot of time.

  6. Nelson Mwebaze says:

    This is good stuff. we need more sessions

Skip to main content