How to await a command-line process, and capture its output
[This post is part of a series How to await a storyboard, and other things]
I want to invoke an external executable, and await until it’s finished, and get back its output. (In my case, the external executable is called “tidy.exe” – an excellent open-source utility for turning HTML into clean parseable XML.) To cut a long story short, here’s my final code. The rest of the article has explanations of how it works and why.
Async Function Attempt4Async() As Task
Dim client As New HttpClient
Dim html = Await client.GetStringAsync("https://blogs.msdn.com/lucian")
Dim tidy = Await RunCommandLineAsync("tidy.exe",
"-asxml -numeric -quiet --doctype omit",
html)
Dim xml = XDocument.Parse(tidy)
End Function
Async Function RunCommandLineAsync(cmd As String, args As String, stdin As String,
Optional cancel As CancellationToken = Nothing
) As Task(Of String)
Using p As New Process
p.StartInfo.FileName = cmd
p.StartInfo.Arguments = args
p.StartInfo.UseShellExecute = False
p.StartInfo.RedirectStandardInput = True
p.StartInfo.RedirectStandardOutput = True
p.StartInfo.RedirectStandardError = True
If Not p.Start() Then Throw New InvalidOperationException("no tidy.exe")
Using cancelReg = cancel.Register(Sub() p.Kill())
Dim tin = Async Function()
Await p.StandardInput.WriteAsync(stdin)
p.StandardInput.Close()
End Function()
Dim tout = p.StandardOutput.ReadToEndAsync()
Dim terr = p.StandardError.ReadToEndAsync()
Await Task.WhenAll(tin, tout, terr)
p.StandardOutput.Close()
p.StandardError.Close()
Await Task.Run(AddressOf p.WaitForExit)
Dim stdout = Await tout
Dim stderr = Await terr
If cancel.IsCancellationRequested Then
Throw New OperationCanceledException(stderr)
ElseIf String.IsNullOrEmpty(stdout) Then
Throw New Exception(stderr)
Else
Return stdout
End If
End Using
End Using
End Function
Process.Exited event
Stephen Toub wrote a very useful article “Await Anything” that I always use as my starting point. He included an example of awaiting for a process. That article used the Process.Exited event, and lead me to this code:
Async Function Attempt1Async() As Task
Dim p As New Process
p.StartInfo.FileName = "notepad.exe"
p.Start()
Dim tcs As New TaskCompletionSource(Of Integer)
p.EnableRaisingEvents = True
AddHandler p.Exited, Sub() tcs.TrySetResult(p.ExitCode)
If p.HasExited Then tcs.TrySetResult(p.ExitCode)
Await tcs.Task
End Function
Note the check for p.HasExited comes after adding the handler. If we didn’t do that, and if the process exited before adding the handler, then we’d never hear about it.
The code works great as is. But once I added in code to capture the output then I uncovered a race condition inside the Process.Exited event – it seems that the triple combination of ReadToEnd(), Dispose() and HasExited can cause the Exited event to fire more than once!
Repro: The following code will non-deterministically throw an exception because the process finishes and fires an event, then “await tcs.Task” finishes, then the event is fired a second time, and its attempt to read “p.ExitCode” is now invalid. The exception also seems to happen even if I RemoveHandler before EndUsing.
Async Function Attempt2Async() As Task
For i = 0 To 1000
Using p As New Process
p.StartInfo.FileName = "cmd.exe"
p.StartInfo.Arguments = "/c dir c:\windows /b"
p.StartInfo.UseShellExecute = False
p.StartInfo.RedirectStandardOutput = True
p.Start()
Dim stdout = Await p.StandardOutput.ReadToEndAsync()
'
Dim tcs As New TaskCompletionSource(Of Integer)
p.EnableRaisingEvents = True
AddHandler p.Exited, Sub() tcs.TrySetResult(p.ExitCode)
If p.HasExited Then tcs.TrySetResult(p.ExitCode)
Await tcs.Task
End Using
Next i
End Function
If the mechanism of communication is a wait-handle, then you need to block some thread
There might be clever ways to make Process.Exited work, but I didn’t care to risk them. So here’s my next attempt at awaiting until the process has finished. I tested it inside a while loop, and it doesn’t suffer from the same race condition as before.
Async Function Attempt3Async() As Task
Dim p As New Process
p.StartInfo.FileName = "notepad.exe"
p.Start()
Await p
End Function
<Extension> Function GetAwaiter(p As Process) As TaskAwaiter
Dim t As Task = Task.Run(AddressOf p.WaitForExit)
Return t.GetAwaiter()
End Function
Here’s an explanation for why the GetAwaiter() is written as it is. Process has a method p.WaitForExit() which blocks a thread until the process has finished. It also exposes an IntPtr property “Handle”, which is the raw win32 handle that will be signalled once the process has completed. I want to await until the process has finished, or equivalently until the handle is signaled. If the mechanism of communication is a wait handle, then I need to block some thread. Since I have to block some thread anyway, I might as well use WaitForExit(), which is slightly easier than the handle.
RunCommandLineAsync
I already started this article with the finished code, “RunCommandLineAsync”. Its code is subtle. Here’s an explanation.
Cancellation. If cancellation is requested, then I need to be able to terminate the process. I do that with “cancel.Register(Sub() p.Kill())”. As soon as cancellation is requested, this will invoke the lambda, which will kill the process. This will terminate the process abruptly, closing the handles cleanly (i.e. without exception). In this state I have no guarantee that stdout is complete. Therefore, I throw an OperationCancelledException. Note that even if cancellation is requested and the process gets killed, I still await until the process has completely finished before returning (rather than returning immediately and letting it clean up asynchronously in the background). I figure this is a more hygienic design.
Tin, Tout, Terr. The precise way that “RedirectStandardInput/Output/Error” will work depends on the internal details of the process we’re launching. All we can say with any certainty is that (1) the process might not finish until we close its StandardInput; (2) we might have to read data from StandardOutput/StandardError before we can write any more data into StandardInput; (3) we might have to write more data into StandardInput before we can read more data from StandardOutput/StandardError. Those constraints imply that the three tasks “tin/tout/terr” must all be launched concurrently, and then awaited using Task.WhenAll().
Await tout. The code does “Dim stdout = Await tout” even after we know that tout has finished (because we awaited Task.WhenAll). That’s fine. It merely means that the call to “Await tout” will complete immediately.
Async vs threads. A few years ago, I wrote similar code using threads instead of async. I'm happy that the async version is cleaner and more readable.
Wrapping it up
My ultimate goal behind this project was to be able to scrape web-pages. And I wanted to use VB’s XML-literals to make that easy. Here’s how I wrapped it up, to scrape all img tags from a page:
Async Function Attempt5Async(Optional cancel As CancellationToken = Nothing) As Task
Dim client As New HttpClient
Dim xml = Await client.GetXmlAsync("https://blogs.msdn.com/lucian", cancel)
For Each img In (From i In xml...<img> Select i.@src Distinct)
Console.WriteLine(img)
Next
End Function
<Extension>
Async Function GetXmlAsync(client As HttpClient, uri As String,
Optional cancel As CancellationToken = Nothing
) As Task(Of XDocument)
Using response = Await client.GetAsync(uri, cancel)
Dim html = Await response.Content.ReadAsStringAsync()
Dim tidy = Await RunCommandLineAsync("tidy.exe",
"-asxml -numeric -quiet --doctype omit", html, cancel)
Return XDocument.Parse(tidy)
End Using
End Function