Find a bitmap on the VM screen

Continuing with my series on how to do interesting automation stuff with Hyper-V – today I want to show you how to use PowerShell and the AForge.NET library to locate a bitmap on a virtual machine screen.

For example – imagine that I had a test running in a Windows Server Core 2016 virtual machine:

source

And I wanted to be able to tell if PowerShell was running interactively – but I did not want to interfere with the guest operating system in anyway.  What could I do?  Well – I could get the screen and look for a bitmap that told me that PowerShell was running.  Something like this (enlarged to make it easier to view):

PShell

Fortunately – this is quite easy to do thanks to the handy AForge.NET libraries.  The result looks something like this:

If I run this script with a single PowerShell Window open I get this result:

Rectangle                        Similarity
---------                        ----------
{X=381,Y=371,Width=13,Height=12}          1

One perfect match at X=381, Y=371

If I open up two more PowerShell Windows and run this script I get this result:

Rectangle                        Similarity
---------                        ----------
{X=303,Y=522,Width=13,Height=12}          1
{X=535,Y=355,Width=13,Height=12}   0.993087
{X=74,Y=700,Width=13,Height=12}    0.993087

One perfect match – and two close matches.  The reason for the difference here is that the close matches are PowerShell Windows that do not have focus – so the graphic is slightly different.

One final point to make about this code sample: Hyper-V produces a 16bit RGB565 image for a screen capture.  Unfortunately the AForge.NET libraries do not accept this format.  So in my sample code you will see that I upsample into 24bit RGB in order to make everything work.

Cheers,
Ben