Building Arcade Game Player (first step in long path)
Article
For few months I have been thinking of building arcade game player. I mean that I would create application that would then play some old classic arcade game (i.e. Blues Brothers, Rick Dangerous, Super Mario Bros. etc.). Previously I have created applications that play puzzle games (i.e. Survo, MineSweeper) and since puzzle games are fairly straightforward I thought I'll take a bit challenging task. And of course all actions between my "player application" and the actual game should be done through user interface and not by tweaking the process memory (this time :-). It's definitely challenging task. So I thought that I'll start with baby steps and try to first solve the graphical user interface challenge.
Currently I'm thinking following approach: I need to take "snapshots" out of the game and then find the differences between last snapshot and current snapshot. From that I could then analyze the differences and probably get location of the character and all the nasty enemies more easily than I could using other methods. But of course that's not enough... I need to do other image recognition things in order to sort out other issues. I'll worry those things later :-)
Okay let's create Live Image Diff as I call it so that we can find the differences between frames. First step is to grab image from the foreground window. Here's code clip for that (I have this code in my Win32Helper -class):
public static readonly int SRCCOPY = 0xCC0020;[StructLayout(LayoutKind.Sequential)]public struct RECT{ public int left; public int top; public int right; public int bottom;}[DllImport("user32.dll")]public static extern IntPtr GetForegroundWindow();[DllImport("user32.dll")]public static extern bool GetClientRect(IntPtr hWnd, ref RECT lpRect);[DllImport("gdi32.dll")]public static extern bool BitBlt( IntPtr hdcDest, // handle to destination DC int nXDest, // x-coord of destination upper-left corner int nYDest, // y-coord of destination upper-left corner int nWidth, // width of destination rectangle int nHeight, // height of destination rectangle IntPtr hdcSrc, // handle to source DC int nXSrc, // x-coordinate of source upper-left corner int nYSrc, // y-coordinate of source upper-left corner System.Int32 dwRop // raster operation code );static public Bitmap GetSnapShot(){ IntPtr hWnd = GetForegroundWindow(); RECT rect = new RECT(); if (GetClientRect(hWnd, ref rect) == true) { int width = rect.right - rect.left; int height = rect.bottom - rect.top; Graphics graphWindow = Graphics.FromHwnd(hWnd); Bitmap bitmap = new Bitmap(width, height, graphWindow); Graphics graphFile = Graphics.FromImage(bitmap); IntPtr dcWindow = graphWindow.GetHdc(); IntPtr dcFile = graphFile.GetHdc(); BitBlt(dcFile, 0, 0, width, height, dcWindow, 0, 0, SRCCOPY); graphWindow.ReleaseHdc(dcWindow); graphFile.ReleaseHdc(dcFile); return bitmap; } return null;}
So now we have image (=.NET Bitmap) of the foreground application.
Now I want to compare previous image to the current snapshot and mark everything that has been changed with red color. Here's clip that does that:
using System.Drawing.Imaging;// ...Bitmap previousSnapshot;Bitmap currentSnapshot;BitmapData currentSnapshotCleanData;Rectangle rectangle; //...previousSnapshot = currentSnapshot;currentSnapshot = Win32Helper.GetSnapShot();if (currentSnapshot == null){ return;}if (previousSnapshot == null || previousSnapshot.Width != currentSnapshot.Width || previousSnapshot.Height != currentSnapshot.Height){ previousSnapshot = currentSnapshot.Clone() as Bitmap; rectangle = new Rectangle(0, 0, currentSnapshot.Width, currentSnapshot.Height);} //...BitmapData previousSnapshotData = previousSnapshot.LockBits( rectangle, ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);BitmapData currentSnapshotData = currentSnapshot.LockBits( rectangle, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);currentSnapshotCleanData = currentSnapshotData;IntPtr ptrPreviousSnapshot = previousSnapshotData.Scan0;IntPtr ptrCurrentSnapshot = currentSnapshotData.Scan0;int bufferSize = previousSnapshotData.Stride * previousSnapshotData.Height - 1;byte[] previousSnapshotBuffer = new byte[bufferSize];byte[] currentSnapshotBuffer = new byte[bufferSize];Marshal.Copy(ptrPreviousSnapshot, previousSnapshotBuffer, 0, previousSnapshotBuffer.Length);Marshal.Copy(ptrCurrentSnapshot, currentSnapshotBuffer, 0, currentSnapshotBuffer.Length);for (int x = 0; x < bufferSize - 4; x += 4){ if (previousSnapshotBuffer[x] != currentSnapshotBuffer[x] || previousSnapshotBuffer[x + 1] != currentSnapshotBuffer[x + 1] || previousSnapshotBuffer[x + 2] != currentSnapshotBuffer[x + 2] || previousSnapshotBuffer[x + 3] != currentSnapshotBuffer[x + 3]) { // This has been changed! Let's make it red: previousSnapshotBuffer[x] = 0; previousSnapshotBuffer[x + 1] = 0; previousSnapshotBuffer[x + 2] = 0xff; previousSnapshotBuffer[x + 3] = 0xff; }}Marshal.Copy(previousSnapshotBuffer, 0, ptrPreviousSnapshot, previousSnapshotBuffer.Length);previousSnapshot.UnlockBits(previousSnapshotData);currentSnapshot.UnlockBits(currentSnapshotData);// Now draw "previousSnapshot" to the screen:pictureBoxAtTheForm.Image = previousSnapshot;
On lines 10 to 22 I'll just grab current snapshot and try to make sure that my snapshots stay in sync. I need to do some checks since user actually could have changed the foreground window and my snapshots can be from different windows and therefore different sizes. If something goes wrong in this bad things will happen later.
On lines 40 to 53 I'll just check that is pixel at the current snapshot different from the one in previous snapshot. And if that is the case then I'll just fill the value to the buffer so that it makes the pixel red. I don't even try to make this code portable to different pixelformats etc. It's okay if it just runs on my machine with my display settings :-)
But the coolest thing is yet to come. I used method described in here to create Silverlight screencast from my application. I used IE in my demonstration of Live Image Diff. I browsed to my blog and then selected some text etc. in order to make changes to the screen. My application is running at the lower right corner and it displays differences in the user interface. Here is screenshot from that:
What are next steps in this one? Well I don't know... so if you have ideas how to proceed on this challenge then drop me email or post comment to this post. I would like to here your ideas.
