Intel just open sourced Stephen Hawking’s speech system and it’s a .NET 4.5 WinForms app that you can try for yourself

Disclaimer: This post does not constitute an endorsement of any Microsoft service, technology or product by Stephen Hawking. The purpose of this post is to discuss Intel’s communication platform for individuals with disabilities and it’s technical details. Pictures are used purely for informational and demonstrational purposes.

I’m typing this sentence with my face. And no, I didn’t somehow smash my face onto a keyboard with laser-like precision. I used Intel’s ACAT, or Assistive Context-Aware Toolkit, an open source platform developed in C# using .NET 4.5 and Visual Studio 2012 at Intel Labs to allow people with disabilities to communicate with ease, even in very constrained situations, like Stephen Hawking’s,

Stephen_Hawking_in_CambridgeAs we all know, the venerable physicist Professor Stephen Hawking is unable to talk as he is afflicted with ALS and thus relies on a computer system to communicate. In 2011, his condition was deteriorating so badly that he could best communicate at a rate of only 2 words per minute. He reached out to Alan Moore at Intel and asked if Intel could come up with new technology to help his plight.

Intel was happy to oblige. For 3 years, Intel Labs worked in close collaboration with Hawking to acutely address his needs and in January of 2014, they announced the first stable release that Hawking would use to replace his decades-old speech system. There was a near instant 10x improvement to all common tasks such as conducting a web search or opening a Word document. Fast-forward a few months and now Intel has open sourced the whole platform to allow researchers and hackers to broaden its use for people with disabilities of all backgrounds.

Exploring ACAT


Intel’s custom PC for Hawking, with the ACAT software in view.

When a friend sent Intel’s announcement to me, I couldn’t help but hunker down and immediately start tinkering with the ACAT software. Contrary to my initial expectations, it was very easy to install and use. It suffices to just run the installer on your PC and within moments, the ACAT suite is fully accessible to you.

The principle in which ACAT operates is very simple. There is essentially only one ‘switch’ and you can trigger a sequence of these switches to do anything. Hawking operates this switch with his cheek. An infrared beam from Hawking’s glasses tracks his cheek and a twitch from it breaks the beam and triggers the switch. With a webcam, we can emulate Hawking’s method of communication as well.


ACAT Vision detecting my face through the webcam and monitoring my cheek twitches.

A portion of the software called ACAT Vision detects your face and monitors your cheek for twitches. When it detects one as shown in the above picture, a switch is triggered. On screen you have the alphabet laid out in columns as well as some editing and navigation options, kind of like a virtual keyboard. There is a yellow box that flashes and alternates between the top and bottom halves of the laid out alphabet and options. You trigger the switch when the yellow cursor box includes the letter or option that you desire to use. Then the box shrinks to the size of a row of letters and goes through each row in the previously selected half and waits for you to twitch your cheek to trigger a switch and select the row. Once the row is selected, the box shrinks to the space of a letter and it goes through each letter in the previously selected row until you twitch to signal the correct letter.

This process is repeated until you have a few letters for the predictive word completion to choose a word for you. After a while of using the system, the world completion will start to understand the context of your phrases and will suggest words based on your previously typed words. For example, when Hawking types ‘t’, it offers to complete the word as ‘the’ and then follows up with a word like ‘black’ and then based on black it will suggest the word ‘hole’.

Screenshot (49)

The ACAT Talk app narrating my cheek twitched sentences, just like the real Hawking!

While the primary intention of ACAT was to allow Hawking to communicate, it does more than just that. It’s actually a comprehensive system that allows Hawking to control the entire operating system. If he selects the ‘M’ character on the bottom right of his menu, he access the menu, from where he can choose to access many options such as the ‘New File’ dialog or the ‘Launch Application’ dialog. From the ‘Launch Application’ dialog he can start a Google search or use Internet Explorer (though Intel should really set him up to use Edge). The principle of navigation is still the same, the yellow box cursor alternates on the visual keyboard and awaits cheek twitches.

Screenshot (50)

The ‘Launch Application’ menu. To select an application to launch, the corresponding number on the keyboard must be selected. For example, to access Notepad, 7 must be selected.

Once a desktop app is selected, a navigation context menu shows up. From there the relevant options appear to support the navigation of the app. For example, once ‘Google’ is selected, an Internet Explorer browser window pops up. The yellow box cursor cycles through the context menu. An option like ‘Search/URL’ can be chosen to then allow the browser bar to be used for searching or accessing URLs.

Screenshot (51)

Well, what are you waiting for? You can try all this out for yourself. Visit the software's Github release page to get the installer. Make sure to check out the user guide, where you can learn advanced navigation techniques like the radial laser scanner, To make a fork and incorporate your own contributions, check out the Github repository. Happy hacking! (or should I say Happy Hawking? Winking smile)

Comments (26)

  1. j says:

    Contrary to my initial expectations, it was very to install and use.

  2. Mittens says:

    Very amazing.

  3. Sanchita says:

    Glad to know all about it !!! …,very helpful !!!!!  

  4. Fred says:

    Is it easy to use with Portuguese instead of English?

  5. Víctor Ortega says:

    Hello Mansib, do you know if prediction text is available in Spanish in ACAT?

  6. Brian says:

    Is it only in English right now?

  7. CB says:

    Its Really interesting, but i get tue following Error in acat App and Talk.

    Fatal error. Error setting word prediction engine to [Presage Word Predictor]

  8. CB, you have to install Presage, it's on the github release page:…/releases

  9. @Brian, @Victor, it is only English for now.

  10. @Fred, it will take some work to use other languages with the system, but the predictive text engine, Presage, is multilingual and will work with Portuguese (or another longer).

  11. Bill says:

    Works well in Windows 10 x64.

    One cause of

      Fatal error. Error setting word prediction engine to [Presage Word Predictor]

    seems to be running it as an unprivileged user (you are… aren't you?!?!?!), you might need to launch with Admin privs.   Otherwise, the presage WCF service doesn't start.

  12. asking says:

    do you have 64bit OS ? because I have 32bit windows and I can't run ACAT vision because of incompatibility !

  13. yo says:

    It picks up the gestures but when i run Tryout it doesn't do anything, Vision still says its receiving input. What gives?

  14. Vicki says:

    I've taught A. T for years and am so glad to know that it is now available to all. Our school paid huge amounts of money for kids to communicate! At last a company that understands this problem. Kudos to you all!!!

  15. anon says:

    will this work on mac

  16. Myles Dear says:

    Could this system be triggered from an input from a 3d Intel RealSense camera + KinesicMouse software ?

  17. Anon says:

    I'm having the same issue as "yo" who comment on 8/22/15.  I followed the install directions exceedingly carefully, downloaded the user guide, and attempted multiple times but no matter what I do I cannot get Vision to interact with anything else.  Vision functions fine, running and accurately detecting facial twitches but Tryout or the full App simply refuse to respond to Vision what-so-ever.  Also, the user guide states that Vision will automatically close when you close the main App but it does not leading me to believe they simply aren't communicating with each other at all.  There are no error messages and googling my issue is netting me nothing but this page over and over.  Anyone else had or resolved this issue?

  18. SAGAR KUMAR says:

    Dear Mansib Rehman, thank you very much for sharing this with us.

    But there is a problem that when i open the release page it says "We are releasing a new version shortly. Please check back".As we don't know when they will have released it. So could someone please tell me another source, or send me source code & Executable files via my email address if you have, from which i can download it.

    My Email Address:

    I shall remain thankful to you!

  19. John says:

    I have the same problem as Yo and Anon. Has anyone solved this?  Mansib Rehman, can you help please?

  20. samalkah says:

    Same problem here. Look like Vision doesn't send anything to Acat or Tryout even if it detects gesture. I'm running Windows 7 family premium, what about the others who have the same problem ?

  21. samlakah says:

    Ok I just found out (but haven't tested it yet) that the problem has been reported in the troubleshooting guide here :…/acat_troubleshooting_guide_2.pdf

    It comes from the Windows fierwall which block a communication port between Vision and Acat/Tryout

  22. samalkah says:

    Well It's me again, I've just tested to disable windows firewall and avast and it still doesn't work…

  23. kishan kuman says:

    Hello Mansib, do you know if prediction text is available in Spanish in ACAT?

Skip to main content