Family Notes: a Windows UWP Sample with Voice, Ink and Face recognition


If you watched the Build keynote this year, you would have seen a lot of emphasis on the Windows 10 features that focus on a more personal aspect of computing. This can sound kinda vague, but it boils down to making software which is easier to use, more intuitive and just plain smarter. After all, we've now got these incredibly powerful, always-computers computers around us all the time: from the phones in our pockets to the PCs on our desks, and it's unlikely we're even using 10% of their brains most of the time. As automagical services like deep learning engines start to become more common, we're really on the verge of making apps that we can interact with in entirely new ways. Think of Cortana, or Siri or Amazon's Alexa - we now just expect computers to understand us: I remember writing a programme (that's how old and British I am) on a Z80 based computer that would sample sounds and after thinking for a few minutes tell me if I'd said "hello" or "goodbye". Now I can say "Alexa, play some Pink Floyd" and it starts into "Shine on you crazy diamond" before I've had a chance to blow up my inflatable pig.

As a Windows developer, you've maybe got the itch to add these features to your own apps. Support for ink, voice commands and recognition, and super-smart stuff like face recognition. I know we - the folks in the Windows documentation team - did, and so we decided to write a sample app for you. Yesterday, we pushed it to GitHub for you to try out for yourself. And use the code.

VirtualFridge

 

Family Notes

The Family Notes app is a noticeboard style app, designed to run on a tablet PC sitting in your kitchen (or glued to your fridge) and be used by the entire family. Once you've added your family members to it, you can create and assign them little sticky notes. You can type these notes (old school), or dictate them or even ink them with a stylus. The app is also listening for voice commands so you can say "Read note" or "Add note for John", and so on.

Most ambitious of all, we've used the Microsoft Cognitive Services facial recognition service (you might have heard of its earlier name Project Oxford) to take images from the PC's webcam, and try to recognize anyone standing in front of it. So when I walk to up to the computer, the app "sees" me, and highlights the notes for me. At first it feels like magic, but then it just works and feels normal, like something computers should just be doing. We decided not to play advertising tailored to you at this time..

Here's a low-budget video clip of the app in action. I promise it's better than this video makes it look:

 

 

 How does it work?

The code sample is well documented, but we've also got a series of blog posts coming up on the Building Apps for Windows Blog in the few weeks that delve into the more interesting aspects and the challenges we faced (so you don't have to!). For example, if you want to know how to serialize ink, yeah, we worked that out. Or have voice commands and dictation in the same app? Nailed it.

Logo


Comments (3)

  1. Brad Morgan says:

    This is really good stuff!
    I find myself constantly hunting down the local app data users folder and purging it while debugging. Any advice for using SQL server to store the information versus local storage? I prefer Azure, but SQL-lite works also. I would like to install this on multiple systems all talking to the same data store.

  2. balasundar says:

    It is really useful for my project. Requesting you detail when face is authenticated would like to validate the face data in on premises data. How do I do that

    1. Currently the service requires the images be sent to MCS, so it's not possible to keep the data on-site. You might want to try contacting the Microsoft Cognitive Services team to see if that is going to change..

Skip to main content