WP8 Voice Commands: Phrase Lists

Today, I ’m going to show you how to extend the app we built in yesterday’s post, using something we call Phrase Lists. Using a Phrase List, we’ll be able to let users of our “Search On” application “Search on” 15 more sites, not just search on Amazon.com.

Here are a couple examples of how that’ll play out for users of the app…

To search on Amazon, it’ll work just like it did yesterday (with a small tweak to the prompt):

User: "Search On Amazon"
Phone: "Search Amazon for what?"
User: "Electronics" (or whatever the user wants)
Phone: "Searching Amazon for Electronics" (or whatever the user said she wanted)

But with the changes today, you’ll also be able to search on a bunch of sites. Here are the sites we’ll add today: Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or Zillow. Feel free to add more when you’re typing it up on your computer.

User: "Search On CNN"
Phone: "Search CNN for what?"
User: "Presidential Election Results”
Phone: "Searching CNN for Presidential Election Results"

… or …

User: "Search On Weather.com"
Phone: "Search Weather.com for what?"
User: "Seattle 10 day forecast”
Phone: "Searching Weather.com for Seattle 10 day forecast"

Sound Good? OK… Let’s get started.

First of all, if you haven’t built the app from yesterday, go do that now. Start here… 

Now, let’s update our Voice Command Definition file. We’re going to make 4 changes to the VCD file. 1.) We’re going to update the examples phrases that get displayed to users in the built in speech UX in WP8, 2.) We’re going to add a Phrase List  that will contain the list of sites that we want to be able to search, 3.) We’ll refer to that PhraseList from our searchSite <Command>, specifically, from our <ListenFor> elements, and, finally 4.) We’ll change the text feedback in the <Feedback> element to react dynamically to what the user said.

Here we go …

To update the example phrases, simply find the existing <Example> elements from the vcd.xml file, and replace them this:

<Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, 
         MSN, Twitter, Weather.com, YouTube, or Zillow</Example>

Now, let’s actually define the PhraseList. You can have up to 10 PhraseLists in your VoiceCommands CommandSet, and the PhraseLists must be after all the Command elements in the XML file. If there are more than 10, or they’re not in the right spot in the VCD file, Visual Studio should give you a red underlined squiggly line that will tell you what’s wrong when you hover over it. If for some reason you miss this, at runtime the call to InstallCommandSetsFromFileAsync will throw an exception. In our sample, we’re not trying to catch those, so if you don’t have a debugger hooked up, you’ll miss it.

We’ll name our PhraseList “siteToSearch”, and define it like this:

   1:  <PhraseList Label="siteToSearch">
   2:    <Item>Amazon</Item>
   3:    <Item>Bing</Item>
   4:    <Item>CNN</Item>
   5:    <Item>Dictionary.com</Item>
   6:    <Item>Ebay</Item>
   7:    <Item>Facebook</Item>
   8:    <Item>Google</Item>
   9:    <Item>Hulu</Item>
  10:    <Item>IMDB</Item>
  11:    <Item>Kayak</Item>
  12:    <Item>Linked In</Item>
  13:    <Item>MSN</Item>
  14:    <Item>Netflix</Item>
  15:    <Item>Twitter</Item>
  16:    <Item>Weather.com</Item>
  17:    <Item>YouTube</Item>
  18:    <Item>Zillow</Item>
  19:  </PhraseList>

Next, let’s update our Command, with ListenFor elements that reference this list.

The text inside a ListenFor element is used as a constraint for the recognizer. In yesterday’s example, we used a simple string, “Amazon”, as the content for our one and only ListenFor element. That meant that the user had to say “Search On”, which is the command prefix for our application, followed by “Amazon”. If you have several things you want the user to be able to say, you can do one, or both, of the following 2 things: have more than one <ListenFor> element in your command, and/or have your <ListenFor> element refer to a PhraseList by it’s Label.

Let’s see what that looks like for our updated application. Replace the <ListenFor> from yesterday, with the following:

   1:  <ListenFor>{siteToSearch}</ListenFor>
   2:  <ListenFor>{siteToSearch} dot com</ListenFor>

The first <ListenFor> element only contains a reference to the PhraseList. The second <ListenFor>, though, both references the same PhraseList, but it also has the words “dot com” after that PhraseList reference. What does this do? Well, it tells the recognizer that it’s OK for the user to say any one of the items inside the PhraseList with the siteToSearch Label, followed by the phrase “dot com”. Because we have both of these <ListenFor> elements, it essentially makes the “dot com” and optional phrase. We could have specified using an “optional” notation, using square brackets, like this:

   1:  <ListenFor>{siteToSearch} [dot com]</ListenFor>

Either of these two representations is essentially result in the same constraint being communicated to the speech recognizer.

OK. We’ve updated our examples, we’ve added our PhraseList for the sites to search, and we’ve updated our phrases for our command. Last thing we need to do is to update the <Feedback> element.

Similar to the <ListenFor> elements’ use of the PhraseList Label, we can substitute what was said from the PhraseList into the text feedback as well. Let’s update our <Feedback> element, and it should be pretty obvious what I mean:

   1:  <Feedback>Search on {siteToSearch} for what?</Feedback>

What that’ll do is this: Once the VoiceCommandService determines that one of your commands was recognized, it’ll take the text inside the corresponding <Feedback> element, and it’ll replace all occurrences of curly brace delimited PhraseLists with the text for the item that was actually recognized from the user speaking. That way, if I said “Search On Amazon”, the VoiceCommandService will replace the “{siteToSearch}” part of “Search on {siteToSearch} for what?” feedback string with “Amazon”. It’s a lot like C# string formatting, except using PhraseList Labels instead of order based parameter references, like 0, 1, 2.

OK. Now, our VCD file is all up to date. Great. Now let’s turn our attention to the code in MainPage.xaml.cs.

Let’s start off by adding a Dictionary of strings that we’ll use to look up the URL for the site we’ll be searching. And, if let’s have a backup way of searching if we can’t find that URL. Scroll down to the bottom of the MainPage class, and add the following:

   1:  private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";
   3:  private Dictionary<string, string> _siteUrlTemplateDictionary = new Dictionary<string, string>()
   4:  {
   5:      { "Amazon", "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" },
   6:      { "Bing", "http://www.bing.com/?q={0}" },
   7:      { "CNN", "http://www.cnn.com/search/?query={0}" },
   8:      { "Dictionary.com", "http://dictionary.reference.com/browse/{0}" },
   9:      { "Ebay", "http://www.ebay.com/sch/i.html?_nkw={0}" },
  10:      { "Facebook", "http://www.facebook.com/search/results.php?q={0}" },
  11:      { "Google", "https://www.google.com/search?hl=en&q={0}" },
  12:      { "Hulu", "http://www.hulu.com/#!search?q={0}" },
  13:      { "IMDB", "http://www.imdb.com/find?s=all&q={0}" },
  14:      { "Linked In", "http://www.linkedin.com/search/fpsearch?keywords={0}" },
  15:      { "MSN", "http://www.bing.com/search?scope=msn&q={0}" },
  16:      { "Twitter", "https://twitter.com/search/{0}" },
  17:      { "Weather", "http://www.weather.com/info/sitesearch?q={0}" },
  18:      { "YouTube", "https://www.youtube.com/results?search_query={0}" },
  19:      { "Zillow", "http://www.zillow.com/homes/{0}/" },
  20:  };

Now, let’s add a method that’ll do the lookup for us:

   1:  private string GetUrlTemplateFromSiteName(string siteName)
   2:  {
   3:      return _siteUrlTemplateDictionary.ContainsKey(siteName)
   4:          ? _siteUrlTemplateDictionary[siteName]
   5:          : string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{1}");
   6:  }

Great. Now let’s update our SearchSiteVoiceCommand method to figure out what site we’re actually searching, update how we speak the feedback based on the site name, use our new GetUrlTemplateFromSiteName method to find the right URL template, and then stick the search string into the URL template appropriately. When it’s all done, it should look like this:

   1:  private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)
   2:  {
   3:      string text = await RecognizeTextFromWebSearchGrammar();
   4:      if (text != null)
   5:      {
   6:          string siteName = queryString["siteToSearch"];
   7:          await Speak(string.Format("Searching {0} for {1}", siteName, text));
   9:          string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);
  10:          NavigateToUrl(string.Format(siteUrlTemplate, text, siteName));
  11:      }
  12:  }

One additional thing I’ll do, that we don’t absolutely need to at this point, but I think it’s good practice, is to make sure that we actually have a “siteToSearch” item in our query string dictionary in our HandleVoiceCommand method.

   1:  private void HandleVoiceCommand(IDictionary<string, string> queryString)
   2:  {
   3:      if (queryString["voiceCommandName"] == "searchSite" && 
   4:          queryString.ContainsKey("siteToSearch"))
   5:      {
   6:          SearchSiteVoiceCommand(queryString);
   7:      }
   8:  }

One more change, and then we can test it out.

Since we were only searching Amazon.com yesterday, it was probably OK for us to open up the Amazon.com home page if you started the Search On application from the Start menu, not by voice. But … It doesn’t make as much sense now, since we can search 15 different sites. Right?

So, I’ll update the code to search for blog posts by me about Windows Phone 8 on my MSDN blog instead. A bit self serving, I know, but … Hey … It’s my sample app, right? 🙂

First, I’ll define a class variable to hold the new location we’ll open in that situation:

   1:  private string _defaultUrlForNewNavigations = 
   2:      "http://m.bing.com/search?q=Rob's%20Rhapsody%20Windows%20Phone%208%20site:blogs.msdn.com";

And then, I’ll update the OnNavigatedTo method to use that new class variable (see line 11 in the snippet below):

   1:  protected override void OnNavigatedTo(NavigationEventArgs e)
   2:  {
   3:      base.OnNavigatedTo(e);
   5:      if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))
   6:      {
   7:          HandleVoiceCommand(NavigationContext.QueryString);
   8:      }
   9:      else if (e.NavigationMode == NavigationMode.New)
  10:      {
  11:          NavigateToUrl(_defaultUrlForNewNavigations);
  12:      }
  13:      else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)
  14:      {
  15:          NavigationService.GoBack();
  16:      }
  17:  }

Now we’re ready to try it out.

If you haven’t typed this in all along the way, you can scroll down below and copy/paste into Visual Studio and give it a go that way.

Press F5, and you should see a Bing Search for my blog. Yeah.

Now, on the phone, Press and hold the “Start” menu, and let’s try out our examples from above.

You: "Search On Amazon"

Phone: "Search Amazon for what?"

You: "Electronics" (or whatever you want)

Phone: "Searching Amazon for Electronics" (or whatever you said you wanted)

Great. That one still works! Let’s try out the others…

User: "Search On CNN"

Phone: "Search CNN for what?"

User: "Presidential Election Results”

Phone: "Searching CNN for Presidential Election Results"

Did that one work as well? Mine did. 🙂 Last one … we’ll try here today…

User: "Search On Weather.com"

Phone: "Search Weather.com for what?"

User: "Seattle 10 day forecast”

Phone: "Searching Weather.com for Seattle 10 day forecast"

Excellent. Well … except it’s going to rain this week. Bummer. But the speech part worked. If only I could say “Clouds, stop raining!!” I’m sure I can do the speech part of that one; it’s the actual stop raining part I don’t know how to do. Yet. 🙂

Well. That’s it for today. Hopefully, you now know how to use PhraseLists with the VoiceCommandService. And … Our “Search On” application is starting to actually become useful. I wonder how far we can take this …

Drop me some feedback below and let me know what you’d like to see, and I’ll try to work that into posts in the future.

Happy coding! … Oh yeah … I almost forgot … Here’s the full listing for the code, and the VCD file.


   1:  using System;
   2:  using System.Collections.Generic;
   3:  using System.Net;
   4:  using System.Windows;
   5:  using System.Windows.Navigation;
   6:  using Microsoft.Phone.Controls;
   7:  using Windows.Phone.Speech.VoiceCommands;
   8:  using Windows.Phone.Speech.Recognition;
   9:  using Windows.Phone.Speech.Synthesis;
  10:  using System.Threading.Tasks;
  11:  using Microsoft.Phone.Tasks;
  13:  namespace SearchOn
  14:  {
  15:      public partial class MainPage : PhoneApplicationPage
  16:      {
  17:          public MainPage()
  18:          {
  19:              InitializeComponent();
  21:              VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));
  22:          }
  24:          protected override void OnNavigatedTo(NavigationEventArgs e)
  25:          {
  26:              base.OnNavigatedTo(e);
  28:              if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))
  29:              {
  30:                  HandleVoiceCommand(NavigationContext.QueryString);
  31:              }
  32:              else if (e.NavigationMode == NavigationMode.New)
  33:              {
  34:                  NavigateToUrl(_defaultUrlForNewNavigations);
  35:              }
  36:              else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)
  37:              {
  38:                  NavigationService.GoBack();
  39:              }
  40:          }
  42:          private void HandleVoiceCommand(IDictionary<string, string> queryString)
  43:          {
  44:              if (queryString["voiceCommandName"] == "searchSite" && 
  45:                  queryString.ContainsKey("siteToSearch"))
  46:              {
  47:                  SearchSiteVoiceCommand(queryString);
  48:              }
  49:          }
  51:          private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)
  52:          {
  53:              string text = await RecognizeTextFromWebSearchGrammar();
  54:              if (text != null)
  55:              {
  56:                  string siteName = queryString["siteToSearch"];
  57:                  await Speak(string.Format("Searching {0} for {1}", siteName, text));
  59:                  string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);
  60:                  NavigateToUrl(string.Format(siteUrlTemplate, text, siteName));
  61:              }
  62:          }
  64:          private async Task<string> RecognizeTextFromWebSearchGrammar()
  65:          {
  66:              string text = null;
  67:              try
  68:              {
  69:                  SpeechRecognizerUI sr = new SpeechRecognizerUI();
  70:                  sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);
  71:                  sr.Settings.ListenText = "Listening...";
  72:                  sr.Settings.ExampleText = "Ex. \"electronics\"";
  73:                  sr.Settings.ReadoutEnabled = false;
  74:                  sr.Settings.ShowConfirmation = false;
  76:                  SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();
  77:                  if (result != null && 
  78:                      result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&
  79:                      result.RecognitionResult != null &&
  80:                      result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)
  81:                  {
  82:                      text = result.RecognitionResult.Text;
  83:                  }
  84:              }
  85:              catch 
  86:              {
  87:              }
  88:              return text;
  89:          }
  91:          private async Task Speak(string text)
  92:          {
  93:              SpeechSynthesizer tts = new SpeechSynthesizer();
  94:              await tts.SpeakTextAsync(text);
  95:          }
  97:          private void NavigateToUrl(string url)
  98:          {
  99:              WebBrowserTask task = new WebBrowserTask();
 100:              task.Uri = new Uri(url, UriKind.Absolute);
 101:              task.Show();
 102:          }
 104:          private string GetUrlTemplateFromSiteName(string siteName)
 105:          {
 106:              return _siteUrlTemplateDictionary.ContainsKey(siteName)
 107:                  ? _siteUrlTemplateDictionary[siteName]
 108:                  : string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{1}");
 109:          }
 111:          private string _defaultUrlForNewNavigations = "http://m.bing.com/search?q=Rob's%20Rhapsody%20Windows%20Phone%208%20site:blogs.msdn.com";
 112:          private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";
 114:          private Dictionary<string, string> _siteUrlTemplateDictionary = new Dictionary<string, string>()
 115:          {
 116:              { "Amazon", "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" },
 117:              { "Bing", "http://www.bing.com/?q={0}" },
 118:              { "CNN", "http://www.cnn.com/search/?query={0}" },
 119:              { "Dictionary.com", "http://dictionary.reference.com/browse/{0}" },
 120:              { "Ebay", "http://www.ebay.com/sch/i.html?_nkw={0}" },
 121:              { "Facebook", "http://www.facebook.com/search/results.php?q={0}" },
 122:              { "Google", "https://www.google.com/search?hl=en&q={0}" },
 123:              { "Hulu", "http://www.hulu.com/#!search?q={0}" },
 124:              { "IMDB", "http://www.imdb.com/find?s=all&q={0}" },
 125:              { "Linked In", "http://www.linkedin.com/search/fpsearch?keywords={0}" },
 126:              { "MSN", "http://www.bing.com/search?scope=msn&q={0}" },
 127:              { "Twitter", "https://twitter.com/search/{0}" },
 128:              { "Weather", "http://www.weather.com/info/sitesearch?q={0}" },
 129:              { "YouTube", "https://www.youtube.com/results?search_query={0}" },
 130:              { "Zillow", "http://www.zillow.com/homes/{0}/" },
 131:          };
 132:      }
 133:  }


   1:  <?xml version="1.0" encoding="utf-8"?>
   3:  <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">
   5:    <CommandSet xml:lang="en-US">
   7:      <CommandPrefix>Search On</CommandPrefix>
   8:      <Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or Zillow</Example>
  10:      <Command Name="searchSite">
  11:        <Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or Zillow</Example>
  12:        <ListenFor>{siteToSearch}</ListenFor>
  13:        <ListenFor>{siteToSearch} dot com</ListenFor>
  14:        <Feedback>Search on {siteToSearch} for what?</Feedback>
  15:        <Navigate Target="MainPage.xaml" />
  16:      </Command>
  18:      <PhraseList Label="siteToSearch">
  19:        <Item>Amazon</Item>
  20:        <Item>Bing</Item>
  21:        <Item>CNN</Item>
  22:        <Item>Dictionary.com</Item>
  23:        <Item>Ebay</Item>
  24:        <Item>Facebook</Item>
  25:        <Item>Google</Item>
  26:        <Item>Hulu</Item>
  27:        <Item>IMDB</Item>
  28:        <Item>Kayak</Item>
  29:        <Item>Linked In</Item>
  30:        <Item>MSN</Item>
  31:        <Item>Netflix</Item>
  32:        <Item>Twitter</Item>
  33:        <Item>Weather.com</Item>
  34:        <Item>YouTube</Item>
  35:        <Item>Zillow</Item>
  36:      </PhraseList>
  38:    </CommandSet>
  40:  </VoiceCommands>
Skip to main content