Language Understanding Intelligent Service (LUIS) offers a fast and effective way of adding language understanding to applications. With LUIS, you can use pre-existing, world-class, pre-built models whenever they suit your purposes -and when you need specialized models, LUIS guides you through the process of quickly building them. see http://www.luis.ai and watch this video to see how to start from scratch. Microsoft Cortana IngellienceSuite previously know as Microsoft Project Oxford, Language Understanding Intelligent Services LUIS is part of the Cortana Intelligence Suite bundle of cloud-based machine-intelligence tools. You can find everything from face detection and emotion recognition through to spell checking (try the demos from your browser).
In the following demo I’m going to be using the Cortana Intelligence Suite speech APIs and the Language Understanding Intelligent Service (LUIS) to create students favourite food. Yes all students love PIZZA.
What I think is great about Project Oxford is that these are the perfect toolset for creating amazing applications at HACKATHONS additionally all of the services are in preview right now and don’t cost anything. As this project is aimed at students getting going with Azure Services I wanted to focus on using most students have experience of so this demo will be using Python 3.5 and the Cortana Intelligence Suite library to access the APIs.
This will install the latest version (32-bit) from the Python homepage. CPython is the most commonly used interpreter and has the most up-to-date set of language features.
Creating our Online Pizza Shop
Step 1. Take the Order
Before integrating speech, we will start by making a pizza ordering script that works with the keyboard. Start with a blank Python file, write a welcome message, and then ask the customer for their order. Feel free to change the name to whatever you like.
print("Hello, welcome to Mario Pizza Shack") order = input("How may I help you?")
It’s always polite to repeat the order back to the customer, so they can correct you if you misheard.
print("You said:", order)
We’ll come back to actually understanding the order soon, but right now we need to thank our customer so they come back again. Poor customer loyalty is a great way to make sure we never become millionaire pizza tycoons. (Make sure you use the same name here as when you welcomed your customer, or else they will be very confused.)
print("Thank you for visiting Mario Pizza Shack")
That’s the basic structure! You can now test your pizza ordering system by pressing F5 (in Visual Studio) or running the script from PowerShell or the command prompt.
Step 2. Talk to your Customer using LUIS
We can very easily add both text-to-speech and speech-to-text functionality and interact directly with them. First, we need to install the projectoxford library from PyPI. The easiest way to do this is using pip, or through Visual Studio by simply clicking on the Python (global default) and selecting install python package
Install Via PIP
C:\Projects\MarioPizzaShack> pip install projectoxford Installing collected packages: projectoxford Running setup.py install for projectoxford ... done Successfully installed projectoxford-0.3.1
Once installed the Project Oxford library will be added to the project
Next, visit https://www.microsoft.com/cognitive-services/en-us/speech-api and subscribe to the speech API service. That will give you a “Primary Key” that you can save for later.
Finally, at the very top of our script, we need to import the speech API client, and then replace the normal print and input functions with ones that will use the microphone and speakers. You’ll need to paste your API key from above. You can also change the locale to match your own, which may improve speech recognition, but be aware that not all combinations of locale and gender will work. (Changing locale changes the language, so if you choose “es-GB” you’ll need to speak UK English and not English. if you used “es-MX” the application will speak Spanish)
from projectoxford.speech import SpeechClient sc = SpeechClient("PASTE-YOUR-KEY-HERE", gender='Male', locale='en-US') print = sc.print input = sc.input
Make sure you have your volume up, microphone ready, and run the script again. Instead of typing your responses, you can just speak after the beep to make an order.
Step 3. Understand the Order
So we now have a voice automated pizza sales assistant, the assistant simple reads text to speak and read back exactly what you said but the accuracy is pretty low. Another big issue issue is that you can actually order anything you like including crazy things which you wouldnt get from a Pizza Shop. So what we need to do next is add some intelligence to the application. To perform AI processing, this is where Project Oxford. LUIS takes simple examples of phrases and uses machine learning to create an service that extrapolates to whatever a user asks. For example, if we teach LUIS that “I’d like pizza with cheese and bacon” is a pizza order, LUIS can also identify other statements that look like pizza orders, and can extract information like the list of toppings. The more examples and more usage, the better it gets. For our pizza shop, you can download a predefined set of examples from here. This is a JSON file with statements like “can I have pizza with pineapple and bacon” and “sausage and pepperoni on my pizza please”, some non-pizza requests such as “I would like a sandwich with bacon, lettuce and tomato”, and it has the classifications already included. Importing JSON File Go to luis.ai/ and sign in with your Microsoft Account. Once you are in, click the “New App” button and select “Import Existing Application”. Download the Pizza.json file and then select the downloaded file in LUIS, then click Import. Feel free to enter an utterance of your own here, or simply click “Train” in the lower-left corner. When training is complete, click the “Publish” button in the upper-left and then “Publish web service” to get your URL.
Back in our Python script, we want to import the LuisClient class from projectoxford and pass in our URL (make sure that you include the “&q=” at the end of the URL).
from projectoxford.luis import LuisClient lc = LuisClient("YOUR-URL-GOES-HERE")
Now, after obtaining the order from the customer, we can call into LUIS to see what the customer is asking for. The three parameters returned from query are the intent, a list of entity names (in this case, the toppings), and a list of entity types (here we only have “Topping”, but you can support different types).
print("You said: ", order) intent, toppings, _ = lc.query(order)
If the intent is “Pizza” and we have at least one topping, we know what the customer has asked for. If a customer asks for something else, such as a sandwich or golf clubs, the intent will not be “Pizza” and so we will apologize. The speech module has a helpful join_and function that will make the list of toppings into something that can be read out loud.
from projectoxford.speech import join_and if intent == "Pizza" and toppings: print("You will be ordering a pizza with", join_and(toppings)) else: print("I am sorry, we only sell pizza here.") print("Thank you for visiting Mario Pizza Shack")
Using Microsoft Cortana Intelligence Suite previously Known as Microsoft Project Oxford and Python, you can very quickly create highly interactive applications that hear what you say, know what you mean, and can talk back to you.
If you want to keep extending the system, here are a few ideas you could try.
- Determine how much the pizza will cost, based on the toppings requested
- Add Pizza size Small, Large or inch
- Expand your empire by training LUIS to recognize sandwich orders
- Change the voice in the speech client (personally, I like