Product and Part Identifier Bot using Custom Vision Service


In this post, we will build upon some of the capabilities that have been explored previously, including Microsoft’s Bot Framework, and the Cognitive Services Custom Vision Service. We will create a Bot that is capable of identifying a part or product from an organization’s Dynamics 365 product catalog, using an organization-specific image classifier that is powered by the Custom Vision Service.

 

By enabling part or product identification through an intuitive and mobile-accessible bot, we can facilitate such scenarios as:

  • Enabling field technicians to quickly identify an unknown part or product while on-site, and providing them instant access to documentation and specifications
  • Enabling customers to quickly identify a part or product to facilitate service requests or product-specific self-service

 

This 40 second video shows the functionality in action:

 

 

We will build on the sample code available in these two resources:

 

Pre-requisites

The pre-requisites for building and deploying our Product and Part Identifier Bot include:

  • An instance of Dynamics 365 for Customer Service (Online)
    • You can request a trial of Dynamics 365 for Customer Service here
  • In our Dynamics 365 organization, we need to have configured:
    • Products in our Products Catalog (ideally populated with product names, descriptions, product numbers, externally-accessible product image URLs, and links to product specifications)
  • Administrator credentials to access the O365 Admin Portal for our tenant that our Dynamics 365 instance is connected to
  • A Microsoft Azure subscription for application registration; A trial account will also work; See this previous post for a walkthrough of how to obtain Dynamics 365 Web API credentials
  • The Dynamics 365 Software Development Kit (SDK)
  • A Microsoft Account, for signing into the Custom Vision Service (https://customvision.ai)
  • Visual Studio 2015 or higher
  • Microsoft .NET Framework 4.6
  • The Bot Builder SDK for .NET
  • The Bot Framework Emulator (for Windows)
  • The Bot Connector Visual Studio Project Template
    • Save the downloaded zip file to your Visual Studio 201X templates directory, which is typically found in %USERPROFILE%\Documents\Visual Studio 201X\Templates\ProjectTemplates\Visual C#

 

Setting Up our Custom Vision Service Project

The first thing we will do is to set up our own Custom Vision Service project. The process of creating a Custom Vision Service project has been covered in a previous post, so to create our project, we can follow the steps outlined here (see Setting Up our Custom Vision Service Project).

After we have created, trained and tested our classifier, we can obtain the necessary credentials from the Custom Vision site to call the Prediction API by selecting the Performance tab, and clicking Prediction URL.

 

predictionapiendpoint2

 

We can extract our Project GUID from the URL, and our Prediction Key from the Prediction-Key header value.

 

Obtaining our Dynamics 365 Web API Credentials

In order to query our Dynamics 365 Product Catalog, we will also need to have the required credentials to authenticate our requests via the Web API.

In addition to an O365 username and password with sufficient D365 privileges to read our Product Catalog, we require:

  • Our Application Client ID
  • Our OAuth 2.0 Authorization Endpoint

 

See this previous post for a walkthrough of how to obtain these credentials (starting at Obtaining our CRM Web API Credentials).

 

Building our Bot

We will now start putting together the code for our bot, using steps found in the Create a bot with the Bot Builder SDK for .NET quickstart.

We Open Visual Studio, and create a new project using the Bot Application template that we downloaded as a part of our pre-requisites:

 

 

In our Project Properties, on the Application tab, we specify .NET Framework 4.6 as the target framework.

 

Installing Packages and Adding References

We need install several packages that we will leverage as a part of our bot code, and ensure our project is referencing the latest version of the Bot Builder SDK. We can do this via NuGet:

 

We also add references to the following assemblies, which we will use in our requests to the Custom Vision Service and Web API:

  • System.Net
  • System.Net.Http
  • System.Runtime.Serialization

 

We will place the Custom Vision Service credentials and Dynamics 365 Web API Credentials into our Web.config settings. We open the file, and add our credentials as appSettings as shown below, to make them accessible throughout our project:

<?xml version="1.0" encoding="utf-8"?>
<!--
  For more information on how to configure your ASP.NET application, please visit
  http://go.microsoft.com/fwlink/?LinkId=301879
  -->
<configuration>
  <appSettings>
    <!-- update these with your BotId, Microsoft App Id and your Microsoft App Password-->
    <add key="BotId" value="Product ID Bot" />
    <add key="MicrosoftAppId" value="Your Bot App ID here" />
    <add key="MicrosoftAppPassword" value="Your App Password Here" />
    <!-- update these with your Custom Vision Service Prediction Key and Project GUID-->
    <add key="visionPredictionKey" value="Your Vision Prediction Key Here" />
    <add key="visionProjectGuid" value="Your Vision Project GUID Here" />
    <!-- update these with your Dynamics and Azure AD credentials-->
    <add key="dynamicsUsername" value="youruser@yourorg.onmicrosoft.com" />
    <add key="dynamicsPassword" value="yourpassword" />
    <add key="dynamicsUri" value="https://yourorg.crm.dynamics.com" />
    <add key="adOath2AuthEndpoint" value="https://login.microsoftonline.com/yourEndpointHere/oauth2/authorize" />
    <add key="adClientId" value="Your AD Client ID Here" />
  </appSettings>
....

 

Adding a Data Contract for the Custom Vision Service

To facilitate interacting with the Custom Vision Service, we add a new item to our project: a Visual C# class which we will name CustomVision.JSON.cs. We will populate this file with JSON data contracts for the data that comes back in the response from the sample code provided for the Prediction API.

using System.Runtime.Serialization;

namespace CustomVision.JSON
{
    [DataContract]
    public class Response
    {
        [DataMember(Name = "Predictions")]
        public Prediction[] Predictions { get; set; }

        [DataMember(Name = "Id")]
        public string Id { get; set; }

        [DataMember(Name = "Project")]
        public string Project { get; set; }

        [DataMember(Name = "Iteration")]
        public string Iteration { get; set; }

        [DataMember(Name = "Created")]
        public string Created { get; set; }

    }
    [DataContract]
    public class Prediction
    {
        [DataMember(Name = "TagId")]
        public string TagId { get; set; }

        [DataMember(Name = "Tag")]
        public string Tag { get; set; }

        [DataMember(Name = "Probability")]
        public double Probability { get; set; }

    }

}

 

Querying Dynamics 365 via the Web API

We will also create a separate class to hold our utility CRMWebAPIRequest method, used to issue our Web API requests to Dynamics 365, as done in previous posts:

  • The method accepts not only a request string, but also an HttpContent request payload, and request type indicator, to allow for not only querying, but creation of records in Dynamics 365
  • Dynamics 365 Web API credentials are retrieved from the Web.config
  • Depending on whether the request type, there is conditional logic to issue either a GET request when we are retrieving Dynamics data, or a POST request with content, if we are creating a Dynamics record

using Microsoft.IdentityModel.Clients.ActiveDirectory;
using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Web.Configuration;

namespace CustomVisionBot
{
    public class Utilities
    {
        // Take a CRM Web API request string (and optionally json data), issue it, and return the HTTP Response:
        public static async Task<HttpResponseMessage> CRMWebAPIRequest(string apiRequest, HttpContent requestContent, string requestType)
        {
            AuthenticationContext authContext = new AuthenticationContext(WebConfigurationManager.AppSettings["adOath2AuthEndpoint"], false);
            UserCredential credentials = new UserCredential(WebConfigurationManager.AppSettings["dynamicsUsername"], WebConfigurationManager.AppSettings["dynamicsPassword"]);
            AuthenticationResult tokenResult = authContext.AcquireToken(WebConfigurationManager.AppSettings["dynamicsUri"], WebConfigurationManager.AppSettings["adClientId"], credentials);

            HttpResponseMessage apiResponse;

            using (HttpClient httpClient = new HttpClient())
            {
                httpClient.BaseAddress = new Uri(WebConfigurationManager.AppSettings["dynamicsUri"]);
                httpClient.Timeout = new TimeSpan(0, 2, 0);
                httpClient.DefaultRequestHeaders.Add("OData-MaxVersion", "4.0");
                httpClient.DefaultRequestHeaders.Add("OData-Version", "4.0");
                httpClient.DefaultRequestHeaders.Accept.Add(
                    new MediaTypeWithQualityHeaderValue("application/json"));
                httpClient.DefaultRequestHeaders.Authorization =
                    new AuthenticationHeaderValue("Bearer", tokenResult.AccessToken);

                if (requestType == "retrieve")
                {
                    apiResponse = await httpClient.GetAsync(apiRequest);
                }
                else if (requestType == "create")
                {
                    apiResponse = await httpClient.PostAsync(apiRequest, requestContent);
                }
                else if (requestType == "update")
                {
                    HttpRequestMessage message = new HttpRequestMessage();
                    message.Content = requestContent;
                    message.Method = new HttpMethod("PATCH");
                    Uri baseUri = new Uri(WebConfigurationManager.AppSettings["dynamicsUri"]);
                    Uri myFullUri = new Uri(baseUri, apiRequest);
                    message.RequestUri = myFullUri;
                    apiResponse = await httpClient.SendAsync(message);
                }
                else
                {
                    apiResponse = null;
                }
            }
            return apiResponse;
        }
    }
}

 

Next, we will adapt our MessagesController.cs class to be able to initiate our main dialog with the user. We replace the default Echo Bot code with code that will initiate a new Dialog using a VisionDialog class that we will create next:

using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using Microsoft.Bot.Connector;
using Microsoft.Bot.Builder.Dialogs;

namespace CustomVisionBot
{
    [BotAuthentication]
    public class MessagesController : ApiController
    {
        /// <summary>
        /// POST: api/Messages
        /// Receive a message from a user and reply to it
        /// </summary>
        public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
        {
            if (activity.Type == ActivityTypes.Message)
            {
                await Conversation.SendAsync(activity, () => new VisionDialog());
            }
            else
....

 

Next, we add a new class to our project, which we call VisionDialog.cs. This class will manage the interactions with the Custom Vision Service and Dynamics 365 to identify our product.

The code in the class will:

  • Present the user with the option to upload an image for identification using the StartingOptionsMessage method (Note: Options to allow dynamic credential setting and viewing are included as placeholder options)
  • If the user opts to upload an image, the AfterImageProvided method will
    • Validate that an attachment has been included with the message
    • Use HttpClient to retrieve the image as a byte array, using a JwtToken if necessary
    • Issue a request to the Custom Vision Service, with the appropriate credentials
    • Serialize the response using the CustomVision.JSON.cs data contract
    • If a prediction is returned from the Custom Vision Service, a request will be made to the Dynamics 365 Web API to retrieve a matching product from the Catalog
    • If a matching product is found in the catalog, a message with a Hero Card will be returned to the user, including product details, and a link to product specifications, if available

using System;
using System.Linq;
using System.Net.Http;
using Microsoft.Bot.Connector;
using Microsoft.Bot.Builder.Dialogs;
using Newtonsoft.Json.Linq;
using System.Threading.Tasks;
using System.Collections.Generic;
using System.Net.Http.Headers;
using System.Runtime.Serialization.Json;
using System.Web.Configuration;

namespace CustomVisionBot
{
    // As per the Dialogs model example in the Bot Builder docs,
    //  we add a class to represent our conversation:
    [Serializable]
    public class VisionDialog : IDialog<object>
    {

        private const string IdentifyOption = "Identify a Part or Product";
        private const string ViewConfigsOption = "View Current Configuration";
        private const string SetConfigsOption = "Set Configuration";

        public void StartingOptionsMessage(IDialogContext context)
        {
            // Present the options to the user, with a continuation back to the appropriate handler:
            PromptDialog.Choice(context, AfterChoiceAsync,
                new PromptOptions<string>("How can I help you?", null, null,
                new List<string>() { IdentifyOption, ViewConfigsOption, SetConfigsOption }, 3, null));
        }

        // Start the dialog:
        public async Task StartAsync(IDialogContext context)
        {
            context.Wait(MessageReceivedAsync);
        }

        // Handle an initial message from the user
        public async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> argument)
        {
            // Present the options to the user, with a continuation back to the appropriate handler:
            StartingOptionsMessage(context);
        }

        // handler for the initial user selection:
        public async Task AfterChoiceAsync(IDialogContext context, IAwaitable<string> argument)
        {
            var option = await argument;

            // Handle user selection as appropriate:
            if (option == IdentifyOption)
            {
                {
                    // Prompt user to upload an image:
                    await context.PostAsync("Please take or upload a photo of the item.");
                    context.Wait(AfterImageProvided);
                }
            }
            else
            {
                await context.PostAsync("This demo bot has not been configured for that yet.");
                StartingOptionsMessage(context);
            }

        }

        private async Task AfterImageProvided(IDialogContext context, IAwaitable<IMessageActivity> result)
        {

            // retrieve the Custom Vision credentials:
            string predictionKey = WebConfigurationManager.AppSettings["visionPredictionKey"];
            Guid projectGuid = new Guid(WebConfigurationManager.AppSettings["visionProjectGuid"]);

            try
            {

                // Retrieve user message, and ensure it has an attachment:
                var message = await result;
                if (message.Attachments != null && message.Attachments.Any())
                {
                    var attachment = message.Attachments.First();
                    using (HttpClient httpClient = new HttpClient())
                    {
                        // Skype & MS Teams attachment URLs are secured by a JwtToken, so we need to pass the token from our bot.
                        if ((message.ChannelId.Equals("skype", StringComparison.InvariantCultureIgnoreCase) || 
                            message.ChannelId.Equals("msteams", StringComparison.InvariantCultureIgnoreCase))
                            && new Uri(attachment.ContentUrl).Host.EndsWith("skype.com"))
                        {
                            var token = await new MicrosoftAppCredentials().GetTokenAsync();
                            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                        }

                        // retrieve our image, and create a byte array:
                        byte[] imageByteArray = await httpClient.GetByteArrayAsync(attachment.ContentUrl);

                        // Create our request client:
                        var client = new HttpClient();

                        // Add a request header with our Custom Vision Prediction Key:
                        client.DefaultRequestHeaders.Add("Prediction-Key", predictionKey);


                        // Construct Prediction URL, using Project GUID:
                        string url = "https://southcentralus.api.cognitive.microsoft.com/customvision/v1.0/Prediction/"
                            + projectGuid.ToString() + "/image?";

                        // Instantiate response:
                        HttpResponseMessage response;

                        using (var content = new ByteArrayContent(imageByteArray))
                        {
                            // Set content type, and retrieve response:
                            content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                            response = client.PostAsync(url, content).Result;

                            using (var stream = response.Content.ReadAsStreamAsync().Result)
                            {

                                // Serialize our response with data contract:
                                DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(CustomVision.JSON.Response));
                                CustomVision.JSON.Response visionJsonResponse = ser.ReadObject(stream) as CustomVision.JSON.Response;

                                // Ensure we have some results from the Custom Vision Service:
                                if (visionJsonResponse != null && visionJsonResponse.Predictions != null && visionJsonResponse.Predictions.Length > 0)
                                {
                                    // Build a D365 Web API request to retrieve a matching product from the catalog:
                                    HttpResponseMessage productResponse = await Utilities.CRMWebAPIRequest("api/data/v8.2/products?" +
                                        "$select=name,productnumber,producturl,description,new_externalimageurl&" +
                                        "$top=1&$filter=name eq '" + visionJsonResponse.Predictions[0].Tag + "'",
                                        null, "retrieve");

                                    // Check to make sure our request was successful:
                                    if (productResponse.IsSuccessStatusCode)
                                    {

                                        // Read our response into a JSON Array:
                                        string myString = productResponse.Content.ReadAsStringAsync().Result;
                                        JObject productResults =
                                            JObject.Parse(productResponse.Content.ReadAsStringAsync().Result);
                                        JArray items = (JArray)productResults["value"];


                                        // Ensure we have some results:
                                        if (items.Count > 0)
                                        {

                                            // format probability for presentation:
                                            string formattedProbability = Math.Round((visionJsonResponse.Predictions[0].Probability * 100), 1).ToString() + "%";

                                            // prepare message back to user:    
                                            IMessageActivity msgProductReply = context.MakeMessage();
                                            msgProductReply.Type = ActivityTypes.Message;
                                            msgProductReply.TextFormat = TextFormatTypes.Markdown;

                                            // prepare rich card, showing product image, and button to see specs:
                                            if ((string)items[0]["producturl"] != null && (string)items[0]["new_externalimageurl"] != null)
                                            {

                                                List<CardImage> cardImages = new List<CardImage>();
                                                cardImages.Add(new CardImage(url: (string)items[0]["new_externalimageurl"]));
                                                HeroCard actionCard = new HeroCard()
                                                {

                                                    Title = visionJsonResponse.Predictions[0].Tag + " (" + (string)items[0]["productnumber"] + ")",
                                                    Subtitle = (string)items[0]["description"] + "Identified with probability of " + formattedProbability,
                                                    Images = cardImages,
                                                    Buttons = {
                                                        new CardAction() { Title = "View Product Specs", Type = ActionTypes.OpenUrl, Value = (string)items[0]["producturl"] },
                                                        new CardAction() { Title = "Done", Type = ActionTypes.ImBack, Value = "Done" }
                                                    }
                                                };

                                                Attachment actionAttachment = actionCard.ToAttachment();
                                                msgProductReply.Attachments.Add(actionAttachment);

                                            }

                                            // Post message::
                                            await context.PostAsync(msgProductReply);
                                            context.Wait(MessageReceivedAsync);
                                        }
                                        else
                                        {
                                            // Inform user and prompt them to try again:
                                            await context.PostAsync("I wasn't able to identify a product.");
                                            StartingOptionsMessage(context);
                                        }
                                    }
                                }
                                else
                                {
                                    // Inform user and prompt them to try again:
                                    await context.PostAsync("I wasn't able to identify a product.");
                                    StartingOptionsMessage(context);
                                }
                            }
                        }
                    }

                }
                else
                {

                    // Prompt user to take a photo:
                    await context.PostAsync("Please take a photo of the item");
                    context.Wait(AfterImageProvided);
                }
            }
            catch (Exception e)
            {
                string reply = "Sorry, something went wrong.  Please try again. Details: " + e.ToString();
                await context.PostAsync(reply);
                context.Done<bool>(true);
            }

        }

    }
}

 

We are now ready to build and test our bot. As before, we press F5 in Visual Studio, which will Start Debugging. Once debugging has started, an Internet Explorer window should pop up, containing your bot endpoint information.

We also launch our Bot Framework Emulator which we downloaded earlier. We take the URL from the IE pop-up address bar, append /api/messages, and enter this in the URL field of the emulator. We can use the MicrosoftAppId and MicrosoftAppPassword from the application template Web.config file, and add them in the appropriate inputs in the emulator as well.

We can now test the identification of a product image through our bot emulator:

 

 

Our bot is ready to be published to Microsoft Azure, registered with the Bot Framework, and made available across channels. If you wish to walk through the deploying of your bot from Visual Studio, see here.

Comments (0)

Skip to main content