Rich messages with Bot Framework (Rich text, Image, Card, etc)


In the previous post, I explained the rest style (raw) messaging format in Microsoft Bot Framework.
In this blog post I will show you the various rich and functional chat bot messages using several examples. These capabilities are almost introduced by Microsoft Bot Framework version 3.

Notice : As I explained in the previous post, the required attributes of json format in the REST (HTTP) is different at the channel (Skype, Facebook, etc). In the following examples I'm using the Skype channel.

Rich Text

You can use not only the plain text but the markdown. (The default message setting is markdown.)
For example, when you send the following HTTP request from your bot, the message is displayed in the chat app like following image (screenshot).

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "# Welcome to **Botland**\n\nPlease visit [my blog](https://blogs.msdn.microsoft.com/tsmatsuz).\n\n---\n\nThis is a test."
}

Of course, you can use this markdown in other channels (Slack, Facebook, etc), if the channel supports this types of the rich text. For example, the following screenshot is the result of the e-mail channel.
If you're using channels which doesn't support the rich text (like Facebook Messenger Bot, etc), this markdown doesn't work.

If the channel is Skype, you can also use the XML text (i.e, HTML markup) as follows.
Note that the supported tag is only <b/>, <i/>, <u/>, <s/>, and <a/>.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Welcome to <b>Botland</b>. Please visit <a href=\"https://blogs.msdn.microsoft.com/tsmatsuz\">my blog</a>."
}

You can also use the special <ss/> tag for displaying emoji (emoticons) in Skype.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "<ss type =\"wink\">;)</ss>"
}

For details, please see the official document "Bot Framework SDK - Activities".

Attachment

You can attach the media (image/audio/video/file) to your activities using a link to the content.

First I show you the very simple example of the attachment. (As I show you later, you can use more rich expression using the attachment.) The following is the image attachment example.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "This is test attachment.",
  "attachments": [
    {
      "contentType": "image/jpg",
      "contentUrl": "http://mydeploy.azurewebsites.net/matsu.jpg",
      "name": "matsu.jpg"
    }
  ]
}

If you attach the binary, the attachment is shown as follows. (This is the Skype screenshot.)

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "This is test attachment.",
  "attachments": [
    {
      "contentType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "contentUrl": "http://myfiles01.blob.core.windows.net/sample/test01.xlsx",
      "name": "test01.xlsx"
    }
  ]
}

Card

If the channel (Skype, Slack, Facebook, etc) is having the button, template, or some sort of these capabilities, you can use the Card as the attachment in Microsoft Bot Framework.
For example, the Skype and Facebook is having this kind of rich expressions. (see "Facebook Messenger Platform - Button Template") And you can send the following HTTP request using the Card attachment.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "attachmentLayout": "list",
  "text": "",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "text": "What kind of sandwich would you like on your sandwich? ",
        "buttons": [
          {
            "type": "imBack",
            "title": "BLT",
            "value": "1"
          },
          {
            "type": "imBack",
            "title": "Black Forest Ham",
            "value": "2"
          },
          {
            "type": "imBack",
            "title": "Buffalo Chicken",
            "value": "3"
          }
        ]
      }
    }
  ]
}

If you're using Skype Bot, the message is displayed as follows.

If it's Facebook Messenger Bot, the message is displayed as follows.

If you send to e-mail channel, the message is displayed as follows.

If the user pushes this button in the Card, the selected value is replied to the bot as the text message. It's same like the user's text input.
These capabilities (button, template, etc) is different at each channel (Skype, Facebook, etc). But the Microsoft Bot Framework converts the expression to fit to each channel, and the developer doesn't need to care about the differences between channels.
For example, if the Facebook channel is used, the Microsoft Bot Framework converts to the button template in Facebook Messenger Bot. (see "Facebook Messenger Platform - Button Template" for details.) If the e-mail channel is used, the Microsoft Bot Framework converts to the "mailto:" alink with several parameters.

More advanced messages by Card

I show you more rich expression using the Card capability.

First, you can use the 3 types of Card: Hero Card, Thumbnail Card, and Receipt Card. (Actually, there's 4 types of Cards including the SignIn Card. I will show you the SignIn Card in the next post.)
Let's see how it looks like.

Hero Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a hero card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "I'm a hero card",
        "subtitle": "Please visit my site.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/matsu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

Thumbnail Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a thumbnail card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.thumbnail",
      "content": {
        "title": "I'm a thumbnail card",
        "subtitle": "Please visit my site.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/matsu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

Receipt Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a receipt card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.receipt",
      "content": {
        "title": "I'm a receipt card",
        "items": [
          {
            "title": "Sushi",
            "subtitle": "2 piece",
            "image": {
              "url": "https://mydeploy.azurewebsites.net/sushi.png"
            },
            "price": "16.25",
            "quantity": "1"
          },
          {
            "title": "Tenpura",
            "subtitle": "1 dish",
            "image": {
              "url": "https://mydeploy.azurewebsites.net/tenpura.jpg"
            },
            "price": "34.50",
            "quantity": "2"
          }
        ],
        "total": "275.25",
        "tax": "27.52",
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

For the "type" attribute in the Card, you can use the following values for your needs. (see "Bot Framework SDK - Attachments, Cards and Actions")
Note that the supported values is different at each channel. For example, the Skype supports "imBack", "openUrl", "call", "showImage", and "signin" now.

imBack Text of message which client will sent back to bot as ordinary chat message. All other participants will see that was posted to the bot and who posted this.
postBack Text of message which client will post to bot. Client applications will not display this message.
openUrl URL to be opened in the built-in browser.
call Destination for a call in following format: "tel:123123123123"
playAudio playback audio container referenced by URL
playVideo playback video container referenced by URL
showImage show image referenced by URL
downloadFile download file referenced by URL
signin OAuth flow URL. This is almost used in the SignIn Card. (Next time I will explain this capability.)

For example, when you send the following HTTP request, the call button (which calls to some Skype user) is displayed. If the user push this button, the Skype call to tsuyoshi.matsuzaki starts.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "text": "Welcome to our help desk !",
        "buttons": [
          {
            "type": "call",
            "title": "Call to tsuyoshi.matsuzaki",
            "value": "skype:tsuyoshi.matsuzaki"
          }
        ]
      }
    }
  ]
}

You can set the multiple cards in one message. In such a case, each Card is displayed vertically (as "list") by default.
If you want to show by horizontal carousel style, you can specify "carousel" style in "attachmentLayout" attribute.
Let's see how it looks like.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "attachmentLayout": "carousel",
  "text": "Please select your food.",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Sushi",
        "subtitle": "Very fresh shrimp or tuna.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/sushi.png"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "sushi,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "sushi,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "sushi,3+"
          }
        ]
      }
    },
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Tenpura",
        "subtitle": "Japanese first-class vegitables.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/tenpura.jpg"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "tenpura,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "tenpura,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "tenpura,3+"
          }
        ]
      }
    },
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Tofu",
        "subtitle": "Super healthy food condensed by soy milk.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/tofu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "tofu,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "tofu,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "tofu,3+"
          }
        ]
      }
    }
  ]
}

For details about Cards, please see the official document "Bot Framework SDK - Attachments, Cards and Actions".

Custom Channel Message - Rich messages for some specific channel

If you want to use the channel specific expression, you can use the "channelData" attribute instead of "attachments" attribute.

For example, if you want to use the flight update template in the Facebook Messenger Bot Platform, you send the following HTTP request.
Of course, this message is supported only in the Facebook channel.

POST https://facebook.botframework.com/v3/conversations/1245513475467461-737400853067333/activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "from": {
    "id": "737400853067333",
    "name": "mytestbot01"
  },
  "recipient": {
    "id": "1245513475467461",
    "name": "Tsuyoshi Matsuzaki"
  },
  "text": "This is test",
  "channelData": {
    "notification_type": "NO_PUSH",
    "attachment": {
      "type": "template",
      "payload": {
        "template_type": "airline_update",
        "intro_message": "Your flight is delayed",
        "update_type": "delay",
        "locale": "en_US",
        "pnr_number": "CF23G2",
        "update_flight_info": {
          "flight_number": "KL123",
          "departure_airport": {
            "airport_code": "SFO",
            "city": "San Francisco",
            "terminal": "T4",
            "gate": "G8"
          },
          "arrival_airport": {
            "airport_code": "AMS",
            "city": "Amsterdam",
            "terminal": "T4",
            "gate": "G8"
          },
          "flight_schedule": {
            "boarding_time": "2015-12-26T10:30",
            "departure_time": "2015-12-26T11:30",
            "arrival_time": "2015-12-27T07:30"
          }
        }
      }
    }
  }
}

For details about the custom channel message, please see the official document "Bot Framework SDK - Configuring Channels".

 

When you're using SDK (.NET), you can use the encapsulated object (class) called "Dialog" or "FormFlow" and this object (class) creates the previous messages internally if possible. These are the very high-level object. (You can also use the primitive elements using SDK.)
But, please remember that these essential elements which I explained here are working in the bottom of APIs.

Please enjoy your Bot development.

Comments (9)

  1. Puneet Singh says:

    did skype support typing action. I see that it is there in schema but doesn’t seems to be working neither the the image/gif is supported in Skype. If this is true then can you confirm it?

    1. What you are saying is right. The image/gif seems not to be supported in Skype, and fixed my sample. Thanks for sharing, Puneet-san.

  2. subhan says:

    you can send me the code in node js for azure directory login for skype bot

    1. If you want to collaborate Azure AD auth for users in your bot, please see the following sample. (You can download the sample code from Github.) But I’m sorry, it’s written by .NET SDK…
      https://blogs.msdn.microsoft.com/tsmatsuz/2016/09/06/microsoft-bot-framework-bot-with-authentication-and-signin-login/

  3. Siva Prathap Reddy N says:

    How to differentiate the channel source(messenger or skype) in code(nodejs), so that I can dynamically compose appropriate message?

    1. Yes, you can. Please see my previous post of https://blogs.msdn.microsoft.com/tsmatsuz/2016/08/19/build-skype-bot-with-microsoft-bot-framework-oauth-and-rest-api/ for details.
      First, the “channelId” is passed to your bot as part of the request body as follows, and your bot can get what type of channel is used.

      “channelId”: “skype”
      “serviceUrl”: “https://skype.botframework.com”

      The serive url (serviceUrl) is having the format of https://{channel specific name}.botframework.com. For example, the url of “https://skype.botframework.com” means the service url of “skype”.

  4. CJ says:

    Hi, does the embed Webchat support rich cards and buttons? If not, what are some other alternatives to this (without having to use Facebook Messenger or Skype) as I want to be able to customize the look and feel of the bot. thanks

    1. I’m sorry, but you must use the direct line api and create the custom code for rendering cards in such a case.

  5. Ajay says:

    How Does video playback work with embedded Web chat. I see there is a “playVideo” action, However this does not seem to work with web chat.

Skip to main content