Call Remote Assist from your HoloLens App


We annonced, in September 2018, new 1st party Mixed Reality business applications from Microsoft:

We recognize that many employees need information in context to apply their knowledge and craft. Not only on a 2-D screen โ€” but information and data in context, at the right place, and at the right time, so employees can produce even greater impact for their organizations. Mixed reality is a technology uniquely suited to do exactly that.

Here is Dynamics 365 Remote Assist โฌ‡

Remote Assis video


Let's explicit how you can leverage on Remote Assist calls directly within you HoloLens Mixed Reality apps!


Step 1. The Unity project has to be set up ๐Ÿ“ for Windows Mixed Reality

Note: You can skip this part if you are familiar with Unity and Mixed Reality.

If not, please refer to Create a Windows Mixed Reality App from 0 โ€“ The Unity project.

For the pre-requisites, here are the details: Create a Windows Mixed Reality App from 0 โ€“ The technology and the tools

As usual, you should have an Unity project configured for Windows Mixed Reality:

  • File menu / Build Settings โžก Universal Windows Platform selected as the Build platform
  • Edit menu / Project Settings / Player โžก Virtual Reality Supported checked
  • Edit menu / Project Settings / Player โžก Other Settings / .NET as Scripting Backend. Please also use the Experimental (.NET 4.6 Equivalent) for the Scripting Runtime Version.

Step 2. Call the Remote Assist ๐ŸŽฅ protocol from the Unity App ๐Ÿ•น

As you know, in a Mixed Reality Application, we can call UWP APIs. We just have to decorate the calls with a Platform dependent compilation directive:

#if ENABLE_WINMD_SUPPORT
// Blablabla in C# for UWP
#endif

We use the LaunchUriAsync method with the new Remote Assist protocol which is

ms-voip-video:?contactids=<ObjectID_From_AAD>

I will introduce what is the Object ID from Azure Active Directory in the next paragraph.

The helper function you can create would look like this:

private async Task LaunchURI(System.Uri uri)
{
#if ENABLE_WINMD_SUPPORT
    // Launch the URI
    try
    {
        var success = await Windows.System.Launcher.LaunchUriAsync(uri);

        if (success)
        {
            Debug.Log("URI launched");
        }
        else
        {
            Debug.Log("URI launch failed");
        }
    }
    catch (Exception ex)
    {
        Debug.Log(ex.Message);
    }
#endif
}

In your app, to call the Remote Assist protocol, just use:

string uriToLaunch 
    = @"ms-voip-video:?contactids=bf538576-xxxx-yyyy-zzzz-1d548b9e7010";
Debug.Log("LaunchUriAsync: " + uriToLaunch);

var uri = new Uri(uriToLaunch);
await LaunchURI(uri);
// If we hit an UI Thread issue, the sample code below gives a workaround

A sample code ๐Ÿ“„ ready to use

If you look for a minimal code for HoloLens, you can just handle the AirTap to initiate the Remote Assist call. Here is the sample:

using System;
using System;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.XR.WSA.Input;

public class Tap : MonoBehaviour
{

    private GestureRecognizer recognizer;

    // Use this for initialization
    async void Start()
    {
        recognizer = new GestureRecognizer();
        recognizer.SetRecognizableGestures(GestureSettings.Tap);
        recognizer.Tapped += GestureRecognizer_Tapped;
        recognizer.StartCapturingGestures();
    }

    private async void GestureRecognizer_Tapped(TappedEventArgs obj)
    {
#if ENABLE_WINMD_SUPPORT        
        string uriToLaunch = @"ms-voip-video:?contactids=bf538576-xxxx-yyyy-zzzz-1d548b9e7010";
        Debug.Log("LaunchUriAsync: " + uriToLaunch);

        Uri uri = new Uri(uriToLaunch);
        await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
        {
            // Work done on the UI thread
            LaunchURI(uri).ConfigureAwait(false).GetAwaiter().GetResult();
        });
#endif
    }


    private async Task LaunchURI(System.Uri uri)
    {
#if ENABLE_WINMD_SUPPORT
        // Launch the URI
        try
        {
            var success = await Windows.System.Launcher.LaunchUriAsync(uri);

            if (success)
            {
                Debug.Log("URI launched");
            }
            else
            {
                Debug.Log("URI launch failed");
            }
        }
        catch (Exception ex)
        {
            Debug.Log(ex.Message);
        }
#endif
    }
}

Note: Be sure to use the Experimental (.NET 4.6 Equivalent) for the Scripting Runtime Version in the Edit menu / Project Settings / Player / Other Settings section.

The explanations about the GestureRecognizer can be found here: Create a Windows Mixed Reality App from 0 โ€“ A simple example.

The entire Unity project with this sample code is on GitHub. If you want to test it, deploy the app on the HoloLens and just do an AirTap to initiate the Remote Assist call.

Step 3. Find the ObjectID ๐Ÿ†” in Azure Active Directory (AAD) ๐Ÿ“™

Note: The programmatic way to get this ObjectID within the app will be explained in an article that I will publish soon (here is the link - https://blogs.msdn.microsoft.com/appconsult/2018/12/06/23775/).

Right now, to get it 'manually', you have to connect to your Azure Active Directory of your Office 365 tenant (again I will explain all of this in a following article):

Users link in Azure Active Directory on the Azure portal

  • Click on the user whom you want to call by using Remote Assist

AAD lists all users

  • The page displays the Objet ID of the user

Object ID displayed in the Azure Portal

For testing, you just have to use it directly in your code in the url for the LaunchUriAsync call.

string uriToLaunch = @"ms-voip-video:?contactids=bf538576-xxxx-yyyy-zzzz-1d548b9e7010";

Do not worry if you are not able to perform this step; Remote Assist is smart enough to propose you an user interface to choose the person to call if you launch it with an object ID that does not exist.

Step 4. Test ๐Ÿงช the code

To test the code, we have to run the code on the HoloLens device; It is because we use UWP APIs and we have to launch the Remote Assist app which is only on the HoloLens device.

Here are the steps to deploy the app on the HoloLens with Visual Studio:

  • Build the Unity project (You choose a folder in which Unity generates a Visual Studio solution ; Usually, we create a new subfolder of the Unity project named App).
  • Open the resulting Visual Studio solution and choose x86 on Device Visual Studio Debug x86 Device (if USB connected) or Remote Machine Visual Studio Debug x86 Remote Machine (if the PC and HoloLens are on the same Wifi).

To get the IP address of your HoloLens, take a look at this video

  • Debug or Start without debugging in Visual Studio: The apps will be deployed and launch on HoloLens.

Debug or Launch without debug in Visual Studio

Note: You can also Deploy the app on the HoloLens by using the Build menu / Deploy Solution. Then, on the device, just locate and launch the app from the Start menu.

Deploy App in Visual Studio

How does the app work ๐Ÿ”Ž?

  1. In your app, you run the LaunchUriAsync method with the Remote Assist protocol. This protocol is registered for the Remote Assist app and Windows opens the app.
  2. The Remote Assist App has the Object ID of the person to call in the url and initiate the call immediately to this person (if the object ID is not found, Remote Assist allows the user to select a person to call).
  3. The user can then switch back to the HoloLens App using the Bloom gesture and by finding the app in the Start menu.
  4. Your HoloLens app runs and Remote Assist continues streaming the video/audio to the remote user.
  5. Do not forget to get back to the Remote Assist app when finished in order to stop the call.

A more interesting sample ๐Ÿ“ฆ with a tagalong billboard following the user gaze

The entire source code is available on GitHub. The sample provide you a Holographic button that is always

  • Following you (the camera).
  • Oriented in front of you.
  • At a fixed distance from you.
  • Visible and not directly in the center of the field of view: it seeks to have a part of itself in the view frustum of the camera

The purpose of this sample is to provide you an out of the box tagalong billboard for you apps ๐Ÿ“ฆ๐ŸŽ!

Here is below, in action, this sample and the Remote Assist call done.

via Gfycat

Conclusion

Remote Assist is a great tool you could leverage on for assistance and collaboration directly from your applications. We are just at the beginning of the journey: we will have further improvements on the way to use all Remote Assist functionalities.
Feel free to take a look and run the minimal sample code explained in this article or the tagalong billboard sample. You can find it on GitHub - https://github.com/Microsoft/Windows-AppConsult-Samples-MixedReality/tree/master/RemoteAssistSamples.

I hope you could reuse or be inspired by with the Tagalong billboard sample. Your feedback is welcome!

Bye,

@sbovo for the ๐Ÿ’ป ๐ŸŽฎ Windows AppConsult team.

--

References

Comments (37)

  1. KamalDua says:

    Nice blog Sebastien and Thanks for useful information.
    On hololens device, I installed the Remote assist app and its working fine by calling to other user.
    I configured the suggested settings for Tagalong billboard sample code and able to deploy build on Hololens device. But application is not launching remote assist app from my app.
    Is there any need of other configuration to be set or problem with [ms-voip-video] protocol activation.
    Please help in this matter.

    Thanks in Advance!

    1. Hello, thanks for your feedback.
      The code for calling Remote Assist in the minimal sample and in the Tagalong billboard sample is the same (or it should be). I may have done a mistake in the code extracts in the blog post or in the GitHub source code. Have you tried building the app from the Blog post instructions or have you directly tried the sample from the GitHub repo?
      Thanks,
      Sebastien.

      1. Hello, the issue might be related to a missing folder of the Mixed Reality Toolkit due to a .gitignore entry. The Actual repo version is updated.

  2. RoXoR21 says:

    The blog is good and well written. But i am facing the same issue as KamalDua. I followed all instruction to setup the environment.
    I have *
    1)unity 2018.1.0f2
    2)MS visual studio 2017 v15.8.2
    3) Hololens with Remote Assist Application installed.

    I can successfully create solution using Unity3D and able to launch the application but Remote Assist application does not open on clicking the GUI. I have used the project from GitHub mentioned in references.

    Please do let me know if i am missing something in this.

    1. Thanks!
      I will give a retry today and let you know the outcome.
      Regards,
      Sebastien.

      1. Anonymous says:
        (The content was deleted per user request)
      2. RoXoR21 says:

        Great!! Thanks Sabastien for quick response. I will let you know if i come across something while debugging.

        1. Hello, a folder of the Mixed Reality Toolkit was missing due to a .gitignore entry. You can retry using the actual repo version

          1. RoXoR21 says:

            Thanks Sebateian. It is working fine now.

  3. So here is an update. You know what? In the .gitignore file, there was “Icon?”. So the full folder /Assets/HoloToolkit/UX/Textures/Icons/ was ignored and not part of the source code on GitHub. I fixed this in the repo. You can grab the source code https://github.com/Microsoft/Windows-AppConsult-Samples-MixedReality.

    Let me know if this is ok.
    Regards, Sebastien.

    1. Anonymous says:
      (The content was deleted per user request)
    2. RoXoR21 says:

      Hi Sebastein,
      The Null error because of missing icon is resolved. I have tested it in emulator and Hololens device.
      Application open remote assist in the emulator, But it does not move further after air tap in Hololens. Is there any versioning related issue here either in Unity or visual studio? The only difference here is
      Scripting Run-time Version : .Net 4.x Equivalent.
      It will be awesome if you just check and revert.
      Thank you.

      1. Hello, I have never used the emulator for this app. I used the Unity 2017.4.15f1 and Visual Studio 15.9. Did you try the app on HoloLens?

        1. RoXoR21 says:

          Yes! i am deploying it in Hololens itself. I have even tried it in Emulator, just to check.
          I am using Unity3D 2018.1 and visual studio 2017 v15.8.2 3.

          1. I always try to keep the LTS version of Unity 2017.4.xxx for HoloLens an Mixed Reality in particular for the Mixed Reality Toolkit. I would recommend you to test using the latest 2017.4.16 version at https://unity3d.com/get-unity/download/archive. If it does not work, feel free to come back to me.
            Sebastien.

    3. RoXoR21 says:

      Hello Sebastien,
      Updated Unity to 2017.4.16. Still i am facing same issue.
      Execution does not goes beyond this point – await Windows.System.Launcher.LaunchUriAsync(uri). Please let me know what am i doing wrong. ๐Ÿ™
      Thank you.

      1. This was the initial code that I updated to this one:

        #if ENABLE_WINMD_SUPPORT

        string uriToLaunch = @”ms-voip-video:?contactids=bf538576-xxxx-yyyy-zzzz-1d548b9e7010″;

        Debug.Log(“LaunchUriAsync: ” + uriToLaunch);

        Uri uri = new Uri(uriToLaunch);

        await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>

        {

        // Work done on the UI thread

        LaunchURI(uri).ConfigureAwait(false).GetAwaiter().GetResult();

        });

        #endif

        1. RoXoR21 says:

          I can able to open ms-store:, ms-settings: and other apps using same code by changing URI. It might be something to do with protocol ms-voip-video.

          1. Hello, does the minimal sample work or do the problem impacts both samples?

          2. RoXoR21 says:

            I have tested both samples and issue resides in both Minimal as well as in TagAlong UI project.

        2. KamalDua says:

          Hi Sebastien,

          I am using Unity 2017.4.15f1 and VS 2017 15.9.2.
          While running this app on Hololens device in debugging mode and click to Remote Assist tagalong billboard, I am getting following stack trace. Please help us to resolve the issue.

          Exception thrown: ‘System.NullReferenceException’ in UnityEngine.UI.dll
          NullReferenceException: Object reference not set to an instance of an object.
          at UnityEngine.EventSystems.BaseEventData.get_selectedObject()
          at HoloToolkit.Unity.Receivers.InteractionReceiver.OnInputClicked(InputClickedEventData eventData)
          at HoloToolkit.Unity.InputModule.InputManager.c.b__123_4(IInputClickHandler handler, BaseEventData eventData)
          at UnityEngine.EventSystems.ExecuteEvents.Execute[T](GameObject target, BaseEventData eventData, EventFunction`1 functor)
          (Filename: Line: 0)

          Thanks in Advance!

          1. Hello, does the minimal sample work or do the problem impacts both samples?

          2. RoXoR21 says:

            Hi KamalDua,

            Your problem is because of missing texture files in path Assets/HoloToolkit/UX/Textures/Icons/. You have to pull the recent update project from Git.

          3. RoXoR21 says:

            Hi,
            The problem was because there were 2 voip application in my hololens Skype and Remote Assist. I have created a native sample application from visual studio 2017 to launch Remote Assist and i got a pop-up displaying which voip application to open Skype and Remote Assist.
            To successfully open remote assist using Unity3D app Uninstall Skype or other voip application you have and run. /n There should be handler function to get list of all installed voip application.

          4. KamalDua says:

            Hi RoXoR21,

            Yesterday I took latest code from git and files are present in Assets/HoloToolkit/UX/Textures/Icons/. should I take it once again. are any new changes occur once again?
            Also I uninstalled Skype from my Hololens device as you suggested.

          5. KamalDua says:

            Right now I am able to launch remote assist app successfully from FocusEnter event(just pasted code from InputUp event to FocusEnter). InputUp is still causing issue of ‘NullReference’. So suggest me what asset file is missing. As I took latest code yesterday.

          6. RoXoR21 says:

            I was getting the same Null Reference issue because the icons where not assigned to Compound Button Icon component attached to RemoteAssistVideoCall gameobject. Just check if the Icon are assigned in Custom Icon of “Compound Button Icon” script.

          7. KamalDua says:

            Hi RoXoR21,
            Can you please share the changes you made in code files if you able to resolve this issue.
            Thanks!

          8. KamalDua says:

            Now I am not getting any NullReference issue.
            This time, after opening the project in Unity Editor, I did not set/click on Menu [Mixed Reality Toolkit]>Configure>Apply Mixed reality project settings, Scene Settings and UWP capability settings and build it.
            Now InputUp event working fine and opening remote assist app.
            But remote assist is not making any call to defined AD. first it get signed in then it displays its contacts.
            Do I need to do more configuration?

          9. KamalDua says:

            Sorry! My mistake, wrong AD mentioned. Now its able to call desired contacts!
            Thanks guys!

        3. Anonymous says:
          (The content was deleted per user request)
  4. Thanks for the feedback and your investigations!

  5. So the issue seems related to the Mixed Reality Toolkit itself. May I suggest you to get the actual version of the repo in a new separate folder?

  6. KamalDua says:

    Thanks Sebastien and RoXoR21 for your guidance!
    Now I want that after making call or dropping/cancelling, it would display us our app (tagalong billboard) again and remote assist app should be disappear or should go to background.

    1. Anonymous says:
      (The content was deleted per user request)
    2. RoXoR21 says:

      Hi KamalDua,
      Great that your problem got resolved and Thanks to Sebastien for his wonderful tutorial and support. I am not sure but we might not have a control over call back events on the state of calls from Remote Assist. Once you disconnect the call just give a bloom gesture it will show your application, and problem here is that on click it reopens the application instead of opening from the same state.

      1. Thanks guys for the discussion here. Yes, when we “reopen” the app, due to the life cycle of app on Windows, the app is launch as we open it for the first time. It is our responsibility as a developer before making the call to save some state. When we open the app, we have to check the state and “simulate” or “load” the saved state. Some information in this page: https://docs.microsoft.com/en-us/windows/uwp/launch-resume/app-lifecycle

Skip to main content