First Look at the Kinect for Windows SDK

The Kinect for Windows SDK has finally arrived, and we couldn’t wait to get our hands on it!

After downloading, installing, and actually getting a Kinect with a USB connector for our PC’s, we were good to go…

Step 1 – What does the SDK give us?

When we started, we didn’t really know what to expect from the SDK.

What you get are a set of 20 points, each corresponding to a specific joint in the body. These points come in standard ‘x, y, z’ coordinates, with an extra coordinate ‘w’ , which we haven’t got round to figuring out what it is there for just yet.

Here comes problem one. The points that are given are within the range -1 and 1, going from left to right for x, and bottom to top with y. Therefore, you have to first convert these to work with WPF applications so that if fits within the coordinates of a standard page.

This part was just a formality and had a very simple fix with a couple of converters.

One for X:

    1: /// <summary>
    2: /// Joint converter
    3: /// </summary>
    4: public class KinectValueToScreenCoOrdinatesConverterX : IValueConverter
    5: {
    6:     /// <summary>
    7:     /// Converts a value.
    8:     /// </summary>
    9:     /// <param name="value">The value produced by the binding source.</param>
   10:     /// <param name="targetType">The type of the binding target property.</param>
   11:     /// <param name="parameter">The converter parameter to use.</param>
   12:     /// <param name="culture">The culture to use in the converter.</param>
   13:     /// <returns>
   14:     /// A converted value. If the method returns null, the valid null value is used.
   15:     /// </returns>
   16:     public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
   17:     {
   18:         double multiplier = Application.Current.MainWindow.Width;
   19:  
   20:         double translation = 0;
   21:         try
   22:         {
   23:             if (parameter != null)
   24:             {
   25:                 translation = double.Parse(parameter as string);
   26:             }
   27:  
   28:             float fl = (float)value;
   29:             return translation + (multiplier + (fl * multiplier));
   30:        }
   31:        catch (InvalidCastException)
   32:        {
   33:             return 0;
   34:         }
   35:     }
   36:  
   37:     /// <summary>
   38:     /// Converts a value.
   39:     /// </summary>
   40:     /// <param name="value">The value that is produced by the binding target.</param>
   41:     /// <param name="targetType">The type to convert to.</param>
   42:     /// <param name="parameter">The converter parameter to use.</param>
   43:     /// <param name="culture">The culture to use in the converter.</param>
   44:     /// <returns>
   45:     /// A converted value. If the method returns null, the valid null value is used.
   46:     /// </returns>
   47:     public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
   48:     {
   49:         throw new NotImplementedException();
   50:     }
   51: }

And one for Y:

    1: /// <summary>
    2: /// Joint converter Y
    3: /// </summary>
    4: public class KinectValueToScreenCoOrdinatesConverterY : IValueConverter
    5: {
    6:     /// <summary>
    7:     /// Converts a value.
    8:     /// </summary>
    9:     /// <param name="value">The value produced by the binding source.</param>
   10:     /// <param name="targetType">The type of the binding target property.</param>
   11:     /// <param name="parameter">The converter parameter to use.</param>
   12:     /// <param name="culture">The culture to use in the converter.</param>
   13:     /// <returns>
   14:     /// A converted value. If the method returns null, the valid null value is used.
   15:     /// </returns>
   16:     public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
   17:     {
   18:         double multiplier = Application.Current.MainWindow.Height;
   19:  
   20:             double translation = 0;
   21:             try
   22:             {
   23:                 if (parameter != null)
   24:                 {
   25:                     translation = double.Parse(parameter as string);
   26:                 }
   27:  
   28:                 float fl = (float)value;
   29:                 return translation + (multiplier + (-fl * multiplier));
   30:             }
   31:             catch (InvalidCastException)
   32:             {
   33:                 return 0;
   34:             }
   35:         }
   36:  
   37:         /// <summary>
   38:         /// Converts a value.
   39:         /// </summary>
   40:         /// <param name="value">The value that is produced by the binding target.</param>
   41:         /// <param name="targetType">The type to convert to.</param>
   42:         /// <param name="parameter">The converter parameter to use.</param>
   43:         /// <param name="culture">The culture to use in the converter.</param>
   44:         /// <returns>
   45:         /// A converted value. If the method returns null, the valid null value is used.
   46:         /// </returns>
   47:         public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
   48:         {
   49:             throw new NotImplementedException();
   50:         }
   51:     }

A converter is assigned to every part of the skeleton that is displayed on the UI. Since Kinect returns a series of skeletons (each one representing a person) with the points already mapped to each joint, we decided to drive the MVVM pattern directly off of the SkeletonData that Kinect returns.

Step 2 – Putting everything into the MVVM pattern

To simplify things and make it easier to create multiple skeletons on screen at any one time, we decided to use the MVVM pattern. There are four main parts to this:

1. The Kinect connection

2. The Main View Model

3. The Skeleton View Model

4. The UI

Kinect Connection

The Kinect connection is an intermediate class between our view models and the Kinect sensor. It fires Skeleton ready events for each skeleton that is returned and skeleton frame complete events when an entire frame has been processed. As it processes the skeletons in a frame, it keeps track of their ID value so we can track all the skeletons that were in a frame during the Skeleton Frame Complete event.

One of the main parts of this class is the InitializeNui() method:

    1: /// <summary>
    2:         /// Initializes the Kinect sensor.
    3:         /// </summary>
    4:         /// <returns>bool value true if the sensor initialised correctly</returns>
    5:         private bool InitializeNui()
    6:         {
    7:             if (this.kinectRunTime == null)
    8:             {
    9:                 return false;
   10:             }
   11:  
   12:             try
   13:             {
   14:                 this.kinectRunTime.Initialize(RuntimeOptions.UseDepth | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
   15:             }
   16:             catch (Exception exception)
   17:             {
   18:                 Console.WriteLine(exception.ToString());
   19:                 return false;
   20:             }
   21:  
   22:             this.kinectRunTime.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
   23:  
   24:             this.kinectRunTime.SkeletonEngine.TransformSmooth = true;
   25:  
   26:             var parameters = new TransformSmoothParameters
   27:             {
   28:                 Smoothing = 0.75f,
   29:                 Correction = 0.0f,
   30:                 Prediction = 0.0f,
   31:                 JitterRadius = 0.05f,
   32:                 MaxDeviationRadius = 0.04f
   33:             };
   34:  
   35:             this.kinectRunTime.SkeletonEngine.SmoothParameters = parameters;
   36:  
   37:             return true;
   38:         }
   39:     }

This method initialises the Kinect sensor with a set of pre define arguments. The actual initialization of the sensor happens in the line :

    1: this.kinectRunTime.Initialize(RuntimeOptions.UseDepth | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The runtime options specify what you want the Kinect sensor to look for. In our example we were looking for skeleton frames and images from the camera.

Main View Model

The main view model controls the allocation of skeletons to the skeleton view models. It contains a list of Skeleton View models and a dictionary of skeleton view models with the skeleton id as the key. Each time a skeleton ready event is fired in the Kinect connection it looks up the skeleton ID value in the dictionary and assigns the new skeleton data to the skeleton in that view model.

Originally the Main view model was also going to sort out the removal of skeletons that are no longer active. However the Kinect sensor does not always return all of the active skeletons in each of the Skelton frame ready events that the Kinect SDK produce. This means that a simple removal of any skeletons not in the current frame will fail and the view model associated with a skeleton will be continuously deleted and re-created. To get around this we placed a timer in each of the skeleton view models. If the view model is not updated within a set period of time the view model raises an event and the main view model then catches it and deleted the skeleton.

The Skeleton View Model

The skeleton view model is essentially a way of exposing the properties of a SkeletonData object so that the UI can bind to them. When the skeleton property is changed a NotifyPropertyChanged event is raised on all of the exposed properties which updates all the bindings on the UI.

The skeleton view model also controls the Gesture Service which we will discuss in our next blog post. This is again updated when the skeleton property changes.

The full code for the skeleton view model can be found in the source code below:

    1: /// <summary>
    2:     /// The main view model
    3:     /// </summary>
    4:     public class SkeletonViewModel : INotifyPropertyChanged
    5:     {
    6:         /// <summary>
    7:         /// The gesture controler for this skeleton
    8:         /// </summary>
    9:         private GestureControler gestures = new GestureControler();
   10:  
   11:         #region fields
   12:  
   13:         /// <summary>
   14:         /// backing field for the gesture text
   15:         /// </summary>
   16:         private string gestureText;
   17:  
   18:         /// <summary>
   19:         /// backing field for the gesture property
   20:         /// </summary>
   21:         private bool gestureDetected;
   22:  
   23:         /// <summary>
   24:         /// The skeleton data
   25:         /// This is used as the backing field for all properties
   26:         /// </summary>
   27:         private SkeletonData skeleton;
   28:  
   29:         /// <summary>
   30:         /// backing field for the joint color
   31:         /// </summary>
   32:         private Color jointColor;
   33:  
   34:         /// <summary>
   35:         /// Occurs when [delete skeleton].
   36:         /// </summary>
   37:         public event EventHandler DeleteSkeleton;
   38:         
   39:         /// <summary>
   40:         /// the timer for the on screen gesture text
   41:         /// </summary>
   42:         private DispatcherTimer textTimer = new DispatcherTimer() { Interval = TimeSpan.FromSeconds(3) };
   43:  
   44:         #endregion
   45:  
   46:         #region constructors
   47:  
   48:         /// <summary>
   49:         /// Initializes a new instance of the <see cref="SkeletonViewModel"/> class.
   50:         /// </summary>
   51:         public SkeletonViewModel()
   52:         {
   53:             this.ChangeTimer = new DispatcherTimer();
   54:             this.ChangeTimer.Interval = TimeSpan.FromSeconds(0.5);
   55:             this.ChangeTimer.Tick += new EventHandler(this.ChangeTimer_Tick);
   56:             this.textTimer.Tick += new EventHandler(this.TextTimer_Tick);
   57:             ChangeTimer.Start();
   58:             DefineGestures();
   59:             this.gestures.GestureRecognised += new EventHandler<GestureEventArgs>(this.Gestures_GestureRecognised);
   60:         }
   61:  
   62:         #endregion
   63:  
   64:         #region events
   65:  
   66:         /// <summary>
   67:         /// Occurs when a property value changes.
   68:         /// </summary>
   69:         public event PropertyChangedEventHandler PropertyChanged;
   70:  
   71:         #endregion
   72:  
   73:         #region properties
   74:  
   75:         /// <summary>
   76:         /// Gets or sets the change timer.
   77:         /// </summary>
   78:         /// <value>
   79:         /// The change timer.
   80:         /// </value>
   81:         public DispatcherTimer ChangeTimer
   82:         {
   83:             get;
   84:             set;
   85:         }
   86:  
   87:         /// <summary>
   88:         /// Gets or sets the gesture text.
   89:         /// </summary>
   90:         /// <value>
   91:         /// The gesture text.
   92:         /// </value>
   93:         public string GestureText
   94:         {
   95:             get
   96:             {
   97:                 return this.gestureText;
   98:             }
   99:  
  100:             set
  101:             {
  102:                 if (this.gestureText != value)
  103:                 {
  104:                     this.gestureText = value;
  105:                     this.NotifyPropertyChanged("GestureText");
  106:                 }
  107:             }
  108:         }
  109:  
  110:         /// <summary>
  111:         /// Gets or sets a value indicating whether this <see cref="SkeletonViewModel"/> is waved.
  112:         /// </summary>
  113:         /// <value>
  114:         ///   <c>true</c> if waved; otherwise, <c>false</c>.
  115:         /// </value>
  116:         public bool GestureDetected
  117:         {
  118:             get
  119:             {
  120:                 return this.gestureDetected;
  121:             }
  122:  
  123:             set
  124:             {
  125:                 if (this.gestureDetected != value)
  126:                 {
  127:                     this.gestureDetected = value;
  128:                     this.NotifyPropertyChanged("GestureDetected");
  129:                 }
  130:             }
  131:         }
  132:  
  133:         /// <summary>
  134:         /// Gets or sets the color of the joint.
  135:         /// </summary>
  136:         /// <value>
  137:         /// The color of the joint.
  138:         /// </value>
  139:         public Color JointColor
  140:         {
  141:             get
  142:             {
  143:                 return this.jointColor;
  144:             }
  145:  
  146:             set
  147:             {
  148:                 if (this.jointColor != value)
  149:                 {
  150:                     this.jointColor = value;
  151:                     this.NotifyPropertyChanged("JointColor");
  152:                 }
  153:             }
  154:         }
  155:  
  156:         /// <summary>
  157:         /// Gets or sets the skeleton.
  158:         /// </summary>
  159:         /// <value>
  160:         /// The skeleton.
  161:         /// </value>
  162:         public SkeletonData Skeleton
  163:         {
  164:             get
  165:             {
  166:                 return this.skeleton;
  167:             }
  168:  
  169:             set
  170:             {
  171:                 if (this.skeleton != value)
  172:                 {
  173:                     this.skeleton = value;
  174:                     this.NotifyPropertyChanged("Skeleton");
  175:                     this.NotifyAllChange();
  176:                     ResetTimer();
  177:                     this.gestures.UpdateAllGestures(this.skeleton);
  178:                 }
  179:             }
  180:         }
  181:  
  182:         #region KinectBodyParts
  183:  
  184:         /// <summary>
  185:         /// Gets the head.
  186:         /// </summary>
  187:         public Joint Head
  188:         {
  189:             get
  190:             {
  191:                 if (this.Skeleton != null)
  192:                 {
  193:                     return this.Skeleton.Joints[JointID.Head];
  194:                 }
  195:                 else
  196:                 {
  197:                     return new Joint();
  198:                 }
  199:             }
  200:         }
  201:  
  202:         /// <summary>
  203:         /// Gets the left hand.
  204:         /// </summary>
  205:         public Joint LeftHand
  206:         {
  207:             get
  208:             {
  209:                 if (this.Skeleton != null)
  210:                 {
  211:                     return this.Skeleton.Joints[JointID.HandLeft];
  212:                 }
  213:                 else
  214:                 {
  215:                     return new Joint();
  216:                 }
  217:             }
  218:         }
  219:  
  220:         /// <summary>
  221:         /// Gets the right hand.
  222:         /// </summary>
  223:         public Joint RightHand
  224:         {
  225:             get
  226:             {
  227:                 if (this.Skeleton != null)
  228:                 {
  229:                     return this.Skeleton.Joints[JointID.HandRight];
  230:                 }
  231:                 else
  232:                 {
  233:                     return new Joint();
  234:                 }
  235:             }
  236:         }
  237:  
  238:         /// <summary>
  239:         /// Gets the left wrist.
  240:         /// </summary>
  241:         public Joint LeftWrist
  242:         {
  243:             get
  244:             {
  245:                 if (this.Skeleton != null)
  246:                 {
  247:                     return this.Skeleton.Joints[JointID.WristLeft];
  248:                 }
  249:                 else
  250:                 {
  251:                     return new Joint();
  252:                 }
  253:             }
  254:         }
  255:  
  256:         /// <summary>
  257:         /// Gets the right wrist.
  258:         /// </summary>
  259:         public Joint RightWrist
  260:         {
  261:             get
  262:             {
  263:                 if (this.Skeleton != null)
  264:                 {
  265:                     return this.Skeleton.Joints[JointID.WristRight];
  266:                 }
  267:                 else
  268:                 {
  269:                     return new Joint();
  270:                 }
  271:             }
  272:         }
  273:  
  274:         /// <summary>
  275:         /// Gets the left elbow.
  276:         /// </summary>
  277:         public Joint LeftElbow
  278:         {
  279:             get
  280:             {
  281:                 if (this.Skeleton != null)
  282:                 {
  283:                     return this.Skeleton.Joints[JointID.ElbowLeft];
  284:                 }
  285:                 else
  286:                 {
  287:                     return new Joint();
  288:                 }
  289:             }
  290:         }
  291:  
  292:         /// <summary>
  293:         /// Gets the right elbow.
  294:         /// </summary>
  295:         public Joint RightElbow
  296:         {
  297:             get
  298:             {
  299:                 if (this.Skeleton != null)
  300:                 {
  301:                     return this.Skeleton.Joints[JointID.ElbowRight];
  302:                 }
  303:                 else
  304:                 {
  305:                     return new Joint();
  306:                 }
  307:             }
  308:         }
  309:  
  310:         /// <summary>
  311:         /// Gets the shoulder center.
  312:         /// </summary>
  313:         public Joint ShoulderCenter
  314:         {
  315:             get
  316:             {
  317:                 if (this.Skeleton != null)
  318:                 {
  319:                     return this.Skeleton.Joints[JointID.ShoulderCenter];
  320:                 }
  321:                 else
  322:                 {
  323:                     return new Joint();
  324:                 }
  325:             }
  326:         }
  327:  
  328:         /// <summary>
  329:         /// Gets the spine.
  330:         /// </summary>
  331:         public Joint Spine
  332:         {
  333:             get
  334:             {
  335:                 if (this.Skeleton != null)
  336:                 {
  337:                     return this.Skeleton.Joints[JointID.Spine];
  338:                 }
  339:                 else
  340:                 {
  341:                     return new Joint();
  342:                 }
  343:             }
  344:         }
  345:  
  346:         /// <summary>
  347:         /// Gets the hip center.
  348:         /// </summary>
  349:         public Joint HipCenter
  350:         {
  351:             get
  352:             {
  353:                 if (this.Skeleton != null)
  354:                 {
  355:                     return this.Skeleton.Joints[JointID.HipCenter];
  356:                 }
  357:                 else
  358:                 {
  359:                     return new Joint();
  360:                 }
  361:             }
  362:         }
  363:  
  364:         /// <summary>
  365:         /// Gets the left knee.
  366:         /// </summary>
  367:         public Joint LeftKnee
  368:         {
  369:             get
  370:             {
  371:                 if (this.Skeleton != null)
  372:                 {
  373:                     return this.Skeleton.Joints[JointID.KneeLeft];
  374:                 }
  375:                 else
  376:                 {
  377:                     return new Joint();
  378:                 }
  379:             }
  380:         }
  381:  
  382:         /// <summary>
  383:         /// Gets the right knee.
  384:         /// </summary>
  385:         public Joint RightKnee
  386:         {
  387:             get
  388:             {
  389:                 if (this.Skeleton != null)
  390:                 {
  391:                     return this.Skeleton.Joints[JointID.KneeRight];
  392:                 }
  393:                 else
  394:                 {
  395:                     return new Joint();
  396:                 }
  397:             }
  398:         }
  399:  
  400:         /// <summary>
  401:         /// Gets the left ankle.
  402:         /// </summary>
  403:         public Joint LeftAnkle
  404:         {
  405:             get
  406:             {
  407:                 if (this.Skeleton != null)
  408:                 {
  409:                     return this.Skeleton.Joints[JointID.AnkleLeft];
  410:                 }
  411:                 else
  412:                 {
  413:                     return new Joint();
  414:                 }
  415:             }
  416:         }
  417:  
  418:         /// <summary>
  419:         /// Gets the right ankle.
  420:         /// </summary>
  421:         public Joint RightAnkle
  422:         {
  423:             get
  424:             {
  425:                 if (this.Skeleton != null)
  426:                 {
  427:                     return this.Skeleton.Joints[JointID.AnkleRight];
  428:                 }
  429:                 else
  430:                 {
  431:                     return new Joint();
  432:                 }
  433:             }
  434:         }
  435:  
  436:         /// <summary>
  437:         /// Gets the left foot.
  438:         /// </summary>
  439:         public Joint LeftFoot
  440:         {
  441:             get
  442:             {
  443:                 if (this.Skeleton != null)
  444:                 {
  445:                     return this.Skeleton.Joints[JointID.FootLeft];
  446:                 }
  447:                 else
  448:                 {
  449:                     return new Joint();
  450:                 }
  451:             }
  452:         }
  453:  
  454:         /// <summary>
  455:         /// Gets the right foot.
  456:         /// </summary>
  457:         public Joint RightFoot
  458:         {
  459:             get
  460:             {
  461:                 if (this.Skeleton != null)
  462:                 {
  463:                     return this.Skeleton.Joints[JointID.FootRight];
  464:                 }
  465:                 else
  466:                 {
  467:                     return new Joint();
  468:                 }
  469:             }
  470:         }
  471:  
  472:         /// <summary>
  473:         /// Gets the Left Hip
  474:         /// </summary>
  475:         public Joint RightHip
  476:         {
  477:             get
  478:             {
  479:                 if (this.Skeleton != null)
  480:                 {
  481:                     return this.Skeleton.Joints[JointID.HipRight];
  482:                 }
  483:                 else
  484:                 {
  485:                     return new Joint();
  486:                 }
  487:             }
  488:         }
  489:  
  490:         /// <summary>
  491:         /// Gets the Left Hip
  492:         /// </summary>
  493:         public Joint LeftHip
  494:         {
  495:             get
  496:             {
  497:                 if (this.Skeleton != null)
  498:                 {
  499:                     return this.Skeleton.Joints[JointID.HipLeft];
  500:                 }
  501:                 else
  502:                 {
  503:                     return new Joint();
  504:                 }
  505:             }
  506:         }
  507:  
  508:         /// <summary>
  509:         /// Gets the Right Shoulder
  510:         /// </summary>
  511:         public Joint RightShoulder
  512:         {
  513:             get
  514:             {
  515:                 if (this.Skeleton != null)
  516:                 {
  517:                     return this.Skeleton.Joints[JointID.ShoulderRight];
  518:                 }
  519:                 else
  520:                 {
  521:                     return new Joint();
  522:                 }
  523:             }
  524:         }
  525:  
  526:         /// <summary>
  527:         /// Gets the Left Shoulder
  528:         /// </summary>
  529:         public Joint LeftShoulder
  530:         {
  531:             get
  532:             {
  533:                 if (this.Skeleton != null)
  534:                 {
  535:                     return this.Skeleton.Joints[JointID.ShoulderLeft];
  536:                 }
  537:                 else
  538:                 {
  539:                     return new Joint();
  540:                 }
  541:             }
  542:         }
  543:         #endregion
  544:  
  545:         #endregion
  546:  
  547:         #region methods
  548:  
  549:         /// <summary>
  550:         /// Handles the Tick event of the textTimer control.
  551:         /// </summary>
  552:         /// <param name="sender">The source of the event.</param>
  553:         /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
  554:         private void TextTimer_Tick(object sender, EventArgs e)
  555:         {
  556:             this.GestureDetected = false;
  557:             this.textTimer.Stop();
  558:         }
  559:  
  560:         /// <summary>
  561:         /// Handles the GestureRecognised event of the Gestures control.
  562:         /// </summary>
  563:         /// <param name="sender">The source of the event.</param>
  564:         /// <param name="e">The <see cref="KinectSkeltonTracker.GestureEventArgs"/> instance containing the event data.</param>
  565:         private void Gestures_GestureRecognised(object sender, GestureEventArgs e)
  566:         {
  567:             if (e.GestureType == GestureType.WaveRight)
  568:             {
  569:                 this.GestureDetected = true;
  570:                 this.GestureText = "Waved with right hand";
  571:                 this.textTimer.Start();
  572:             }
  573:             else if (e.GestureType == GestureType.WaveLeft)
  574:             {
  575:                 this.GestureDetected = true;
  576:                 this.GestureText = "Waved with left hand";
  577:                 this.textTimer.Start();
  578:             }
  579:             else if (e.GestureType == GestureType.LeftSwipe)
  580:             {
  581:                 this.GestureDetected = true;
  582:                 this.GestureText = "Swiped left";
  583:                 this.textTimer.Start();
  584:             }
  585:             else if (e.GestureType == GestureType.RightSwipe)
  586:             {
  587:                 this.GestureDetected = true;
  588:                 this.GestureText = "Swiped right";
  589:                 this.textTimer.Start();
  590:             }
  591:             else if (e.GestureType == GestureType.Menu)
  592:             {
  593:                 this.GestureDetected = true;
  594:                 this.GestureText = "Menu";
  595:                 this.textTimer.Start();
  596:             }
  597:         }
  598:  
  599:         /// <summary>
  600:         /// Defines the gestures.
  601:         /// </summary>
  602:         private void DefineGestures()
  603:         {
  604:             IRelativeGestureSegment[] waveRightSegments = new IRelativeGestureSegment[6];
  605:             WaveRightSegment1 waveRightSegment1 = new WaveRightSegment1();
  606:             WaveRightSegment2 waveRightSegment2 = new WaveRightSegment2();
  607:             waveRightSegments[0] = waveRightSegment1;
  608:             waveRightSegments[1] = waveRightSegment2;
  609:             waveRightSegments[2] = waveRightSegment1;
  610:             waveRightSegments[3] = waveRightSegment2;
  611:             waveRightSegments[4] = waveRightSegment1;
  612:             waveRightSegments[5] = waveRightSegment2;
  613:             this.gestures.AddGesture(GestureType.WaveRight, waveRightSegments);
  614:  
  615:             IRelativeGestureSegment[] waveLeftSegments = new IRelativeGestureSegment[6];
  616:             WaveLeftSegment1 waveLeftSegment1 = new WaveLeftSegment1();
  617:             WaveLeftSegment2 waveLeftSegment2 = new WaveLeftSegment2();
  618:             waveLeftSegments[0] = waveLeftSegment1;
  619:             waveLeftSegments[1] = waveLeftSegment2;
  620:             waveLeftSegments[2] = waveLeftSegment1;
  621:             waveLeftSegments[3] = waveLeftSegment2;
  622:             waveLeftSegments[4] = waveLeftSegment1;
  623:             waveLeftSegments[5] = waveLeftSegment2;
  624:             this.gestures.AddGesture(GestureType.WaveLeft, waveLeftSegments);
  625:  
  626:             IRelativeGestureSegment[] swipeleftSegments = new IRelativeGestureSegment[3];
  627:             swipeleftSegments[0] = new SwipeLeftSegment1();
  628:             swipeleftSegments[1] = new SwipeLeftSegment2();
  629:             swipeleftSegments[2] = new SwipeLeftSegment3();
  630:             this.gestures.AddGesture(GestureType.LeftSwipe, swipeleftSegments);
  631:  
  632:             IRelativeGestureSegment[] swiperightSegments = new IRelativeGestureSegment[3];
  633:             swiperightSegments[0] = new SwipeRightSegment1();
  634:             swiperightSegments[1] = new SwipeRightSegment2();
  635:             swiperightSegments[2] = new SwipeRightSegment3();
  636:             this.gestures.AddGesture(GestureType.RightSwipe, swiperightSegments);
  637:  
  638:             IRelativeGestureSegment[] menuSegments = new IRelativeGestureSegment[20];
  639:             MenuSegments1 menuSegment = new MenuSegments1();
  640:             for (int i = 0; i < 20; i++)
  641:             {
  642:                 //gesture consists of the same thing 20 times 
  643:                 menuSegments[i] = menuSegment;
  644:             }
  645:  
  646:             this.gestures.AddGesture(GestureType.Menu, menuSegments);
  647:         }
  648:  
  649:         /// <summary>
  650:         /// Handles the Tick event of the ChangeTimer control.
  651:         /// </summary>
  652:         /// <param name="sender">The source of the event.</param>
  653:         /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
  654:         void ChangeTimer_Tick(object sender, EventArgs e)
  655:         {
  656:             if (this.DeleteSkeleton != null)
  657:             {
  658:                 this.DeleteSkeleton(this, null);
  659:             }
  660:         }
  661:  
  662:         /// <summary>
  663:         /// Resets the timer.
  664:         /// </summary>
  665:         public void ResetTimer()
  666:         {
  667:             this.ChangeTimer.Stop();
  668:             this.ChangeTimer.Start();
  669:         }
  670:  
  671:         /// <summary>
  672:         /// Notifies all change.
  673:         /// </summary>
  674:         private void NotifyAllChange()
  675:         {
  676:             this.NotifyPropertyChanged("Head");
  677:             this.NotifyPropertyChanged("LeftHand");
  678:             this.NotifyPropertyChanged("LeftHandX");
  679:             this.NotifyPropertyChanged("LeftHandY");
  680:             this.NotifyPropertyChanged("RightHand");
  681:             this.NotifyPropertyChanged("LeftWrist");
  682:             this.NotifyPropertyChanged("RightWrist");
  683:             this.NotifyPropertyChanged("LeftElbow");
  684:             this.NotifyPropertyChanged("RightElbow");
  685:             this.NotifyPropertyChanged("ShoulderCenter");
  686:             this.NotifyPropertyChanged("HipCenter");
  687:             this.NotifyPropertyChanged("LeftKnee");
  688:             this.NotifyPropertyChanged("RightKnee");
  689:             this.NotifyPropertyChanged("LeftAnkle");
  690:             this.NotifyPropertyChanged("RightAnkle");
  691:             this.NotifyPropertyChanged("LeftFoot");
  692:             this.NotifyPropertyChanged("RightFoot");
  693:             this.NotifyPropertyChanged("Spine");
  694:         }
  695:  
  696:         /// <summary>
  697:         /// Notifies the property changed.
  698:         /// </summary>
  699:         /// <param name="propertyName">Name of the property.</param>
  700:         private void NotifyPropertyChanged(string propertyName)
  701:         {
  702:             if (this.PropertyChanged != null)
  703:             {
  704:                 this.PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
  705:             }
  706:         }
  707:  
  708:         #endregion
  709:     }
  710: }
Step 3 – Getting some sort of visual representation.

Before we do anything with the points that are coming through, the first thing to do is to simply display the camera feed from the Kinect. This is very easy to do.

In the XAML, add an image control (i.e. called “cameraFeed”). In the code, when the main window is created, set up an event handler for the Kinect’s ImageFrameReady property; and it will be called once the SDK has picked up an image.

Within the handler, create a ‘PlanarImage’ from the event args, and then set the image source of the image control to a new BitmapSource as seen below:

    1: public MainWindow()
    2: {
    3:     InitializeComponent();
    4:     //other stuff here        
    5:     model.Kinect.ImageFrameReady += new EventHandler<Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs>(kinect_ImageFrameReady);
    6: }
    7:  
    8: void kinect_ImageFrameReady(object sender, Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs e)
    9: {
   10:     PlanarImage image = e.ImageFrame.Image;
   11:     cameraFeed.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null, image.Bits, image.Width * image.BytesPerPixel);
   12: }

Now let’s do something with the actual points we have obtained.

What we wanted to do here was to create a very simple “image” of a skeleton on the screen.

One thing we did here was to disregard some points from the skeleton. We found it irrelevant to display a few points (i.e. shoulder blades, and sides of the hips etc.), as it didn’t really make the skeleton look or perform any differently, and by excluding them, improved the performance of the entire application.

So, with what remained, we used a combination of ellipses and lines to represent the skeleton.

Simple enough to start off with, add an items control to the page, and within its Data Template, place the 16 ellipses (representing each joint), and a line to connect to each set of relevant ellipses (sort of like the song…”the wrist bone is connected to the elbow”, and so forth).

Each ellipse was created as per below:

    1: <Ellipse Height="14" Name="LeftHand" Stroke="Black" Fill="{Binding JointColor, Converter={StaticResource ColorToSolidColorBrushConverter}}" Width="14" Visibility="{Binding LeftHand, Converter={StaticResource JointToVisibilityConverter}}">
    2: <Ellipse.RenderTransform>
    3: <TranslateTransform X="{Binding LeftHand.Position.X, Converter={StaticResource                KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=0}" Y="{Binding LeftHand.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=0}" />
    4:        </Ellipse.RenderTransform>
    5: </Ellipse>

The lines were also very simple:

    1: <Line StrokeThickness="3" Stroke="Black" StrokeEndLineCap="Round" 
    2: X1="{Binding LeftHand.Position.X, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=7}" 
    3: Y1="{Binding LeftHand.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=7}" 
    4: X2="{Binding LeftWrist.Position.X, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=7}" 
    5: Y2="{Binding LeftWrist.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=7}" 
    6: Fill="Black" />

By binding the ellipses to each relevant point in the Main View Model, and using the converter that we created earlier, you should obtain something similar to this:-

Untitled

The Kinect sensor has picked up the skeletons of the two people standing in front of it. By then updating the collection of skeletons and updating the bindings of the items control source, the skeletons should be displayed correctly on the screen.

Now, as the frames come in, and the bindings update, the ellipses will seamlessly move around the screen as you move, and there you have it, you’re very own living skeleton!

In our next post, we will talk about how to write a gesture service with the SDK, which will recognize the gestures the skeletons on screen are making.

Written by Michael Tsikkos and James Glading