Mark Gilbert's Blog

Science and technology, served light and fluffy.

Kinect in the Abstract: Working with the Sealed SkeletonData and JointsCollection classes

My latest side project involving the Kinect started to get a bit hairy.  The logic for what we were trying to do was at least an order of magnitude greater than the Target Tracking system my colleagues and I built last year.  It functioned, but it was getting exponentially more difficult to add features to it, let alone debug it.

So, suffering from a lull in my regular project work over the holiday break, I decided to start building some unit tests for it.  If nothing else, having a solid test suite would allow me to regression-test the application whenever I monkeyed with the code, and THAT would enable some good-sized refactorings that were long overdue.  My first task, then, was to figure out how to mock out the data coming off the Kinect.  My first task quickly hit a wall.

The application uses the SkeletonData object available in the SkeletonFrameReady event.  My original event handler looked something like this:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{

    List<SkeletonData> ActiveSkeletonsRaw;
    SkeletonFrame allSkeletons = e.SkeletonFrame;

    ActiveSkeletonsRaw = (from s in allSkeletons.Skeletons
                          where s.TrackingState == SkeletonTrackingState.Tracked
                          select s).ToList();

    this._MyManager.UpdatePositions(ActiveSkeletonsRaw);
}

The UpdatePositions() method would handle moving the objects around based on the new positions of the skeletons/joints, and that was the primary method I wanted to test.  I figured if I could create my own SkeletonData object, and pass that into UpdatePositions, I could test any scenario I wanted.  Unfortunately, the SkeletonData class is sealed, and there aren’t any public constructors on it.  So, I went the route of writing my own version of SkeletonData ā€“ one that I could create objects from, and would effectively function the same as SkeletonData:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Research.Kinect.Nui;


public class SkeletonDataAbstraction : ISkeletonData
{
    public JointsCollectionAbstraction Joints { get; private set; }
    public Vector Position { get; set; }
    public SkeletonQuality Quality { get; set; }
    public int TrackingID { get; set; }
    public SkeletonTrackingState TrackingState { get; set; }
    public int UserIndex { get; set; }


    public SkeletonDataAbstraction() 
    {
        this.InitializeJoints();
    }
    public SkeletonDataAbstraction(Microsoft.Research.Kinect.Nui.SkeletonData RawData) : this()
    {
        foreach (Joint CurrentJoint in RawData.Joints)
        {
            this.UpdateJoint(CurrentJoint);
        }
        
        this.Position = RawData.Position;
        this.Quality = RawData.Quality;
        this.TrackingID = RawData.TrackingID;
        this.TrackingState = RawData.TrackingState;
        this.UserIndex = RawData.UserIndex;
    }

    private void InitializeJoints()
    {
        this.Joints = new JointsCollectionAbstraction();
        foreach (JointID CurrentJointID in Enum.GetValues(typeof(JointID)))
        {
            this.Joints.Add(new Joint()
                                        {
                                            ID = CurrentJointID,
                                            Position = new Vector() { X = 0.0f, Y = 0.0f, Z = 0.0f, W = 0.0f },
                                            TrackingState = JointTrackingState.NotTracked
                                        });
        }
    }

    public void UpdateJoint(Joint NewJoint)
    {
        this.Joints[NewJoint.ID] = new Joint() 
                                                { ID = NewJoint.ID,
                                                  Position = new Vector()
                                                                            { X = NewJoint.Position.X,
                                                                              Y = NewJoint.Position.Y,
                                                                              Z = NewJoint.Position.Z,
                                                                              W = NewJoint.Position.W
                                                                            },
                                                  TrackingState = NewJoint.TrackingState
                                                };
    }
}

When the class is instantiated, the Joints collection is also instantiated with a "blank" Joint object for every joint defined by the Kinect (the complete list is defined by the Microsoft.Research.Kinect.Nui.JointID enumeration).  Then, the UpdateJoint method is called to overwrite those blank joints with the real values.  I also use this method in the unit tests to precisely place the joints I was interested in, just before running a given test.

I thought I would end up needing to mock out portions of the class, so I created an interface for it as well:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Research.Kinect.Nui;

public interface ISkeletonData
{
}

As it turns out, I didn’t need to mock anything out ā€“ I can just create SkeletonDataAbstraction classes, and pass them directly into UpdatePositions.  I decided to keep the interface around, just in case I later found something that required a mock.

I also needed to be able to construct a JointsCollection object (what the SkeletonData.Joints property is defined as), but that was also marked sealed with no public constructors.  So, I created a JointsCollectionAbstraction object for it:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections;
using Microsoft.Research.Kinect.Nui;
using System.Collections.ObjectModel;
using System.ComponentModel;

public class JointsCollectionAbstraction : List<Joint>, IEnumerable
{

    public Joint this[JointID i]
    {
        get
        {
            return this[(int)i];
        }
        set
        {
            this[(int)i] = value;
        }
    }


}

After putting these together, I rewrote my original application code using the new abstraction layer, to make sure I had captured everything I needed to:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) 
{

    List<SkeletonData> ActiveSkeletonsRaw;
    SkeletonFrame allSkeletons = e.SkeletonFrame;

    ActiveSkeletonsRaw = (from s in allSkeletons.Skeletons
                          where s.TrackingState == SkeletonTrackingState.Tracked
                          select s).ToList();

    List<SkeletonDataAbstraction> ActiveSkeletons;
    ActiveSkeletons = new List<SkeletonDataAbstraction>();
    foreach (SkeletonData CurrentSkeleton in ActiveSkeletonsRaw)
    {
        ActiveSkeletons.Add(new SkeletonDataAbstraction(CurrentSkeleton));
    }

    this._MyManager.UpdatePositions(ActiveSkeletons);
}

That worked like a charm.  With each SkeletonFrameReady event-raise, I copy the key pieces of information from the Kinect over to my own structures, and use those from that point on.  Now the task of writing tests around this could begin in earnest.  I wrote a "CreateSkeleton" method for my unit tests that would encapsulate setting one of these up:

private SkeletonDataAbstraction CreateSkeleton(SkeletonTrackingState NewTrackingState, int NewUserIndex)
{
    SkeletonDataAbstraction NewSkeleton;

    NewSkeleton = new SkeletonDataAbstraction();
    NewSkeleton.Position = new Vector();
    NewSkeleton.Quality = SkeletonQuality.ClippedBottom;
    NewSkeleton.TrackingID = NewUserIndex + 1;
    NewSkeleton.TrackingState = NewTrackingState;
    NewSkeleton.UserIndex = NewUserIndex;

    NewSkeleton.UpdateJoint(new Joint()
                                        {
                                            ID = JointID.HandLeft,
                                            Position = new Vector() { X = X_WHEN_HAND_MOVES_AWAY, Y = this._OriginalY, Z = this._OriginalZ, W = this._OriginalW },
                                            TrackingState = JointTrackingState.Tracked
                                        });
    NewSkeleton.UpdateJoint(new Joint()
                                        {
                                            ID = JointID.HandRight,
                                            Position = new Vector() { X = X_WHEN_HAND_MOVES_AWAY, Y = this._OriginalY, Z = this._OriginalZ, W = this._OriginalW },
                                            TrackingState = JointTrackingState.Tracked
                                        });

    // Other joints overwritten here...

    return NewSkeleton;
}

(Note, the values for X_WHEN_HAND_MOVES_AWAY, _OriginalY, _OriginalZ, and _OriginalW are merely floats, defined specific to the application.)

Now I could easily create a list of Skeletons to track, with joints positioned just so, and pass that structure into UpdatePositions.


After I had most of this built out, I found a couple of other posts from people doing essentially the same thing:

The first one is an interesting forum post where one of the Microsoft guys admits that declaring the SkeletonData and other classes Sealed was probably not the brightest idea.

Thankfully, the wall I hit ended up coming only up to my knees, so after a few bumps and bruises I was over it.

Advertisements

January 10, 2012 - Posted by | Microsoft Kinect, Visual Studio/.NET

Sorry, the comment form is closed at this time.

%d bloggers like this: