Kinect Calibration

This program adjusts Kinect to better see the user, this way the user doesn’t have to move back or forward (unless Kinect’s angles aren’t enough to see the user). I think this is a useful thing to have when you start your Kinect application.

The code is very simple to follow. Use Kinect’s elevation angle to change the Kinect’s “view” and track the user in sight. For each angle count how many joints Kinect is tracking and save that angle as the best if we reach a new maximum number of tracked joints.

void KinectSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
	// Get skeleton information
	using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
	{
		if (skeletonFrame != null && m_skeletonData != null)
		{
			skeletonFrame.CopySkeletonDataTo(m_skeletonData);
		}
	}

	m_curTime = m_watch.ElapsedMilliseconds;

	if (m_calibrationState == CalibrateState.GoingDown)
	{
		// If the sensor reach its lowest angle, let's do a full scan from min angle to max angle.
		if (m_curTime >= WaitTime)
		{
			m_watch.Reset();
			m_watch.Start();

			m_bestAngle = m_kinect.MinElevationAngle;
			m_maxNumTracked = 0;

			m_calibrationState = CalibrateState.GoingUp;
			m_kinect.ElevationAngle = m_angles[m_curAngleIndex++];
		}
	}
	else if (m_calibrationState == CalibrateState.GoingUp)
	{
		if (m_curTime >= WaitTimeGoingUp)
		{
			m_watch.Reset();
			m_watch.Start();

			// If we scanned all the angles, lets adjust kinect to the best angle.
			if (m_curAngleIndex > m_angles.Length - 1)
			{
				m_calibrationState = CalibrateState.GoingBest;
				m_kinect.ElevationAngle = m_bestAngle;
				return;
			}

			m_kinect.ElevationAngle = m_angles[m_curAngleIndex++];
		}

		// For each skeleton, count the number of tracked joints and save the best
		// angle when the number of tracked joints is greater than previous values.
		foreach (Skeleton skeleton in m_skeletonData)
		{
			if (skeleton == null)
				continue;

			if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
			{
				//TODO: Improve algorithm by using the number of inferred joints.
				int numTracked = 0;
				int numInferred = 0;
				foreach (Joint joint in skeleton.Joints)
				{
					if (joint.TrackingState == JointTrackingState.Tracked)
						numTracked++;
					else if (joint.TrackingState == JointTrackingState.Inferred)
						numInferred++;
				}

				if (numTracked >= m_maxNumTracked)
				{
					m_maxNumTracked = numTracked;
					m_bestAngle = m_kinect.ElevationAngle;
				}
			}
		}
	}
	else if (m_calibrationState == CalibrateState.GoingBest)
	{
		// Just wait until kinect adjusts itself to match the best angle.
		if (m_curTime >= WaitTime)
		{
			m_watch.Reset();

			m_calibrationState = CalibrateState.Idle;
			m_kinect.SkeletonFrameReady -= KinectSkeletonFrameReady;

			// Reset Kinect state.
			DisableStreams();
			EnableStreams();

			// Signal that we finished the calibration.
			if (OnCalibrationComplete != null)
				OnCalibrationComplete();
		}
	}
}

You may notice two things about the code:

  • First is that I’m using time to control between Kinect movements. The reason for this is that if we try to compare Kinect with the angle we set, we might not get the same value for different reasons (Kinect sensor is not 100% accurate or it couldn’t physically rotate the sensor to that exact angle).
  • And second is the fact that I’m only scanning the skeleton for some angles. I don’t know if this is a Kinect’s limitation or if I did something wrong in the code but I couldn’t track any joints while Kinect was moving. So what I did was move the sensor between some angles and wait there a bit to count the bones for each one of those angles.

Known limitations:

  • it only supports 1 user at this moment;
  • the scanning process is not ideal.

Download:

Kinect Calibration (13Kb)

Kinect Slide Tutorial

Part of my Kinect Portfolio presentation had slides controlled by gestures. This short tutorial explains how I made it work. As I said in the other post, I used a Gesture Library to help with the gesture recognition, Kinect SDK Dynamic Time Warping (DTW) Gesture Recognition. This is a great project that is available on CodePlex and uses a cool algorithm called Dynamic Time Warping which is pretty cool for pattern recognition such as gestures or voice, among others. If you’re curious like me and want to know how this algorithm works, take a look at this link.

But onto the slide tutorial. I started by taking the core files to handle the gesture recognition and plug it in with my project. After a couple of little changes mostly to call an event handler when a gesture is recognized and to work with the final Kinect SDK 1.0., I used the original program to generate the gestures file which contains the information needed to detect gestures. I actually recorded two different gestures for each “SlideLeft” and “SlideRight” gesture just to make sure that different gestures done by different people would be caught by the algorithm.

The next step was to figure out how to animate the slides. I didn’t have experience with WPF animations but I arrived at this solution which works but might not be the best.

VisualStates and their transitions

I created 3 visual states to handle the slide left and slide right: previous (VisualState 1), current (VisualState 2) and next (VisualState 3). There are 2 visual transitions (left and right) between each 2 states and every time we change from one visual state to another we update the “next” target with new slides.

To automatize the transitions I decided to create a SlideManager that starts by reading all the “slides” (Grid elements) under a parent element and initializing the visual states by adding the event handlers. I created a simple structure to keep track of slides and transitions between slides because initially I wanted to include vertical transitions by ended up not doing them.

// Initialize slides.
for (int i = 0; i < mainSlideContainer.Children.Count; i++)
{
	Grid slide = mainSlideContainer.Children[i] as Grid;
	if(slide == null)
		continue;

	Slide s = new Slide()
				            {
				                Name = slide.Name,
				                GUIElement = slide,
				            };

	m_slides.Add(s);
}

for (int i = 0; i < m_slides.Count; i++)
{
	Slide nSlide = null;
	Slide pSlide = null;

	if (i < m_slides.Count - 1)
		nSlide = m_slides[i + 1];
 	if (i > 0)
		pSlide = m_slides[i];

	Slide s = m_slides[i];
	s.Transitions = new Dictionary();
	if(nSlide != null)
		s.Transitions.Add(Transition.RightToLeft, nSlide);
	if (pSlide != null)
		s.Transitions.Add(Transition.LeftToRight, pSlide);
}

There are two other important methods that change slides: MoveToNextSlide() and MoveToPrevSlide(). Each method checks if the current slide has the transition we want to perform, updates the current slide and visual state attribute members and calls the method that actually performs the transition.

public void MoveToNextSlide()
{
	if(m_inTransition)
		return;

	Slide nextSlide;
	if (m_curSlide.Transitions.TryGetValue(Transition.RightToLeft, out nextSlide))
	{
		m_prevSlideIndex = m_curSlideIndex;

		m_curSlideIndex = Math.Min(m_curSlideIndex + 1, m_slides.Count - 1);
		m_curSlide = m_slides[m_curSlideIndex];
		m_curVSIndex = (m_curVSIndex + 1) % m_visualStates.Count;
		m_curVS = m_visualStates[m_curVSIndex];

		m_inTransition = true;
		ExtendedVisualStateManager.GoToElementState(m_vsManagerContainer, m_curVS.Name, true);
	}
}

Another part of the code that is worth mentioning is the two event handlers at the end of this manager: Storyboard_Completed and VSGCurrentStateChanged.

private void Storyboard_Completed(object sender, EventArgs e)
{
	m_slides[m_curSlideIndex].GUIElement.Opacity = 1;
	m_slides[m_curSlideIndex].GUIElement.Visibility = Visibility.Visible;

	if (m_prevSlideIndex != m_curSlideIndex)
	{
		m_slides[m_prevSlideIndex].GUIElement.Opacity = 0;
		m_slides[m_prevSlideIndex].GUIElement.Visibility = Visibility.Collapsed;
	}
}

The visual state transitions do the slide movement and the storyboard controls the opacity and visibility of the element in that state. Whenever we change from one state to the other we have to reset the opacity and visibility properties so that the next time we do the transitions the new slides have the correct properties.

private void VSGCurrentStateChanged(object sender, VisualStateChangedEventArgs e)
{
	VisualState nvs = m_curVS;

	// update the next visual state time line (only if necessary)
	if (m_curSlideIndex < m_slides.Count - 1)
	{
		Timeline tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLOpacName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);

		tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLVisName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);
	}

	m_inTransition = false;
}

The same goes for the storyboard target name property, whenever we change to a new state we have to update the next slide. I also used a flag to prevent it from starting a new transition while we’re still doing the animation from the current transition.

The KinectManager is a simple class to handle the Kinect’s initialization that I adapted from the samples. It’s far from complete as it is but it serves well for this tutorial.

You may ask yourself why go through all this trouble when there is PowerPoint and Kinect applications to do the slide animation controls? Firstly, because it is fun to explore and learn new things and secondly because there are parts of the complete interactive portfolio that weren’t so easy to do in PowerPoint such as the particle system or the pong game.

Note: The transitions between texts were programmed in a similar way but instead of using 3 Visual States I only used 2, one for display the text and the other and to hide it, then it’s just a matter of setting the right text when you do the transitions.

Download
Kinect Slide Tutorial (requires: .Net4, Kinect SDK 1.0, VS2010 and up)

Interactive Portfolio Update

I just posted an updated version of my Interactive Portfolio using Kinect. It was updated to work correctly with the latest Kinect SDK 1.0. The application is now installed via ClickOnce which checks if you have the software requirements installed to run the app. It took a bit to figure out while ClickOnce wasn’t including some required files in the installer but it appears that there is a problem when mixing XNA and WPF projects.

You can grab the project here.

Kinect and WP7 interaction

By the same time I did my portfolio I also did another experiment that’s in this post. A few days ago I saw the KinectoPhone project and thought it would be cool to adapt it to another project I was developing with Francisco Campelo. This project allows a desktop application using Kinect to communicate with another application on the Windows Phone 7 synching its state between the two apps by communicating through a server.

I changed a bit the network code and plugged in with other code I had done for the particles and here’s the result:

Kinect and WP7 interaction

The red particles on the big screen are from Kinect’s skeleton and the blue particles are sent from the WP7 through touch screen input. On the WP7 screen, the blue particles are from Kinect’s skeleton and the red/white ones from the user’s input.

This could get many improvements, network and feature wise but I’ll leave that to some other time. I have many ideas but the time is short so I’ll be moving to another one soon.

THANKS TO

  • My parents for helping me out testing and recording the video.
  • Anabela Faria for the video editing.
  • Francisco Campelo for the original idea which led me to the Kinect version.

Interactive portfolio using Kinect

I like to explore different technologies that relate with computers and that’s why I own a Lego Mindstorms NXT . More recently I bought a Kinect to transform some ideas into real applications.

I’m currently looking for job in software development in the Toronto (Canada) area and it occurred to me that I could present my portfolio in a fun and entertaining way. So I decided to create my first Kinect application to be this interactive portfolio. Although I have some years of experience in business applications, portals, intranets, etc., I decided to compile only my experience in game development and interactive 3d projects.

If you don’t own a Kinect you can watch the video below to see how it works. I encourage you to skip the video in case you want to try the application or you’ll ruin the experience.

Interactive portfolio

The code uses the following technologies: C#, .NET, XNA and WPF. I also used two open source libraries to help with the gestures and the XNA integration with WPF.

DOWNLOAD

Software requirements:

Kinect application:

Note: Please make sure you install the software requirements or you won’t be able to install the application.

THANKS TO

  • My parents for helping me out recording the video.
  • Anabela Faria for the video editing.
  • Paulo Silva for the animated silhoutte.