Kinect Slide Tutorial

Part of my Kinect Portfolio presentation had slides controlled by gestures. This short tutorial explains how I made it work. As I said in the other post, I used a Gesture Library to help with the gesture recognition, Kinect SDK Dynamic Time Warping (DTW) Gesture Recognition. This is a great project that is available on CodePlex and uses a cool algorithm called Dynamic Time Warping which is pretty cool for pattern recognition such as gestures or voice, among others. If you’re curious like me and want to know how this algorithm works, take a look at this link.

But onto the slide tutorial. I started by taking the core files to handle the gesture recognition and plug it in with my project. After a couple of little changes mostly to call an event handler when a gesture is recognized and to work with the final Kinect SDK 1.0., I used the original program to generate the gestures file which contains the information needed to detect gestures. I actually recorded two different gestures for each “SlideLeft” and “SlideRight” gesture just to make sure that different gestures done by different people would be caught by the algorithm.

The next step was to figure out how to animate the slides. I didn’t have experience with WPF animations but I arrived at this solution which works but might not be the best.

VisualStates and their transitions

I created 3 visual states to handle the slide left and slide right: previous (VisualState 1), current (VisualState 2) and next (VisualState 3). There are 2 visual transitions (left and right) between each 2 states and every time we change from one visual state to another we update the “next” target with new slides.

To automatize the transitions I decided to create a SlideManager that starts by reading all the “slides” (Grid elements) under a parent element and initializing the visual states by adding the event handlers. I created a simple structure to keep track of slides and transitions between slides because initially I wanted to include vertical transitions by ended up not doing them.

// Initialize slides.
for (int i = 0; i < mainSlideContainer.Children.Count; i++)
{
	Grid slide = mainSlideContainer.Children[i] as Grid;
	if(slide == null)
		continue;

	Slide s = new Slide()
				            {
				                Name = slide.Name,
				                GUIElement = slide,
				            };

	m_slides.Add(s);
}

for (int i = 0; i < m_slides.Count; i++)
{
	Slide nSlide = null;
	Slide pSlide = null;

	if (i < m_slides.Count - 1)
		nSlide = m_slides[i + 1];
 	if (i > 0)
		pSlide = m_slides[i];

	Slide s = m_slides[i];
	s.Transitions = new Dictionary();
	if(nSlide != null)
		s.Transitions.Add(Transition.RightToLeft, nSlide);
	if (pSlide != null)
		s.Transitions.Add(Transition.LeftToRight, pSlide);
}

There are two other important methods that change slides: MoveToNextSlide() and MoveToPrevSlide(). Each method checks if the current slide has the transition we want to perform, updates the current slide and visual state attribute members and calls the method that actually performs the transition.

public void MoveToNextSlide()
{
	if(m_inTransition)
		return;

	Slide nextSlide;
	if (m_curSlide.Transitions.TryGetValue(Transition.RightToLeft, out nextSlide))
	{
		m_prevSlideIndex = m_curSlideIndex;

		m_curSlideIndex = Math.Min(m_curSlideIndex + 1, m_slides.Count - 1);
		m_curSlide = m_slides[m_curSlideIndex];
		m_curVSIndex = (m_curVSIndex + 1) % m_visualStates.Count;
		m_curVS = m_visualStates[m_curVSIndex];

		m_inTransition = true;
		ExtendedVisualStateManager.GoToElementState(m_vsManagerContainer, m_curVS.Name, true);
	}
}

Another part of the code that is worth mentioning is the two event handlers at the end of this manager: Storyboard_Completed and VSGCurrentStateChanged.

private void Storyboard_Completed(object sender, EventArgs e)
{
	m_slides[m_curSlideIndex].GUIElement.Opacity = 1;
	m_slides[m_curSlideIndex].GUIElement.Visibility = Visibility.Visible;

	if (m_prevSlideIndex != m_curSlideIndex)
	{
		m_slides[m_prevSlideIndex].GUIElement.Opacity = 0;
		m_slides[m_prevSlideIndex].GUIElement.Visibility = Visibility.Collapsed;
	}
}

The visual state transitions do the slide movement and the storyboard controls the opacity and visibility of the element in that state. Whenever we change from one state to the other we have to reset the opacity and visibility properties so that the next time we do the transitions the new slides have the correct properties.

private void VSGCurrentStateChanged(object sender, VisualStateChangedEventArgs e)
{
	VisualState nvs = m_curVS;

	// update the next visual state time line (only if necessary)
	if (m_curSlideIndex < m_slides.Count - 1)
	{
		Timeline tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLOpacName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);

		tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLVisName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);
	}

	m_inTransition = false;
}

The same goes for the storyboard target name property, whenever we change to a new state we have to update the next slide. I also used a flag to prevent it from starting a new transition while we’re still doing the animation from the current transition.

The KinectManager is a simple class to handle the Kinect’s initialization that I adapted from the samples. It’s far from complete as it is but it serves well for this tutorial.

You may ask yourself why go through all this trouble when there is PowerPoint and Kinect applications to do the slide animation controls? Firstly, because it is fun to explore and learn new things and secondly because there are parts of the complete interactive portfolio that weren’t so easy to do in PowerPoint such as the particle system or the pong game.

Note: The transitions between texts were programmed in a similar way but instead of using 3 Visual States I only used 2, one for display the text and the other and to hide it, then it’s just a matter of setting the right text when you do the transitions.

Download
Kinect Slide Tutorial (requires: .Net4, Kinect SDK 1.0, VS2010 and up)

More movies as code

I must admit that that Movies as Code site got me addicted Smile Here are a few other entries of mine:

Back To The Future

svn merge -r 1985:1955 .

Indiana Jones and the Raiders of the Lost Ark

void RetrieveGoldenIdol()
{
    while(true)
    {
        float dif = bag.Weight - indy.CalculateApproxWeight(idol);
        // 200g of margin should be enough.
        if(dif < -100) bag.AddSand(rand.Next(100, -dif));
        else if(dif > 100) bag.RemoveSand(rand.Next(100, dif));
        else break;
    }
    Swap(bag, idol);
    DeployTraps();
    indy.Run();
}

Jumper

void Jump()
{
    if(Seen("place"))
        goto place;
    return;
place: ;
}

These ones weren’t posted because they were similar to others on the site:

Inception

function goDeeper() {
    if(kick) {
        if(wakeFromKick)
            return;
        else
            Limbo();
    }
    goDeeper();
}

Stargate

void Stargate()
{
earth:
    goto newPlanet;

    // travelling

newPlanet:
    Explore();
    KillGoaulds();

    goto earth;
}

Back To The Future as Code

David Amador twitted about this cool site Movies as Code and I decided to make a simple entry for Back To The Future as C++. Here is my entry: http://moviesascode.net/adventure/back-to-the-future-2/

The code doesn’t appear correctly on that site so here it is the full version:

#include <future>
#include <iostream>

int main()
{
  auto future = std::async([] { return "Future"; });
  std::cout << "Back To The " << future.get() << std::endl;
  return 0;
}

For those who didn’t get it: C++11 added a new feature that is used in the sample called “future” that will hold the result of an async function somewhere in the future. It was just to play around with the name of that feature :)

Kinect and WP7 interaction

By the same time I did my portfolio I also did another experiment that’s in this post. A few days ago I saw the KinectoPhone project and thought it would be cool to adapt it to another project I was developing with Francisco Campelo. This project allows a desktop application using Kinect to communicate with another application on the Windows Phone 7 synching its state between the two apps by communicating through a server.

I changed a bit the network code and plugged in with other code I had done for the particles and here’s the result:

Kinect and WP7 interaction

The red particles on the big screen are from Kinect’s skeleton and the blue particles are sent from the WP7 through touch screen input. On the WP7 screen, the blue particles are from Kinect’s skeleton and the red/white ones from the user’s input.

This could get many improvements, network and feature wise but I’ll leave that to some other time. I have many ideas but the time is short so I’ll be moving to another one soon.

THANKS TO

  • My parents for helping me out testing and recording the video.
  • Anabela Faria for the video editing.
  • Francisco Campelo for the original idea which led me to the Kinect version.

Interactive portfolio using Kinect

I like to explore different technologies that relate with computers and that’s why I own a Lego Mindstorms NXT . More recently I bought a Kinect to transform some ideas into real applications.

I’m currently looking for job in software development in the Toronto (Canada) area and it occurred to me that I could present my portfolio in a fun and entertaining way. So I decided to create my first Kinect application to be this interactive portfolio. Although I have some years of experience in business applications, portals, intranets, etc., I decided to compile only my experience in game development and interactive 3d projects.

If you don’t own a Kinect you can watch the video below to see how it works. I encourage you to skip the video in case you want to try the application or you’ll ruin the experience.

Interactive portfolio

The code uses the following technologies: C#, .NET, XNA and WPF. I also used two open source libraries to help with the gestures and the XNA integration with WPF.

DOWNLOAD

Software requirements:

Kinect application:

Note: Please make sure you install the software requirements or you won’t be able to install the application.

THANKS TO

  • My parents for helping me out recording the video.
  • Anabela Faria for the video editing.
  • Paulo Silva for the animated silhoutte.

SilverMenu library for WP7

Inspired by the menus of WP7 game Puzzle Jumble, I decided to write a library that would help me and others to build menus for WP7 games easily. The Puzzle Jumble’s GUI code is a bit of a mess because it wasn’t thought for this kind of interface from the beginning so I decided to write SilverMenu from scratch.

Puzzle Jumble’s GUI

 

SilverMenu aims to bring some of the Silverlight user interface that is now present on WP7 to the XNA. To make it simple to use you can build simple animated menus with just a few lines of XML. It has only one week of development but you can see some progress already:

First SilverMenu sample.

 

With time I hope it grows into something more complete and useful. I’ll still update my blog about this project but if you want to follow the project closely and keep up to date, as well as, download the last build, you can do it at http://silvermenu.codeplex.com. I hope you find it useful and I appreciate all the feedback I can get. Thanks!

Windows Phone 7 – Tips for programming games

Puzzle Jumble started as an experiment to test and learn a little bit about XNA and Windows Phone 7. I decided to write some tips to help those who like me started experimenting with XNA and WP7.

Stencil

The stencil was used to cut the hole for doors so that pieces would appear to be sliding over the doors but under the board. The stencil was active in every frame which lowered the frame rate quite a lot (18-20 FPS). The solution was to activate the stencil only when a piece is going to leave through a door, so be careful when using the stencil buffer on the WP7.

DXT

GPUs like the DXT format and the WP7 is no exception, but unlike desktop’s GPUs the WP7 processor only loads DXTs that are power of two. My regret is not using DXT from the beginning when I started working on Puzzle Jumble. When I changed all my textures to DXT I had to change a lot of code, all the draw calls where I was using a position to draw a sprite had to be changed to a rectangle because the textures had a different resolution (power of two).
But the change was very useful:
– I gain some FPS;
– the memory footprint was reduced from 95MB (total peak) to about 45MB;
– the XAP file was reduced from 25MB to 9MB;
– thus, as a consequence reduced the loading times.

Floats (culture)

When parsing floats it’s easy to add bugs in your application if you ignore the culture format. Make sure to use the correct format provider or at least use the culture independent System.Globalization.NumberFormatInfo.InvariantInfo.

Float precision

Another problem that I found when using floats was that this comparison: if(a < float.Epsilon) (when a = 0) would work everywhere except on the WP7 device itself where it would always return false.

HTML page requests

This is not a WP7 specific problem but I thought it was worth mentioning it. Since Microsoft didn’t provide us with an API for leader boards I implemented an online score system using our servers to keep track of the users’ scores and best times.
The system was working fine to post data online but there was a problem when retrieving it. The first time the data was retrieved everything was fine but the subsequent retrievals would always be the same as if the data was cached somewhere.
After searching for a while I found out that the problem was that the HTTP request was being cached since the URL didn’t change, it would cache the result for that URL. The solution was to add a random attribute to the URL so that the the request couldn’t be cached. Not pretty but it was the only way I found to make it work.

And those were the problems that I found during the Puzzle Jumble’s development phase. I hope this useful for you!

VSX: How to detect when Property Pages (Project Properties) window is closed

I posted a thread on the MSDN forums to seek for help in detecting when the property pages window (accessible through project->properties) in Visual Studio is closed. You can follow up the thread here: Detect when project properties window is closed. Ziwei Chen was really helpful and could find me a way to achieve this in C# projects but to my surprise, to achieve the same results for C++ projects proved to be harder than I thought.

I decided to write this short tutorial to help those who have the same problem in future.

First of we need to detect when the user click in any project->properties menu selection. Put this code where you initialize your add-in:

m_commandEventsAfterExecuteProperties1 =
    m_applicationObject.DTE.Events.get_CommandEvents("{5EFC7975-14BC-11CF-9B2B-00AA00573819}", 396);
m_commandEventsAfterExecuteProperties1.AfterExecute +=
    new _dispCommandEvents_AfterExecuteEventHandler(CommandEvents_AfterExecuteProperties);

m_commandEventsAfterExecuteProperties2 =
    m_applicationObject.DTE.Events.get_CommandEvents("{5EFC7975-14BC-11CF-9B2B-00AA00573819}", 397);
m_commandEventsAfterExecuteProperties2.AfterExecute +=
    new _dispCommandEvents_AfterExecuteEventHandler(CommandEvents_AfterExecuteProperties);

These command events ids were found using the technique described in this post: How to find Visual Studio command bars. When registering the event handlers be careful with this: Visual Studio .NET events being disconnected from add-in. So declare these as member attributes and not as method variables:

private CommandEvents m_commandEventsAfterExecuteProperties1;
private CommandEvents m_commandEventsAfterExecuteProperties2;

Now, lets dig into some rather awful code to get our property pages handle. Property pages in .NET projects are document windows but, unfortunately, if we try to call the ActiveDocument property directly through the ActiveEnvDTE library, it will throw an exception. So, with the precious help of Chen, this is the code came up with:

// For C# projects the property pages is a document window, lets try to find it.
// I was unable to get a handle using the EnvDTE library which could have eased things a bit.
bool propertyPagesFound = false;

IVsWindowFrame[] frames = new IVsWindowFrame[1];
uint numFrames;

IEnumWindowFrames ppenum;
m_uiShell.GetDocumentWindowEnum(out ppenum);

while (ppenum.Next(1, frames, out numFrames) == VSConstants.S_OK && numFrames == 1)
{
	m_frame = frames[0] as IVsWindowFrame;
	object title;
	m_frame.GetProperty((int)__VSFPROPID.VSFPROPID_Caption, out title);

	// I really don't like the way I'm using to retrieve the property pages handle.
	//TODO: CHANGE IF BETTER SOLUTION IS FOUND.
	if ((title as String).ToLowerInvariant() == GetStartupProject().Name.ToLowerInvariant())
	{
		propertyPagesFound = true;
		((IVsWindowFrame2)m_frame).Advise(this, out m_cookieWindowFrameNotify);
	}
}

First, all document windows are retrieved to the ppenum variable and next we go one by one and test its caption name to see if it’s equal to the project’s name. After we got our desired window frame we register the IVsWindowNotifyFrame event handler.

In the C++ case, things are a bit different. The property pages window isn’t an instance of document or frame, it’s a dialog window. To catch the dialog window events we use an auxiliary class that listens for the destroy message in the WndProc method.

// If we didn't find any property pages before then we need to go and search for a C++ one since they're
// different from C# property pages.
if (!propertyPagesFound)
{
	//NOTE: This might not work well if we happen to have two Visual Studio instances with the same
	//      project names.
	//TODO: TRY TO FIND A WAY TO FIX IT.
	//HACK: Go through all windows to find the project properties dialog window that matters.
	FindWindowLike.Window[] list =
	    FindWindowLike.Find(IntPtr.Zero, GetStartupProject().Name.ToLowerInvariant() + " property pages", "");
	
	// Release any handle we might have.
	if (m_subclassedMainWindow != null)
	{
		m_subclassedMainWindow.ReleaseHandle();
		m_subclassedMainWindow = null;
	}

	if (m_subclassedMainWindow == null && list != null && list.Length >= 1)
	{
		m_subclassedMainWindow = new SubclassedWindow(list[0].Handle);
	}
}

Apart from that FindWindowLike that we’ll check later, I think the code is pretty simple to understand. I really don’t like to have to search through all windows in the system but apparently the dialog window couldn’t be found under the dte’s MainWindow.HWnd hierarchy.

 

As for the event implementations, here is the C# first:

public int OnShow(int fShow)
{
	switch (fShow)
	{
		case (int)__FRAMESHOW.FRAMESHOW_TabDeactivated:
			// Property pages lost focus, put your code here!
			((IVsWindowFrame2)m_frame).Unadvise(m_cookieWindowFrameNotify);
			m_cookieWindowFrameNotify = 0;
			break;

		default:
			break;

	}

	return VSConstants.S_OK;
}

We need to implement the IVsWindowFrameNotify interface and in the OnShow you can check when the tab is closed. I used the FRAMESHOW_TabDeactivated because it is called when the tab loses focus and also when the tab closes.

Now for the C++ version:

/// 
/// Auxiliary window to intercept messages sent to the property pages dialog window.
/// 
public class SubclassedWindow : NativeWindow
{
	private const int WM_NCDESTROY = 0x0082;

	public SubclassedWindow(IntPtr hWnd)
	{
		AssignHandle(hWnd);
	}

	protected override void WndProc(ref Message m)
	{
		if (m.Msg == WM_NCDESTROY)
		{
			// Property pages window was closed, put your code here!
		}
		base.WndProc(ref m);
	}
}

As explained before, in this case we have an auxiliary window to catch the destroy event for the dialog window.

I found this script over at FindWindowLike for C# and changed it a bit to fit my needs. I modified the code that compared the window title; created a new class and changed p/invoke methods’ signature just to keep VS quiet about some warnings.

/// <summary>
/// Now we have to declare p/invoke methods in a class called NativeMethods (or similar) to
/// shut up some VS warnings.
/// </summary>
internal static class NativeMethods
{
	[DllImport("user32")]
	internal static extern IntPtr GetWindow(IntPtr hwnd, int wCmd);

	[DllImport("user32")]
	internal static extern IntPtr GetDesktopWindow();

	[DllImport("user32", EntryPoint = "GetWindowLongW")]
	internal static extern int GetWindowLong(IntPtr hwnd, int nIndex);

	[DllImport("user32")]
	internal static extern IntPtr GetParent(IntPtr hwnd);

	[DllImport("user32", EntryPoint = "GetClassNameW", CharSet = CharSet.Unicode)]
	internal static extern int GetClassName(IntPtr hWnd, [Out] StringBuilder lpClassName, int nMaxCount);

	[DllImport("user32", EntryPoint = "GetWindowTextW", CharSet = CharSet.Unicode)]
	internal static extern int GetWindowText(IntPtr hWnd, [Out] StringBuilder lpString, int nMaxCount);
}

// FROM: http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_21611201.html
// Just changed the way it checked strings.
internal class FindWindowLike
{
	internal class Window
	{
		public string Title;
		public string Class;
		public IntPtr Handle;
	}

	private const int GWL_ID = (-12);
	private const int GW_HWNDNEXT = 2;
	private const int GW_CHILD = 5;

	public static Window[] Find(IntPtr hwndStart, string findText, string findClassName)
	{
		ArrayList windows = DoSearch(hwndStart, findText, findClassName);

		return (Window[])windows.ToArray(typeof(Window));

	} //Find


	private static ArrayList DoSearch(IntPtr hwndStart, string findText, string findClassName)
	{
		ArrayList list = new ArrayList();

		if (hwndStart == IntPtr.Zero)
			hwndStart = NativeMethods.GetDesktopWindow();

		IntPtr hwnd = NativeMethods.GetWindow(hwndStart, GW_CHILD);
		while (hwnd != IntPtr.Zero)
		{
			// Recursively search for child windows.
			list.AddRange(DoSearch(hwnd, findText, findClassName));

			StringBuilder text = new StringBuilder(255);
			int rtn = NativeMethods.GetWindowText(hwnd, text, 255);
			string windowText = text.ToString();
			windowText = windowText.Substring(0, rtn);

			StringBuilder cls = new StringBuilder(255);
			rtn = NativeMethods.GetClassName(hwnd, cls, 255);
			string className = cls.ToString();
			className = className.Substring(0, rtn);

			if (NativeMethods.GetParent(hwnd) != IntPtr.Zero)
				rtn = NativeMethods.GetWindowLong(hwnd, GWL_ID);

			if (windowText.Length > 0 &&
				windowText.ToUpperInvariant().Contains(findText.ToUpperInvariant()) &&
				(className.Length == 0 ||
					className.ToUpperInvariant().Contains(findClassName.ToUpperInvariant())))
			{
				Window currentWindow = new Window();

				currentWindow.Title = windowText;
				currentWindow.Class = className;
				currentWindow.Handle = hwnd;

				list.Add(currentWindow);
			}

			hwnd = NativeMethods.GetWindow(hwnd, GW_HWNDNEXT);
		}

		return list;
	} //DoSearch
} //Class

And that’s it! I’d really appreciate if someone could tell me how to improve some of this code, mostly the parts where I have TODOs and/or HACK.
Thank you and I hope you find this useful!

Thanks to:

I wanted to thank Ziwei Chen for the excellent help at the forums.

References

VSSDK IDE Sample: Combo Box
Detect when project properties window is closed
I need FindWindowLike for C#
How to find Visual Studio command bars
Visual Studio .NET events being disconnected from add-in

Links:

Visual Studio Extensibility Forum
VSX Home on Code Gallery
Visual Studio on MSDN

Monkey Island 2 Special Edition

This weekend I “replayed” the special edition of this great game nobody has heard of, Monkey Island 2 Special Edition: LeChuck’s Revenge. It was a great pleasure to play this game in full HD with beautiful art. The new sounds and music were also great and immersive, although the iMuse system didn’t have the same smooth transitions as the old version (as expected), they did a pretty good job in this version.
The developer commentary are very welcome but most unfortunately short.

Here are a few glitches and surprises I found while playing this version:

Sam and Max (in old version)

Sam and Purple Tentacle (in new version)

Guybrush looking at Manny :)

I had read on the internet that some people complained about these kind of problems mostly on Xbox360. I don’t know if those are glitches in the graphics system or if the artists didn’t paint every pixel they should have paint. In the wharf of Phatt Island there’s even a bunch of pixels that stay on top Guybrush in the middle of the street.
Also, the waterfall sound got stuck for some reason and it was playing in every room. I had to save and load the game to make the sound go away.
One other bug that I found is when you steal the monocle from Wally, if you look closely, the animation frames still show Wally with the monocle every time he looks down. Interesting is the fact that this is also a glitch in the old version of the game but much harder to spot due to the graphics resolution. My guess is that it was to spare a few more Kilobytes for other animations.
Another annoying bug was the cursor when trying to play with the XBox360 game pad. I couldn’t move the cursor because it would jump to the previous mouse cursor position.

And before I go, I also found a major bug in The Secret of Monkey Island Special Edition a while ago, here are the shots:

But nonetheless, they are both great remakes, specially Monkey Island 2 Special Edition. I’ll be back when the true Monkey Island 3 comes out.

Don’t be nervous

Reading an interesting article about comedy in games at GDC I found something rather interesting and wanted to share it with you:

Tim: When we started on Monkey Island, I don’t know about Dave [Grossman], but I thought we were writing the temporary dialogue for that game. Cause we were really new, and there was a big company there with Lucasarts and George Lucas and everyone. So I was sure they’d have professionals come and write the dialogue. So we were just kind of goofing around and writing really silly dialogue. And it took the pressure off us cause we didn’t sit there and wonder, “Is this good enough?” We were just making each other laugh in the office. And then, as it went on, Ron said, “No, no, no. This is the dialogue for the game.” And I was like, “Oh, god.”

And we never wrote anything funny after that.