Combining experimental tech during the Nordic Game Jam

Just a short while ago the Nordic Game Jam 2015 wrapped up, concluding a truly spectacular event with over 700(!) people making over 150(!) games. I was a programmer on a team of seven, comprised mainly of students. We took this event as a chance to work with some stuff we've never worked with before. An opportunity to do something weird, new and exciting. The concept we went with used two gadgets we borrowed on-site; a Tobii EyeX eye tracker and a DreamOC HD3 hologram display. We were very pleased with the tech and the results, and even won the Tobii Tech award for best use of the Tobii eye tracker. In this blog post I would like to highlight one particularly interesting problem we encountered and solved: how to combine these two pieces of hardware.

The tech

Our eye was drawn to a piece of hardware called the DreamOC HD3 early on in the event. The DreamOC is a hologram display case made by RealFiction using HDMI input to project images into “reality”. If you click the link you will see some photos which show how it looks. I've added some (programmer art) diagrams to roughly show how it works.

Essentially what you have is a monitor placed above a pyramid of partially see-through mirrors. The monitor projects HDMI input (for example from a laptop) onto the mirror. If the monitor is black at a certain point, the mirror does not reflect anything and you can see through it to the other side. If you place an object inside of the pyramid, the projected images get overlaid. Even with some simple illusory tricks you can get some great results using this setup.

Mapping the different sides to their corresponding positions was not that difficult, aside from some mirroring/rotation corrections. We decided to have a game with roughly the same gameplay area on every side. Using Unity's render-to-texture system and some correctly placed planes, the gameplay area can be projected onto every side of the pyramid.

Likewise, the eye tracker was also very easy to set up. The Unity plug-in and example projects worked quickly and the eye tracking was really quite precise. The eye tracker is a small black bar, placed underneath your screen (so for a laptop in between your screen and your keyboard). You sit in front of it, calibrate it once and the eye tracker will be able to gather where you are looking on the screen. It did not take long to get a cursor on screen that followed my eyes.

The problem

You may be wondering then what the problem we encountered was. If both devices worked accurately, where is the issue? Well, the issue lies in the calibration. The Tobii eye tracker is set up using a special program which makes you look at all the corners of the screen. To use the eye tracker with the DreamOC, it has to be calibrated to a person behind a laptop and then moved to the front of the hologram display. The problem is that the laptop screen does not correspond with the area you will be looking at on the display case. This area is smaller, rotated and mirrored. Furthermore, the eye tracker might not be on the same angle as it was in front of the laptop.

We had to find a way to translate the eye position from a non-existing screen to a new area. Briefly we thought of doing the calibration while the eye tracker was on the display case, but the calibration program could not be used for anything except the full screen. We also thought of directly making a laptop screen position to a rectangle on screen, but with people moving in between sessions and the eye tracker not guaranteed to be at the same position or angle, this would lead to shifted positions.

However, with this idea we did find the seed for our final solution. While directly mapping to a static box would not work, we could use a dynamic box for good results. The box can be created by having the player look at four points on your gameplay area in sequence. The coordinates that you get from the eye tracker together describe a box. Just mirror and rotate this box for the correct side and you're done! As long as the player is keeping his eyes on the target, and the samples are averaged over a certain time, a fitting box can be established. A few script additions later and we had a cursor following our gaze.

Now, there were still some issues left after applying this approach. For one thing, the second calibration used filtered results through the plug-in, so we had to redo it sometimes. Aside from that, the chain of calibrate → move eye tracker → calibrate again could lead to inaccuracies. We also had issues with getting someone's head in the correct position for the eye tracker to work after moving it. However, by applying a light smoothing to the eye input positions we got good enough results for our game design. We have a space ship which follows the eye cursor of one player, so it does not matter if the cursor is sometimes a bit off. Players quickly adjusted and seemed to have a lot of fun with this new way of playing a game.

Some extra stuff

One thing we noticed was that people were really eager to be tricked by the device. Halfway through the weekend we placed a piece of styrofoam, cut and painted to look like a rock, inside of the pyramid. The projection of the space ships on top of it was a hit with players and onlookers alike. Having a rotating backdrop was also really nifty.

We also entertained the idea of having multiple eye trackers connected, for multiple players. However, because of the way eye trackers are generally used, the devices did not easily support a second eye tracker on the same device. One way to solve that was to have a separate laptop connected to the second eye tracker and send over the data to the second laptop. However, by the time we started experimenting with this solution (with the generous help of the Tobii guys at the event), we did not have confidence we could finish that ánd the game in time. 

Final thoughts

I had a blast at the Nordic Game Jam. I got the chance to work on some stuff that I would normally never get the chance to, which led to interesting challenges. Aside from the problem listed above, there were many little and larger obstacles to tackle during the project. Some of them we solved, like this one, some of them we had to avoid. The game we made was a very conventional multi player space shooter, aside from the input and the display. This allowed us to spend more time on the tech and aesthetics we were so enthusiastic about. However, I do think we could have done even more with the available gadgets and really explore their uses, if we had gone for a more elaborate game concept. Overall though, I am very satisfied with the experience and I am looking forward to next year.