Postcard From SGS 2005: Inside The Institute for Creative Technologies

Presented on the first day of Serious Games Summit 2005, this fascinating session explained some of the work carried out by the University of Southern California-connected Institute for Creative Technologies (ICT), a university-affiliated research center which is majorly funded by the U.S. Army. It is particularly known in the video game community for having produced the Pandemic co-created Full Spectrum Warrior game for the Xbox, originally made for army training on the go, but then released as a successful THQ-published game, but it also does a great deal of other bleeding-edge research for the Army, DARPA, and other government entities here and abroad on the intersection of entertainment aand technology.

Thousand-Yard Stare

The lecture started with David Wertheimer, managing director of the ICT, who started by discussing the history of the Institute. It traced its roots to a study in 1997, made by the National Research Council, and linking the entertainment and defense industries - it was felt that the entertainment industry had something to bring to defense, and vice versa. Thus, the Department of Defense (DoD) and particularly the Army wanted to put together a vehicle to allow the government to act on this.

Therefore, the ICT was set up in 1999 as a university-affiliated research center, which is managed by RDECOM/STTC. It has multiple goals, but in particular focuses on immersive VR for simulation/training, research into AI, graphics, and sound, and the need to build prototype applications for cognitive decisionmaking, as a bridge between the DoD and the filmed entertainment community and entertainment software industry.

As well as major products such as Full Spectrum Warrior (produced for the U.S. Army), and strategy title Full Spectrum Command (initially produced for the Singapore Army), Wertheimer points out that there have been some interesting consequences of the ICT's reputation as an interface with the entertainment industry, since the DoD has come to them to help with concept development on the future of army technology, and to help Congress understand potential for potential funding purposes. He gave the example of the DARPA-funded film "Nowhere To Hide," produced by the ICT.

Drilling Down

Nonetheless, the majority of ICT's main research is into subjects such as artificial intelligence, which isn't necessarily covered in the way that the Army would need it in video games. For example, Wertheimer noted, in a first-person shooter, every person is a shooter, and AI is more often twitch-based, or strategy games are highly scripted - there's nothing wrong with this, but ICT is looking to develop cognitive decision-making for training purposes, and this requires simulated people to people interaction.

Another important area of research is graphics, particular elements such as dynamic real-time lighting and HDR (high dynamic range) technology, recently implemented in Half-Life 2's Lost Coast expansion, using techniques partly pioneered by graphics guru Paul Debevec, who works at ICT. Wertheimer noted that cutting-edge PC graphics cards are fueling the growth, and art complexity is becoming increasingly important, commenting that Electronic Arts mentioned to him that last year, for the first time ever, the number of graphics files created for their games exceeds the lines of code.

This problem doesn't just exist in game industry, hence ICT's research, since Wertheimer noted that 20% of a CG movie's $100 million budget is spent on lighting. If ICT can solve these problems in an automated way, it will mean a huge difference in our ability to recreate the world as we know it, which is as important for military simulation as it is for entertainment use.

Wertheimer then cued a movie created after ICT graphics staff went to the Parthenon in Greece, and scanned it in extreme detail with a laser scanner. The movie, directed by Paul Debevec, is extremely impressive, and almost indistinguishable from the real thing, even replacing figures removed from the Parthenon that became the much-debated Elgin Marbles, and finally, showing the roof replaced and the Parthenon re-colored as it would have been in ancient days. As Wertheimer pointed out, to train people for real-world activities, the feeling of being there is incredibly important.

On Virtual Humans

The lecture then went on to discuss the concept of virtual humans, particularly important to ICT, since it wants ways to train the military to interact with other people, without lots and lots of actors being paid to play those people. Wertheimer then showed an example of virtual humans in action, an exercise using the Unreal Tournament 2004 game engine in which a soldier has to persuade a Middle Eastern doctor to relocate his clinic without revealing his operational plans. This demonstration, shown in movie form, showcases a number of technologies, including speech recognition, task and domain reasoning, dynamic gesture recognition, natural language understanding, dialog management, and emotional modeling. It's also pointed out that all of the AI systems were talking to an ICT-designed middleware layer which itself is integrating into a game engine, allowing for much greater usability.

Wertheimer then posted the question - how do we build systems to make people play and learn, and make things better for both the military and entertainment sectors? He particularly referenced ICT's Full Spectrum Warrior, which has you in-game and issuing orders, and was developed as a training product, but looks visually somewhat like a first-person shooter, and has since found success in the game market. In the end, it was claimed, the military got the training tool for less than the cost of the product, thanks to the commercial release of the game, and now THQ is porting a version to the PC, the Army is getting that version for free - effectively a win/win situation, according to ICT staff.

ICT representatives also ran through a number of other projects it had taken on in the "serious game" arena, including a "self-directed learning Internet module" which allowed people (presumably military personnel) to train to be more aware of threats (apparently Iraqi-related ambushes, though Wertheimer could not directly state this), all completed on a 90-day development schedule. Also shown in movie form was a location-based entertainment system in a room modeled after a Middle Eastern town, featuring helmets and binoculars with LCD displays, so that trainees could look out of a virtual window and see the same view enlarged in the binoculars, and train for deployment without actually being there.

Lent's Concepts

The lecture was then handed over to Dr. Mike van Lent, the research project leader at ICT, who took the second half of the lecture, and talked a little more about the serious game community and his vision for it, covering some of the same touchpoints as Doug Whatley's keynote speech earlier on Monday, but also expanding on many of them.

Lent's first point is that "serious games" fit more into a community than an industry - he counted 43 case studies but just 25 methodology studies as being presented at the Serious Games Summit this year. His concern is that developing serious games is still an art, and seen as a bit of a black art - each effort is a gamble, and each title is a one-off. Sure, it's not a science, but van Lent suggested that there aren't many developers dealing with serious games as a long-term thing, developing both serious gaming sequels and a larger infrastructure.

Another important point that was made is that the researchers/developers and the funders on serious games are often coming from very different angles - game designers are extremely tech savvy, but some of the military commissioners on products ICT works on are self-described "air-assault typists," using one finger at a time from on high to type out sentences, and there needs to be plenty of careful discussion between the two sides.

Further Ruminations

Van Lent went on to echo a comment made more than once at SGS, though it's far from sure that it's actually possible - that developers need to move towards a "science" for serious games, and need an industry-wide baseline technique. In particular, it was argued, developers need to avoid inconsistent deliverables for all customers, wasting resources, and most of all, becoming a fad (van Lent worries about what happens when games aren't "hot" any more as an emerging media, though he believes that serious game creators can overcome this in the long-term.)

The ICT employee also touched on what he thinks is particularly important, new business models for serious games. While making this point, he mentioned again that not many serious games/developers think about sequels. As in the Full Spectrum franchise, ICT believes that developing a series of games for a customer can be much more valuable than a one-off.

Also discussed was the toolkit of available technologies to ICT, and the way they try to abstract the AI side from the graphics side. For example, the previously mentioned Virtual Human is built in the Unreal engine, whereas another project, JFITS is made in the Gamebryo engine. Thus, the organization is trying to build an abstracted layer to make all its serious games on.

Adaptive AI Issues

Finally, Van Lent talked a little about a project that was important to him - the concept of adaptive AI. In this case, this means that the AI would adapt to the trainee's scenario history in ICT's work. Generally, game AI is designed so that "gaming the game," finding loopholes to get past, is part of the fun for the user, van Lent suggested, citing conversations he had with the Halo AI designer at Bungie. But, he suggested, if you learn one set of experiences that will beat the AI, in training, that's no help for real opponents.

Thus, ICT is working on military training software that adapts to the trainee's training needs, and tailors meta-goals to the trainee. They believe that the fixed AI model will be out of date quickly, suggesting that "automated behavior generation is the long-term solution." Citing the JFETS 05 example, which includes AI-controlled bad guys and good guys, ICT has rigged this so that if the trainee does smart things, bad guys may choose to lose. In fact, the whole simulation has variable behavior, but previous efforts have varied individual behavior, whereas new work is allowing the players in the simulation to coordinate their variable behavior, making for much richer variations.

Van Lent's central point was that winning isn't everything - most AI opponents focus on trying to win. A great example of this is chess computer Deep Blue, which is, according to him, just not that much fun. ICT's goal for the bad guy AI in its training software is, simply: "How can I achieve this training goal while appearing to be trying to win this scenario?" Van Lent then showed a Pepsi ad starring chess champion Garry Kasparov, in which machines ganged up on Kasparov after he beat a Deep Blue clone, reinforcing the point that machines or machine AI working together makes for the most effective training.

Conclusion

Van Lent concluded by announcing that ICT is intending to host a workshop on how to create a concrete pipeline for making serious games, hopefully helping to contribute its expertise to the issue and leveraging solutions that can be used over the entire industry.

Although the workshop is not currently scheduled, ICT hopes to have it before this time next year, and more details will likely be available on ICT's website. Van Lent then wrapped things up with questions from the audience, and overall, this was a very interesting lecture at SGS 2005, which helped illuminate ICT's multiple projects and mission in fine form.

______________________________________________________