The Future Human: An Interview with Tim Sweeney

As technology evolves, how do we interact with it? An iPhone is far more powerful than your average desktop was just a few years ago. And it has a camera in it. And it's got your schedule, and your contact list. And it has games. And, occasionally, it's even a phone.

We rely on our technology to an incredible degree. Have you ever been caught without your smartphone in a foreign country, or an unfamiliar town? It's almost existentially terrifying to realize how little we can do without our contemporary tech; that's why we have it around us all the time.

What does this convergence mean, not just for the future of games, but the future of technology in general, and the way humans interact with it?

Epic Games' Tim Sweeney knows a lot about tech, there's no denying it. In addition to being one of the paragons of game code, he has also read extensively about the limits of human perception, and the nature of technological convergence.

In this extensive interview, we speak with him about the possibility of a graphics plateau, the promise of Unreal Engine 4, and what might happen if technology were all around you, all the time -- even more integrated into your life than it is now.

Back in 2008 I wrote about something that I think you would totally disagree with -- I felt we were starting to see diminishing returns on graphics, just in terms of whether people really cared about new particle effects or lighting, when "good enough" seems to work for many, in games like League of Legends that aim for compatibility over poly-pushing.

I know that you're very interested graphical advances from a code perspective, and from a what-you-can-achieve perspective, but do you think that also pays dividends for the audience?

Tim Sweeney: Yes. We're still at the point where improvements in graphics technology are enabling major improvements in gameplay. Just the ability to do real-time lighting on environments now means you can construct a completely dynamic environment -- or destruct a completely dynamic environment -- and have all of the lighting respond accurately. It turns out that the technical features you need for that are really elaborate and expensive.

If you have your own support for real-time lighting like Doom 3 had, then all of your areas that are directly hit by light are bright, and all of the areas that aren't directly hit by light are completely black. So you need real-time indirect lighting, which means calculating two bounces of light on them, and so on, which really is only becoming possible now with today's GPUs, that are 2.3 to 2-and-a-half teraflops.

Even if your thesis is that we're getting diminishing returns on graphical effects, I think we're still at the point where making graphics innovations greatly improves our capability of implementing new kinds of games.

I saw the Unreal 4 demo, and it's very good-looking. But the thing that I found interesting, at least framed by what you're talking about, is that in a certain respect the closer you get to reality, the less impressive it is in a way, because as it gets closer to a "real thing," I know what real things look like, and we get into an uncanny valley situation. Obviously, we do this in a more fantastical setting, but that move toward reality to me almost seems like it's going to, at a certain point, start being less impressive.

TS: We don't necessarily want to simulate reality because reality is pretty boring, right? (laughs) Simulate realistic characters in a game, and they're probably just sitting around sending their friends stupid messages on Twitter. You want fantastical environments and fantastical characters, and that's really the big job of an engine -- it's not just to enable graphical realism but also to give our artists and designers the capability to really tweak things to create a custom look and feel for the game, and a custom enhanced version of reality that they can play around with consistently.

You're trying to solve a lot of new problems with UE4 -- indirect lighting, more efficient and dynamic particle effects, and that sort of thing. But what about some of the legacy problems that are still not totally solved, like shadows that are jaggy everywhere, and dynamic texture loading so that it doesn't have a pop effect; these sorts of things?

TS: Well, each generation, we improve. We greatly reduce the flaws that you see, but we're still far from having enough hardware performance to completely eliminate them. The jaggies in shadows in Unreal Engine 1 were 3 feet wide, and now they're just a few inches wide. And that's great, but until they're much smaller than a millimeter you'll still notice those artifacts. Really, the amount of performance you need to solve this completely is immense. I think we're just slowly moving in the right direction there.

The technology is solving other problems. For example, texture streaming has been a huge challenge given optical media. When you're playing Gears of War off a DVD, sometimes you see textures popping in just because we can only move the DVD head four or five times a second in order to load the textures in. If textures are coming into view at a faster rate, then you're screwed. If you look at what's possible now with solid state disk technology and flash memory storage, you have a factor of 10,000 less latency.

It's pretty significant.

TS: Oh, yeah! It's gigantic! It's able to greatly, greatly reduce some of those flaws. Every generation we're improving a lot of things, but we're still a long way from being able to simulate reality. For a long time, the Holy Grail was completely destructible environments; that means you basically have to build your game levels using architectural tools and engineering analysis so that, when the right amount of force is applied to your wall, it breaks. Then your level designers aren't just creative folks; they're structural engineers. There are significant barriers to a lot of advancements in those areas.

The thing I find funny about completely destructible environments is that any game could just become a flat plane at a certain point if you just blow everything up.

TS: (Laughs) You want to be able to completely destroy the world?

Yes -- some sort of antihero complex, probably, or maybe I just like playing Earth Defense Force.

So the next generation of consoles is going to have whatever success it has, and we'll see how that works out. But I want to ask in advance, do you see console as a viable future platform? Because it seems to me that PC is going to continue to evolve and push forward in terms of graphical advances -- and mobile is coming up so fast. Obviously, with Infinity Blade Dungeons, that's a proper Diablo-style game on an iPad. Maybe you're not going to need a console at a certain point. When you're pushing these graphical advances, where do you see them living?

TS: The platform equation is getting a lot more interesting nowadays, but gamers aren't changing that much. There are still hardcore gamers who want to play games where you sit down and have a very immersive experience for two or three hours at a time, and the iPad just isn't a good device for that. It's too small. It's not enough of a viewing window, and the sound isn't moving around you.

But I think you're seeing at the core, as we improve graphics on all platforms, there's still going to be a difference for the sort of game you design for iPad versus console or PC.

Certainly Infinity Blade: Dungeons is based on re-envisioning the Diablo-style game around the sort of experience you want when you're sitting there with your iPhone; a game which you can play for a few minutes at a time if you want, a game that doesn't require the deep and lasting commitment to have some fun a little bit at a time, and a game that's not as huge -- a project that's being developed over the course of about nine months as compared to six or seven years.

So it's a very different scope and scale, and I don't envision the core experience that you have playing a game like Gears of War or World of Warcraft going away. I think consoles are basically just a mechanism for playing games on your TV when you want to sit back on your couch and have an awesome game experience. That's very different from sitting at your computer: better in some ways and worse in some ways.

Infinity Blade: Dungeons

Right. I just wonder, as these things do appear to be converging, what about an Apple TV or a Microsoft Surface kind of situation where you essentially have that console experience there if you want? I'm not trying to bait you or anything, but I feel like this is the last console generation. I do believe console gamers will still exist -- I hope they will, but I feel like they'll wind up moving to PC.

TS: There is no question of whether gamers are going away; the question is, do they move to different platforms? There's some plausibility in that, but the console is a very immersive way to play games. They're more immersive than sitting in a chair in front of your computer with a mouse in a bright room. I have a hard time seeing that experience going away, but I think a lot of these games that are being developed now exclusively for console are going to become more important also on the PC. Most of the companies shipping games on console are also shipping them on the PC -- and doing extraordinarily well, especially through services like Steam, where it's all online and there's no hassle of going into a retail store. So certainly the PC will be a growing part of the new ecosystem.

That's kind of the thing for me. It will become so much easier to port things to different platforms, and you guys are trying very hard to make that happen. In that universe where, say, you could stream something from your Microsoft Surface or whatever to your TV, does there need to be an Xbox and PlayStation 3 and a PC when things seem to all be converging to one point? Portability is important, but convergence is also happening.

TS: Yeah, the convergence is happening, but it comes in fits and starts, and often ideas don't completely work. Also, the thing that's awesome about the controls on an iPad or an iPhone or maybe a future Microsoft tablet is the fact that you have this screen that displays images and you're able to touch it and interact with that image; when you take that device and broadcast that image to the TV, then suddenly you just have a big, flat mouse-like surface. It's not a very compelling control device compared to a game controller, or even mouse and keyboard on the PC. I think you need more to the equation than just a smartphone or tablet beaming a signal to a TV; the controls are a problem there.

So what happens there eventually? I don't know. Are you going to have an Apple TV or a game controller or a console that won't have a game controller? Will all these devices have compatibility in terms of transferring video back and forth? I don't know. There are a lot of unanswered questions, but when the next generation of consoles comes out, you know that they're going to be highly polished devices that will work perfectly for playing games.

A lot of the other ideas that are being thrown around have potential, but are not anywhere near being a finished, polished, usable format. I think you have another solid generation of consoles coming out ahead. After that, who knows? Maybe you'll be wearing your display device around with you everywhere you go and your TV, computer, monitor, and iPhone screen have vanished because it's all mounted to your head.

I know you've done some research into the limits of what humans can see graphically. We were talking earlier about this idea that, to paraphrase a bit, technology will become an extension of yourself and your body; it will be all around you, happening everywhere in your world, with gestures and cameras and whatnot. Have you thought at all about the ramifications of that on the human body, being inundated with all of these screens and waves and things all of the time?

TS: It's hard to say. I think it's a simplification in a lot of ways, because I think right now we are surrounded by a whole lot of separate display devices, and most of them are really crappy. When I'm at my iPad, that's an awesome experience; but then I go to my Windows XP-based computer, and that's a pretty slow and clunky experience; and then I go to my car's navigation system -- I'm sorry, but General Motors makes lousy navigation systems. I like the idea of all of those disparate display devices just going away and being replaced with this pervasive display overlaid on top of the world that's with you everywhere. It really enables a much greater degree of polish than all of these separate experiences.

But it also enables you to be advertised to against your will, potentially.

TS: I don't think anybody would ever put up with a device like that. If Google tries to sell you these glasses that are always popping up ads in your face, they can go to hell!

But people do kind of put up with it. On the plane over here to Taiwan, you have to sit through some advertisements; they're mandatory. Or, in a cab, they're trying to sell you stuff, but you already paid money! I could foresee it happening.

TS: You're right. It's a brave new world.

I've been reading lots of Kurt Vonnegut lately, too, so...

TS: If you're scared about the information Facebook collects, for example, then just imagine what happens when there's a company that's basically beaming a live feed of what you see all day, every day, and all of your daily interactions.

Absolutely. It's kind of horrifying. But, yeah, it is interesting to think about what a game would be in that kind of environment. Does it become a Second Life kind of thing but finally done well, where you are a superhero flying through this universe and you're actually moving around in the real world? I don't know!

TS: Yeah, we could totally go into those sorts of games. Another neat thing is the realization that, if you have a device like that that you are always wearing, then any game scenario continues to work, but you don't need a display to make it work. You could use that device to project a TV to a particular place in your house, and it would work and look just like a real TV. So you could continue to use it in legacy scenarios. Going beyond that, it's impossible to predict. The only way to figure that out is to have a million smart developers each out trying new ideas, most of them failing, but finding a few things that work.

I feel like, for core games, the idea of a virtual controller is not that compelling. A core Kinect game is maybe possible, but the precision required is not really there. But in that kind of a scenario, I just have this vision of a dude that has a wireless controller that's snapped to his belt, so if he wants to play something he can just...

TS: I agree. The best game controllers and best devices in general don't only sense your motion, but also provide tactile feedback in response to your motion. That's why a mouse is so satisfying; you're moving your hand, and your hand has an enormous amount of precision, so as you move your hand you're also feeling the motion.

You want to see a high-bandwidth connection between your brain and the device. That just doesn't exist. Even with a touchscreen, you have this great ability to touch what's on the screen, but you can't feel key boundaries on your virtual keyboard. When you have a game that has a little joystick controller on-screen, you don't have any touch response there. Those are lousy.

If all you're doing with a Kinect-type of device is putting in some virtual input device, it theoretically works just as well, but if you can't feel it then it's just going to fail. I don't know what the ultimate answer to that is. Maybe there will be some advances in tactile feedback. Ideally, you want a touch device that can impart a force on you with some sort of dynamic feel. I can imagine a virtual keyboard having the feel of a keyboard through some sort of mechanism like that, but that's been largely unexplored so far.

While that does seem like it would be the solution, again, it's a little scary to me. What are those impulses going to do to me? How are they going to alter my sense of touch in general, using them long-term?

TS: Yeah. I have the same complaint with cell phones that fry your brain. That's a real problem; after I talk on my cellphone for a long period of time, I definitely feel some --

Your ear is hot.

TS: It feels a little like I drank a beer or something. It's definitely doing something bad to you there.

That's the kind of thing I was getting at before. You're mostly talking about a unified device, but if the world has all of these things it can beam at you, what's going to happen? We're going to turn into mutants or something.

TS: Well, we're already well along that path if you look at the way we live our lives. The typical person with an iPhone has a completely different view of what's out there in the world and how to navigate through it. You can't get lost anymore if you have a GPS, and you can't forget anything because you can just look it up online on Google. It's a different world.

I keep getting reminded of what the world used to be like when I come to other countries because I can't afford a smartphone that will work everywhere. At one point, when I was sending a text message on a feature phone, I tried to touch the screen to move the cursor, and of course that doesn't work because it's just a screen. I have to go around looking for specific restaurants because I'm a vegetarian, but I don't speak the language; so I have to write down instructions for myself and leave the place where my internet is. Then I'm just around with no backup. It's really made me realize how reliant on this technology I am.

TS: Yeah, it's crazy. At the rate things are advancing, we're going to have to explain to our grandkids what a book was! I mean that seriously!