Dirty Java: Using the Java Native Interface Within Games

Editor's note: this article appears in the July 1999 issue of Game Developer magazine. Due to space limitations, the magazine was not able to include a long list of references that the author supplied. So we are simulanteously publishing the article online, and include the aforementioned references at the end of the article.

You have heard of Java. Actually, you probably have had a hard time trying to escape the hype surrounding it since 1995. There are, however, compelling reasons to use Java in shrink-wrapped games. Two major options that have recently been examined by professional game developers are using Java as your scripting language, and using Java for safe run-time downloadable code that gets executed on the client. In both cases, it is the interface to native code that you end up dealing with the same glue that is required to make Java work on your PC in the first place. If using a portable, standard, object-oriented programming language with built-in security within your game sounds appealing, then you should become acquainted with the Java Native Interface (JNI), which is your tool to write "dirty" Java — Java code that is tightly integrated with your native code.

There is an abundance of information about "pure" Java, and this is not the place to explain all the actual and alleged advantages of Java technology. Unfortunately, many of the highly touted "pure" solutions quietly omit the sophisticated native machinery at work under the hood. Instead of looking at game applets running in web browsers, this article sizes up possible real-world uses for Java, and looks at the ways some game developers are already using Java for their titles.

A Brief Recap of Virtual Machines

Briefly, here are the key components that are relevant to this discussion of Java:

Java bytecode. This is pseudocode for a stack-based processor described in the Java specification. Valid bytecode has to satisfy a lot of requirements, but if you really wanted to, you could actually write it by hand with a decent hex editor and the specifications.

The Java Virtual Machine (JVM). Java CPUs have not yet conquered the market, so software must translate Java instructions into the language the PC hardware can understand. The JVM is a multithreaded program for your operating system, supplied from various vendors, that executes Java bytecode and maps bytecode to native instructions. The Java specification places few restrictions on the actual implementation of a JVM. You can find Open Source JVMs on the Internet, or even write your own clean-room implementation. You can also get the sources of Sun's reference Java Development Kit (JDK) and negotiate a license for commercial use.

The Java Programming Language. This is an object-oriented language, designed with a subset of C++ in mind. It has run-time bounds checking, it is restricted in terms of memory access and handling, and it has simplified inheritance patterns. The language features an overwhelming inflation of extension APIs and core classes.

It is important to understand that these three components are entirely separate. A JVM will happily execute valid bytecode that was not generated from Java source — for instance, from a compiler that maps a language like Scheme to bytecode. You could also implement a simplified virtual machine that skips validation and executes bytecode no compliant JVM would accept, or create a virtual machine that has no garbage collector thread at all. In addition, there are compilers that generate native code from Java source files — you don't need a JVM to write an application in Java. From the "amusing avenue of hand-tuned virtual assembly" (as John Carmack of id Software referred to virtual CPUs) to using a JVM without ever coding in Java (converters mapping C or C++ subsets to Java are feasible), all your tampering needs can somehow be addressed.

A lot of confusion originates from discussions that fail to distinguish the various components of Java technology. All of these are required parts of any JDK, however. The only JDK that is actually called JDK is the reference implementation provided by Sun Microsystems for Solaris and Win32, and its licensed ports to other platforms.

The concept of a virtual machine for game scripting is not new, and not restricted to Java. John Carmack recently decided to use a custom virtual machine and bytecode created by a modified C compiler (LCC by Fraser and Hanson) for id's upcoming game, Quake 3: Arena. Carmack once pointed out that game coders "have more urgent things to do than design languages." Ironically, he is now engaged in designing his own virtual machine and native interface. Technologies like Java's just-in-time compilation and HotSpot optimization originate in the Java technology mainstream, and they are powered by more resources than a single game company could ever command. If you can make Java work for your game, then you will benefit from this momentum.

Talking to the Natives

Game coders usually do not trust cross-platform APIs based on layers of abstraction. Interpreted bytecode does not typically appeal to an industry that still counts on assembly to get a performance edge. Windows-based games sell, period, and portability is not really an issue.

Compare this to the holy grail of "100 percent pure Java." Mainstream Java technology is seemingly meant for tiny "gamelets," not serious games. Besides, despite the bloat, there are bits and pieces missing from the Java core classes: access to certain devices and system services is simply not available (and might never be, for design and security reasons), and politics sometimes gets in the way (witness the lack of Java OpenGL bindings). However, if you take a closer look, it turns out that there is always native code at the very heart of all that "pure Java": there is a JVM written in native code, and core classes partly implemented as native code. Here reigns the Java Native Interface, gluing together Java and native C/C++ code, and it is the key to combining your native code with Java technology.

JNI is part of the Java specification, and it's a mandatory part of all Java implementations. Sun was recently granted a court injunction forcing Microsoft to add JNI to the Microsoft Java implementation. Ideally, .DLLs and binaries using JNI should be byte-compatible for a given platform. The 1997 JNI specification is available online, and there are also books on the subject, so this article includes only a brief summary of it before we get into its applicability in games.

The first task JNI must solve is getting the JVM and user-written native code to agree on built-in types and memory layout to exchange data (see Figure 1). Only some of the core classes are represented for the native code (see Figure 2) — most of them arrays of the built-in types.

What about jclass and jobject? JNI will not hand you the memory layout of a Java object, but it must provide you a handle. It even preserves the relationship between java.lang.Object and java.lang.Class. A jclass object can be cast to jobject safely in any JNI that complies to the Java specification (non-compliant implementations have been found). JNI is foremost aimed at C (the C++ bindings are just inlined wrappers), and no support for object-oriented programming on the native side is offered. With the exception of Throwable, String, and arrays, all classes have to be squeezed through the jobject and jclass representation. Arrays of arbitrary classes (including String) will always be mapped to jobjectArray. JNI defines JNI_FALSE/TRUE, a jvalue union type, and a jsize for your convenience.

Further, handles have access to fields, and also call methods of classes and objects. It might look like java.lang.reflect.Field and java.lang.reflect.Method are the Java equivalents to JNI's jfieldID and jmethodID, but JNI predated the Reflection API, and actual reflection support was added only to the latest JNI revision.

Caching field and method IDs is a good idea, as retrieval involves a string lookup. Be warned that caching can get tricky in applications with multiple threads and class loaders. You will have to keep an eye on the garbage collector as well — without a strong reference acquired by NewGlobalRef(), the garbage collector might remove the object your native code is still referring to. Likewise, dangling references not removed by DeleteGlobalRef() can keep obsolete Java objects from being collected. Use DeleteLocalRef() to avoid accumulating temporary references within loops. JDK 1.2 offers limited support for weak references, too.

Within your native code, all will revolve around the JNIEnv interface function table — your door to the Java side. It provides methods to handle references, create objects, load classes, access fields and call methods. Further, you get utility functions to iterate arrays, throw exceptions, and perform monitoring to make the native code threadsafe. Finally, an executable can also register functions as native methods, making code known to the JVM without dynamic linkage.

Listing 1 is a small example showing how native code can use Java to get things done, and how a native callback is registered with the JVM. Listing 3, discussed later on, does the opposite: it shows how Java calls native code.

Listing 1. Native code making use of Java.

// Java class, server-side Game Logic scripting.

package somegame;

class ScriptEngine {

// This uses native error() callback.

public static void init() {..}

...

public static native void error( int code );

}

#include

// Native error callback provided to Java.

extern void SV_ErrorScriptEngine( jint i );

JNINativeMethod svSE = {"Java_somegame_ScriptEngine_error","(I)V", SV_ErrorScriptEngine };

// Native function to set up scripting module.

void SV_InitScriptEngine( JNIEnv* env ) {

jclass clazz = NULL;

jmethodID method = NULL;

jint err;

// Lookup class, loads the class if not yet done.

clazz = (*env)->FindClass( env, "somegame.ScriptEngine" );

if ( clazz != NULL ) {

// Register native method, returns zero on success.

err = RegisterNatives( env, clazz, &svSE, 1 );

// Lookup Java methodID.

method = (*env)->GetStaticMethodID(env, clazz, "init", "()V");

}

// Handle errors.

...

// Call static method.

(*env)->CallStaticVoidMethod(env,clazz,method);

}

It is tempting to use static (class) methods, as you do not have to handle an object in addition to the class. In many cases this is absolutely sufficient — servers will likely not run two script engines in parallel. In many other cases though, this leads to bad object-oriented design on the Java side.

Double Indirection: The Catch-22

There have been competing native interface APIs proposed, most notably Microsoft's Raw Native Interface (RNI). The problem with RNI is that it exposes the underlying operating system and JVM implementation, which makes it impossible to port to another VM.

In some ways, the problem with JNI is that it does not expose the VM implementation. JNI makes you go through pains to ensure that native code never gets to see how Java objects are laid out in memory. Consequently, the native code has to deal with indirections every step along the way, many of them ultimately leading to table and string lookups. This is not good for application performance. See the example in Listing 2, where we pass command line arguments from Java to C, which involves references, arrays, and conversion to UTF-8 (the canonical two-byte Unicode encoding used by Java).

Listing 2. Passing Arguments from
Java to C.

// Minimal Java control code wraps legacy engine.

package somegame;

class GameMain {

protected static native int nativeMain( String[] args );

public static void main( String[] args ) {

int ret = nativeMain( args );

...

}

static { System.loadLibrary("Game"); }

}

// Handing over the arguments to native code.

// This code will be put into the Game.DLL.

#include

extern int gameMain( int argc, char** argv );

JNIEXPORT jint JNICALL

Java_somegame_GameMain_nativeMain ( JNIEnv* env, jclass cls, jobjectArray jargv ) {

jint res;

jint argc;

jint i;

jboolean isCopy;

jstring jstr;

jsize len;

const char* cstr;

jargc = env->GetArrayLength(jargv);

for ( i = 0; i

jstr = (jstring)(*env)->GetObjectArrayElement(env,jargv,i);

cstr = (const char*)((*env)->GetStringUTFChars( env, jstr, &isCopy ));

// We copy - we have to release, and we don't want to accumulate local references.

argv[i] = (char*)malloc( strlen(cstr)+1 );

strcpy( argv[i], (const char*)cstr );

// Did the JVM copy as well?

if ( isCopy == JNI_TRUE ) {

(*env)->ReleaseStringUTFChars( env, jstr, cstr );

}

// Clear local reference.

(*env)->DeleteLocalRef(env,jstr);

}

// Call our main() now.

res = (jint)gameMain( (int)argc, argv );

// Release allocated memory.

for ( i=0; i

free( argv[i] );

}

// Return to Java.

return res;

}

Tools such as javah generate C header files containing proper function prototypes (name and signatures) from a Java class. These tools are already Unicode-aware, thus using underscores and other special characters in Java method and class names can lead to surprising results. The code in Game.dll will be linked to the class by the JVM by calling java.lang.System.loadLibrary("Game") automatically.

The example in Listing 2 implements a minimal Java wrapper around native legacy code. Given all the implicit and explicit copying, we somehow seem to have come full circle: to get rid off some portability-related Java overhead, we decided to use native code, only to find out that the JNI design hampers the interaction between Java and native code to ensure portability. Now what? Well, there are basically two ways left to increase performance:

1. Brute force. You could switch tools and compile to native code. If you pursue this option, make sure your Java compiler supports JNI as well, and that it doesn't just compile pure bytecode.

2. Smart design. You could accept the limitations of the JNI, and design your native and Java modules in a way that streamlines the interface between them.

Mind you, your native code by itself will be as fast as it gets. It's only the transfer of parameters and results back and forth that, inside an inner loop, incurs significant performance penalties.

The Invocation API

Now that you have seen some means to glue Java and native code together, where does a game developer actually get access to the virtual machine? The common answer is the Invocation API. The Invocation API allows you to embed the JVM into your native applications. It provides the means by which you can retrieve an existing JVM attached to your application, or launch one with proper configuration settings. Listing 3 shows how to invoke the JVM in an application, using JDK 1.2. You can use code like that shown in Listing 1 to get Java classes loaded and executed. If you do not want to encapsulate native method code into .DLLs, then your application can use the JNI function RegisterNatives() to make native functions from the executable known to the JVM. That way your game would ship without any .DLL.

Listing 3. How to invoke the JVM in an application using JDK 1.2.

// JDK 1.2 Invocation example

#include

// Setting some standard options.

extern jint JNICALL Printf( FILE *f, const char *fmt, va_list args );

extern void JNICALL Exit( jint code );

extern void JNICALL Abort( void );

#define NOPTIONS = 5;

JavaVMOption OPTIONS[5] =

{

{ "classpath", (void*)"C:\\java\lib\classes.zip; D:\Game\classes" },

{ "verbose", (void*)"jni,gc" },

{ "vfprintf", (void*)Printf },

{ "exit", (void*)Exit },

{ "abort", (void*)Abort }

};

// Create a JDK 1.2 JavaVM as desired.

JNIEnv* SV_InitJavaVM( JavaVMOption* options, jint nOptions ) {

JavaVMInitArgs vm_args;

JavaVM* vm_handle; // not preserved

JNIEnv* env; // return to caller

jint ret;

// Request version 1.2

vm_args.version = JNI_VERSION_1_2;

ret = JNI_GetDefaultJavaVMInitArgs(&vm_args);

if ( ret==0 ) {

vm_args.options = options;

vm_args.nOptions = nOptions;

vm_args.ignoreUnrecognized = JNI_TRUE;

ret = JNI_CreateJavaVM( &vm_handle, &env, &vm_args );

if ( ret==0 ) {

return env;

}

}

... // error handling

}

Most JVM and compiler implementations fully support JNI, as it is needed to handle cleanly implemented core classes. (Interestingly, even Sun's own JDK does not always use JNI internally.) But the Invocation API was sometimes omitted from JDK ports and third-party JVMs. If you want to use the Invocation API, make sure your tools and targets support it.

Worse still, some JDK ports support Invocation, but do not do so properly (including some revisions of Sun's own Solaris reference implementation). Invocation requires threadsafe .DLL handling, which is not always granted (for instance, in some Solaris and Linux revisions). If the dynamic linking is not threadsafe, your application will suffer spurious errors during startup. Furthermore, the official Java specification now sanctions limitations of the reference JDK that affects DestroyJavaVM(). It is not possible to destroy a JDK JVM. Consequently, you can't invoke another one from the same application — multiple JVMs, whether subsequent or in parallel, are not possible. Once you lose your JVM for whatever reason, your application must terminate. Fortunately, most of the other pitfalls were smoothed out last year.

In Listing 3, ignore the JVM handle, which you can always retrieve by calling the JNI function GetJavaVM(). A more flexible approach to invoke the JVM calls GetCreatedJavaVMs() first to check whether a JVM already exists, and uses AttachCurrentThread() to make itself known if one is found. The question is, do you really have to invoke the JVM this way?

Two Architectures: Embedded Java and Encapsulated Native Code

How you choose to obtain the JVM for your game is, in all likelihood, the most important decision you have to make when rigging up a Java-based game project. To a C coder, invocation might seem a natural choice. This architecture is known as embedded Java. Your application is linked to a .DLL that provides a JVM, which happily lives and dies within your application, completely at your disposal. It looks like this:

// Engine piggy-backed with JVM or

// Engine retrofitted with JVM

// Set options, possibly parsing commandline.

nOpt = SV_GetOptions(&options,argc,argv);

// Invoke JVM, get script engine started.

SV_StartVM( SV_InitJavaVM(options, nOpt));

// Start the actual game.

return gameMain( argc, argv );

On the other hand, if you write a pure Java game, or use the somegame.GameMain class shown in Listing 2, then some other application loads the JVM and hands it the Java bytecode of your game. This scenario is used when a web browser runs "gamelets", for example, or when JDK's java loads an executable .JAR file. Whether your game uses native code or not, you do not have to concern yourself with invocation if the main loop is written in Java. Native method code will be encapsulated in Java classes, as long as the .DLLs required are loaded in time. It does not look like much of a difference, but choosing one or the other might have a huge impact on your project.

Embedded Java: A Natural Choice?

Let's look at an example that I call the "Quake 3 scenario." Your team has nearly finished a game engine written in C or C++. The game has a large and stable legacy code base that you don't want to tamper with, yet there is a clear-cut need that Java might address, such as a new server-side scripting language, or support for client-downloadable code. In short, you want to retrofit an existing application with a Java component.

The history of the Quake engine is a great example. Quake featured a custom scripting language (QuakeC), Quake 2 introduced a server-side .DLL (game.dll), and some Quake engine offspring now deploy client-side .DLLs.

Embedding is possibly the best answer in all cases where you have to deal with C legacy code that is not implemented in an object-oriented fashion. The JVM is just another device that is initialized, configured, started and shut down again. There are some restrictions (for instance, you cannot restart the JVM once it has shut down), but in general, all you do is provide raw data (bytecode) to the embedded JVM much the same way you'd feed .WAV files to a sound device. If you do not want to use .DLLs at all, embedding is your solution. You also get a lot more control over the JVM that is used by your game. Shipping a Java Runtime Environment (JRE) with an embedded solution might save you support and maintenance headaches. It might also address some reverse engineering, tampering and cheating issues.

If embedded Java is used, either C control code executes Java methods on the JVM which return the data, or the Java code in turn calls native methods to write back. You could have Java threads run in parallel to your application, but debugging an application that moves back and forth between native and Java execution stack frames can be a challenge to you and your tools — multiple threads will make it even tougher.

What problems are specific to using an embedded JVM? Some have already been mentioned, such as negligence or outright omission of the Invocation API from some Java implementations, and potential problems you might face when falling back on compiling Java to native code. All of these problems can be overcome one way or the other, however. The real danger might be much more subtle.

Your legacy code has a certain design — possibly not object oriented at all if you used C, or possibly an object-oriented design that maps badly to Java if you used C++ excessively (if you made use of templates and/or multiple inheritance). In these cases, taking a single component of your game (for instance, the server-side game logic) and converting it to Java could introduce bugs and errors in formerly stable and tested code. Worse still, through JNI the design used in native code will proliferate into Java code, resulting in badly designed Java code. For example, if you never handle objects (see Listing 1 and its use of class methods), it is unlikely that you are using an object-oriented design. Legacy code tends to share memory using pointers for speed and convenience, which is not possible with JNI. You have to think hard and make judicious cuts to get a lean interface between Java and native code. High levels of abstractions implemented as abstract base classes and interfaces usually work best: the more details you hide, the better JNI will work for you.

Consider handling structured data on both sides of JNI, such as that used for collision handling. Collision response is part of the game logic (does the player take damage, bounce, or die?) and is thus handled in the Java code in our example. Collision detection might be performed within the scene representation that is also used by the rendering code — almost surely native code you want to keep. In this scenario, your Java game logic might call native code to trace an object's movement through the scene.

This is where the level of abstraction is relevant. Take the Quake 2 representation of a vector in 3D space: float[3]. In Java, this is best represented as a class with float x,y,z fields. This avoids array bounds checking overhead, and frees us from worrying whether JNI pins or copies the array. For objects that small and likely to have all their fields accessed, the simplest way to pass them back and forth is to unfold them on the stack as primitive data types, much the same they would be flattened for serialization.

This solution is more the exception than the rule, however. In general, it pays off to hide as much detail as possible on both sides. The game logic does not need to know whether axis-aligned bounding boxes or spheres are used for collision detection, it only has to initiate updates to position and size. For the actual trace in native code, it is irrelevant whether a given entity within the bounding volume is a player, a monster, or a fireball.

Using a high level of abstraction on the Java side by sticking to abstract base classes and interfaces makes retrieving and caching method IDs in your native code much easier, since all objects within an inheritance tree will share the same signatures. You might find it safer to cache field and method IDs in class descriptor structs or C++ objects. Field access is more efficient, but exposes the internal implementation of your Java objects. JNI methods like GetFieldID and GetFloatField can be used instead of, say, GetMethodID and CallFloatMethod to access instance fields directly.

You pass references as jobject handles instead of pointers to make Java data accessible to native code. The reverse is not possible: neither C structs nor C++ objects are visible to Java. You can address C++ objects or C structs with jint handles on the Java side, using more (hash table look-up) or less (typecast) safe ways to retrieve the effective address. A proxy class would then wrap native methods with public accessors, like that sketched out in Listing 4.

Listing 4. Java Proxy for a C++ object.

package jni;

class Proxy {

/** Handle to retrieve C++ object, native side. */

private final int handle;

/** Native LUT/constructor. */

private static final native int newNative();

/** Constructor, gets/creates a handle to a native object. */

public Proxy() {

handle = newNative();

}

/** Simplified fieldID. */

private final int SOME_FIELD = 1;

/** Accessor hiding the simplified retrieval. */

public final float getSomeField() {

return getFloat( handle, SOME_FIELD );

}

/** Method that saves us many retrieval() functions. */

private final native float getFloat( int handle, int field );

}

// Minimal C++ object, and JNI glue.

class NativeObject {

public: NativeObject( jobject owner ) {

this.owner = owner;

}

public: jobject getOwner() {return owner;}

public: float someField;

public: jobject owner;

};

#include

extern jclass InvalidProxyOwnerException;

extern jclass InvalidFieldIndexException;

// extern "C" implied

JNIEXPORT jint JNICALL Java_jni_Proxy_getFloat

( JNIEnv* env,

jobject owner)

{

return (jint) (new NativeObject(owner));

}

JNIEXPORT jfloat JNICALL Java_jni_Proxy_getFloat

( JNIEnv* env,

jobject owner,

jint handle,

jint field )

{

// Truly dirty. Trust on blank finals.

NativeObject* obj = (NativeObject*)handle;

if ( obj->getOwner() != owner )

(*env)->ThrowNew(env, InvalidProxyOwnerException, "access attempted by non-owner");

switch( field ) {

// Enums to be kept in sync manually...

case 1: return obj->someField;

default:

(*env)->ThrowNew(env,InvalidFieldIndexException,

"access attempted by non-existing field");

}

}

}

Of course, if you want to avoid switch statements in the native method, or you want to wrap C++ accessor methods instead of exposing fields, you could also implement the public Java accessor as a native method. Incidentally, maintaining the same set of enums in Java and native code is one of the problems that does not yet seem to have an elegant solution. If you are using a look-up table to retrieve pointers for handles, the jobject argument might already be sufficient.

A native proxy implemented as a C++ object or C struct could cache a global jobject reference along with method IDs and field IDs. Caching actual game data inside native code means that your proxies have to be kept synchronized with the master objects, or you will end up with consistency errors that are very difficult to track down.

Alternatively, you could encapsulate the results or take a snapshot of a native object's state in a new Java object created in native code, using the JNI function NewObject() to call a Java constructor. This approach works even better if your native and Java modules communicate by passing event descriptor objects to a queue.

Encapsulated Native: A Magic Bullet?

If you go down the road of Java control code, be prepared to throw out legacy code whenever necessary. Encapsulation means dividing and splitting your code base into tiny pieces — heaven if you are at liberty to design from scratch, hell if you have to handle code that is just sticking together. If you can't isolate native code modules and wrap them with Java classes, then there won't be a secure migration path. Handling a native legacy code base might well take more time than gutting it and starting from scratch. If you consider abandoning C/C++ as your main language, an encapsulation architecture is definitely the way to go.

Java and JNI will always have some performance disadvantages that make them unsuitable for time-critical inner loops. However, it can be very efficient write your control code in Java (which can account for up to 90 percent of a game's total code base) even though that often takes up less than 10 percent of the overall processing time — especially for games that make the CPU spend half of the time in an OpenGL driver. This was the reasoning behind the Prax War architecture that Billy Zelsnack implemented at the now defunct game development studio, Rebel Boat Rocker (see sidebar, "Java wraps native code in Prax War"). There are only a few areas in which native code is really needed, such as managing raw data (textures and sound resources, for example), and rendering. Collision detection might best be done in native code shared with the renderer. Collision response, however, is a natural part of high-level game logic. In some cases, the lackluster performance of Java core classes might force you to replace them with your own custom Java code, or even use some native code instead. In the end, you will have a few cleanly separated native code modules you can optimize to your heart's content, controlled by robust Java code. The object-oriented design propagates top-down into your native code, which should be another benefit.

The Holy Grail: 100 Percent Pure Java

Supporting the multitude of Internet server platforms (Solaris, Linux and other UNIX flavors, OS/2, and Windows NT) has become increasingly important for multiplayer games. Presuming the existence of decent Java networking core classes and acceptable performance of Java-based scene lookup and collision detection code, a dedicated server implemented entirely in Java is an attractive possibility — portable by default and, in the absence of JNI, easily compiled to native code.

Portability issues are not as pressing for clients, the majority of which are Win32-based. At the same time, a real need for native code might only be found for the client, which is pushing a lot of raw data (textures and sounds) from local disk to local memory to native driver code. Unfortunately, only a few games, such as id's experimental QuakeWorld release, have separated the client and server completely. Consequently, a dedicated Java server means a separate code base that partly duplicates the shared client/server sources. Dedicated servers have become quite common recently, but in the long run, the code duplication is not acceptable. Automated Java-to-C or C-to-Java conversion might offer a temporary workaround only.

Ultimately, shipping a client written in Java will require decent Java bindings around reliable, cross-platform APIs, and these are nowhere to be found. Java does not have official OpenGL bindings, there is not even a portable native API for 3D sound, and the politics surrounding Sun's proprietary Java3D scene graph API doesn't help matters. For the foreseeable future, commercial games will not be feasible without JNI and native code.

Q2Java Today: What's Next?

Few games (Red Storm Entertainment's Politika is among them) have shipped with Java built into them, but if you want to see a full-sized example of an embedded Java VM running Quake 2 deathmatch right now, you should visit the Q2Java web site at http://www.planetquake.com/q2java. Q2Java, orginally by Barry Pederson, is a cooperative open source implementation of the Quake 2 multiplayer game logic and works with the native Quake 2 executables on Windows 95/98/NT, Linux and (as a dedicated server) Solaris.

This article has introduced the two major roads to using Java for your game, though admittedly, a lot of details have been omitted and major issues (like security) were not touched upon.

Bernd Kreimeier is a physicist, a writer of novels and articles, and a coder. Currently living in Ireland, he is doing contract work on Java for games, and working on Warped Space, his own game design. This gun for hire — contact [email protected].

Java for Games

This is an annotated list of references for Java. It is by no means complete, but it should give you some starting points to track done the tools and information you require.

Java for commercial games is one of the few topics that is not excessively covered yet. Usually the examples and documents cover "Gamelets", like http://java.sun.com/features/ 1998/08/games.html. There are a few games that use, or tried to use, Java. The Rainbow Six Postmortem in the May 1999 issue of Game Developer magazine reports some experiences with Java-based technology. The Java3D-related programming articles here on Gamasutra are probably more useful with respect to applets, and maybe for writing portable editing tools, but do not relate to full-sized games:

http://gamasutra.com/features/programming/ 19990115/java3d_01.htm

http://gamasutra.com/features/programming/ 19990122/java3d_02.htm

On the topic of using Java in commercial games, Robert Huebner of Nihilistic gave a talk at the 1999 GDC (found in the proceedings on pages 403-414), the slides of which are available at http://www.nihilistic.com/GDC99/ Java/index.htm. Huebner focuses on using Java as an embedded script interpreter, and the print version has a very detailed list of requirements and constraints and how they are met by Java. His review also puts some emphasis on the Embedded Java Application Environment (EJAE) http://java.sun.com/products/ embeddedjava. I suspect that EJAE is not really relevant. Sun has created various Java specifications in attempts to cover potential markets, but I am not aware of anything actually delivered that would be significantly different from the core JDK. For example, the GUI toolkit for embedded devices (called Truffle) will basically implement AWT. For our purposes, "embedded" Java is simply equivalent to a VM created and destroyed by the game engine.

The Java Native Interface (JNI)

By now there should hopefully be a full specification for JNI in Java 2, at http://java.sun.com/products/jdk/1.2/docs/guide/jni/index.html. If not, the document to get is the "Java Native Interface Specification" as of Release 1.1, revised May 1997, available at http://java.sun.com/products/jdk/1.1/download-pdf-ps.html. This describes the Java 1.1 JNI in its entirety. The few JDK 1.2/Java 2 changes to JNI are described at http://java.sun.com/products/jdk/1.2/docs/guide/jni/jni-12.html. There are also some very simple tutorials on how to use JNI.

JNI has not been dealt with in many books yet. A good reference is Rob Gordon's Essential JNI (Prentice Hall, 1998). It covers the JDK 1.1 JNI and Invocation in lots of detail with both C and C++, including, for example, a UNIX/Win32 example which shows how to access serial devices via JNI. The sample code is at: ftp://ftp.prenhall.com/pub/ptr/professional_computer_science.w- 022/gordon/essential_jni/.

Check out the book Java Native Methods, to be published under the Alligator Descartes alias (it's written by the main developer of Magician). It is scheduled for release in August 1999. See http://www.oreilly.com/catalog/javanm/noframes.html for some details. The book will cover both Sun's JNI and Microsoft's RNI, with insights gained during the implementation of the Magician OpenGL interface, which is supposed to serve as the basis for the ARB specification of JavaGL bindings (see http//www.arcana.co.uk/products/magician/). The author is very critical of JNI and AWT design (performance is his top priority), and for those developers that are willing to wed themselves to Microsoft's Java VM and tools in exchange for possibly better performance, the book might give some insights in what gains are actually possible.

Java Print Documentation

There is, of course, a plethora of printed books on Java, e.g. the O'Reilly "White Series" on Java. Note, though, that many Java books out there do not cover JDK 1.2 yet. For topics like Java multithreading, which underwent significant changes, you should wait for updated versions. See the Java 2 documentation on deprecated API parts: http://java.sun.com/products/jdk/1.2/ docs /guide/misc/threadPrimitiveDeprecation.html

To understand the inner workings of Java a bit better, a book like The Java Virtual Machine by Jon Meyer & Troy Downing (O'Reilly, 1997) comes in handy. This one features a Java assembler, jasmin, and a bytecode-by-bytecode introduction to the way the JVM gets to see your code. The hooks for JNI are not described in depth, unfortunately. Neither is the JVMDI debugger interface, or garbage collection, and both require a technical background that is important if performance becomes an issue. A similar book is Inside the Java Virtual Machine by Bill Venners (McGraw-Hill).

A basic introduction and print reference is Java in a Nutshell by David Flanagan, the 1997 edition of which covered JDK 1.1. It is not very useful once you get beyond the basics, and as with many other Java introductory books, its applet-based approach to the AWT GUI is not very efficient if you are heading for a Java-based, in-house tool chain.

Electronic Documents

The web, of course, will drown you in more or less useful information. With brief annotation, here is a list of (hopefully persistent) URLs:

The Horse's Mouth:

http://java.sun.com

http://java.sun.com/docs/white/langenv

Java Virtual Machine Specification:

http://java.sun.com/docs/books/vmspec

Java Language Specification:

http://java.sun.com/docs/books/jls

Tutorials:

http://java.sun.com/docs/books/tutorial

There are a lot of very useful reports and FAQs scattered all over the web. IBM has an entire section of their corporate website, http://www.ibm.com/java, dedicated to the Java cause. I definitely recommend getting Mark Davis' paper on porting C++ to Java, found at http://www.ibm.com/java/education/portingc/. On that site you will also find IBM's Server Optimization Report, http://www.software.ibm.com/developer/library/javahipr/javahipr.html, which gives some insights in Java performance tuning. It also has Mark Davis's introduction to Java-based software localization, at http://www.ibm.com/java/education/globalapps/Converting.html, which might come handy for Java-written games.

Further, there are the JavaWorld Developer Resources, at http://www.javaworld.com/#res. There, you will find covered specific issues, like threads and pooling (at http://www.javaworld.com/javaworld/jw-05-1999/jw-05-toolbox.html, http://www.holub.com/goodies/javaworld/jw_index.html), and an introduction to object recycling techniques (http://www.javaworld.com/javaworld/jw-09-1996/jw-09-indepth.html).

An introduction to garbage collection is found at Cygnus' site, at http://sourceware.cygnus.com/java/papers/nosb.html. Bill Venner's http://www.javaworld.com/javaworld/jw-08-1996/jw-08-gc.html might also be a good starting point. Another example from JavaWorld, relevant for TCP-based applications like chat servers, is the Volano-Mark Benchmark at http://www.javaworld.com/javaworld/jw-08-1998/jw-08-volanomark.html.

Some other useful documents can be found at Sun's Javasoft Developer Connection, and the FAQ for the site is a good start. It's found at http://developer.javasoft.com/developer/techDocs/faqs.html. A typical example is http://developer.javasoft.com/developer/technicalArticles/ Programming/Stacktrace/index.html. In particular, check out the Bug Parade at http://developer.java.sun.com/developer/bugParade, which, if operational, might save you some grief with known bugs.

Finally, here is a list of a few FAQs related to USENET discussion groups.

Elliotte Rusty Harold's comp.lang.java FAQ:

http://sunsite.unc.edu/javafaq/javafaq.html

The Java networking FAQ:

http://www.io.com/~maus/JavaNetworkingFAQ.html

Peter van der Linden's Java Programmer's FAQ:

http://www.best.com/~pvdl/javafaq.html

Some benchmarking and performance insights can be gained from related FAQs, but keep in mind that Java technology is changing quickly -- some of the observations of 1997 did not hold in 1998, and so on. It is always good to benchmark and profile the VM you depend on.

Java Coding and Optimization FAQs:

http://www.cts.com/browse/wholder/Doug/FAQs

Benchmarks and Optimization FAQs

www.cs.cmu.edu/~jch/java

Tools & Products

You can download the reference JDK from Sun's website for free at http://java.sun.com/ products/jdk/1.2 and you should keep it available during development, if only for purposes of comparison. In general, the JDK represents "Java according to Sun", which means that issues not covered in the specification might be gathered from the accompanying HTML API documentation, or from the behavior of JDK-compiled and executed Java code.

IBM has put a lot of effort into promoting Java, and has released a bunch of freely available tools, such as a preliminary implementation of the Java Speech API. One particularly useful tool is their Java compiler, which can be used in addition to JDK's javac to check for potential compiler issues, found at http://www.research.ibm.com/jikes.

Notably, Jikes is available as open source. IBM is also offering commercial products, like its Visual Age IDE for Java.

A commercial product to compile Java to native code that supports many platforms is TowerJ, as http://www.towerj.com. Tools like this are a valuable fallback if you want to prototype and develop in Java, but you are afraid of the performance penalty.

A similar commercial product, albeit based on open source, is Cygnus' egcs/gcc-based GCJ:

http://www.cygnus.com/client_services/java.html

http://sourceware.cygnus.com/java/gcj.html

Cygnus is also offering a Java-capable IDE, the Source Navigator, at http://www.cygnus.com. Unfortunately, Cygnus's Java products are not yet integrated with their PlayStation2 development kit, so there is no readily available solution to get Java ported to a console.

A potentially very interesting solution for game developers aiming for a tailor-made VM with low memory footprint is Transvirtual's Kaffe Open Source JVM, at http://www.transvirtual.com, which can be modified and adapted to your needs and is available for licensing at reasonable conditions. It is a clean-room implementation of Java that has received quite a bit of acclaim, and it is definitely worth a look if you are wary of making yourself dependant on black-box, third-party tools.

Microsoft, recently forced by court order to support JNI, offers its own line of Java related tools http://www.microsoft.com/java.

Open Source

Java has attracted a lot of interest from the Open Source communities, despite the less than satisfying support for Java on non-Win32 platforms. It is worth noting that the JDK source is available from Sun under various licenses. If you do not intend to use a clean-room Java implementation, and instead you are considering creating a custom version of the Sun VM, it might be worth looking into the licensing details. In general, a deal with Transvirtual will probably give you more freedom at less cost.

Free clean-room implementations of Java serve well for research purposes, but are not yet ready for use. Examples include the GNU Japhar JVM at http://www.japhar.org and GNU Classpath core classes at http://www.classpath.org. Both are under the less-restrictive LGPL license and thus they are available for use in commercial products. TYA, at ftp://
gonzalez.cyberus.ca/pub/Linux/java, is another interesting project.

Other source for Java examples and sources are the GNU/FSF Java repository, at http://www.gnu.org/software/java/java- software.html, the Giant Java Tree at http://www.gjt.org, and the Gamelan Repository at http://www.gamelan.com.

In particular, there is the outstanding Java port of the ANTLR parser at http://www.antlr.org, which (like Sun's JavaCC) allows you to create Java parser classes for a given grammar. If you are looking for ways to compile a different language (say, Scheme) to Java bytecode, or you're looking for readily available script interpreters, there is a comprehensive list at http://grunge.cs.tu-berlin.de/~tolk/ vmlanguages.html which might contain your language of choice. Unfortunately, tools for converting C or C++ to Java are not readily available. The C2J and C2J++ conversion tools listed cover only a fraction of the languages. Another noteworthy reference is http://java.sun.com/products/jfc/tsc/tech_topics /generic-listener/listener.html, which provides source to handle runtime-interpreted Java source.

You might be skeptical about the relevance of Open-Source solutions and third-party compilers, but remember that id Software's Quake relied on the DJGPP compiler, and that Quake 3: Arena will use a derivative of LCC (see http://www.cs.princeton.edu/software/lcc). Incidentally, such C/C++ compilers could also serve as the starting point for creating your own Virtual Machine.

Java wraps native code in Prax War

Written by Billy Zelsnack during his days at Rebel Boat Rocker, Prax War was destined to be the first major game to use Java for most of its code. With the exception of a C renderer, the Prax War engine was written entirely in Java. Zelsnack explained the game this way: "We use JDK and JNI. The game itself starts from Java. I use Java as controller code for C. Java is very good at calling C code, but [it is] not necessarily as clean the other way around." The design kept raw data (such as textures and sounds) on the native side, but made them accessible to Java as needed. Billy Zelsnack found few problems with the core classes (at one point, UDP networking performance was an issue), and found no problems with the most feared Java component, the garbage collector. The engine used just two threads — one thread that listened for incoming packets, plus the main loop itself.

Unfortunately, Rebel Boat Rocker's publisher, Electronic Arts, decided earlier this year to cancel the project, stating that the game had "missed its technology window". If Electronic Arts had had the kind of faith in Rebel Boat Rocker that Sierra has shown in Valve Software, we might have found that their assessment was straight to the point — maybe Prax War missed its technology window by being too early.

Zelsnack summed up his Java experience up this way: "Java opened up possibilities for the product that could not have been realized without its power. It was one of the things I was most excited about and proud of."

id Abandons Java for Quake 3: Arena

John Carmack considered using Java in id's games for quite some time, ever since he announced that the company was leaning towards client-downloadable code for the Trinity project. "The QA game architecture so far has two separate binary .DLLs: one for the server-side game logic, and one for the client side presentation logic." Games that licensed the Quake 2 engine, notably Half-Life and Heretic 2, also wound up using client side .DLLs. However, with the hacking attacks on Quake 2 servers in mind, Carmack states that, "While it was easiest to begin development like that, there are two crucial problems with shipping the game that way: security and portability. If we were willing to wed ourselves completely to the Windows platform, we might have pushed ahead, but I want Quake 3: Arena running on every platform that has hardware-accelerated OpenGL and an Internet connection."

His solution: "I had been working under the assumption that Java was the right way to go, but recently I reached a better conclusion. The programming language is interpreted ANSI C. The game will have an interpreter for a virtual RISC-like CPU." Unreal followed a similar approach: companies that license the engine can opt to use compiled C or C++ code, and interpreted UnrealScript is available for homebrew scripting.

The advantages of using a C or C++ subset for your VM are obvious when it comes to handling legacy code. Ironically, it was Java portability problems that led id to develop the Quake 3 custom VM. Sun's promise of "write once, run anywhere" did not hold for the Invocation API on important server platforms, so Carmack decided to abandon the embedded JVM he had planned to use. "My ideal situation," he stated, "would be to include the interpreter in the QUAKE3.EXE, and just treat class files as data to be loaded and dealt with like anything else." Unfortunately, while this solution works fine on Win32 platforms, this was not guaranteed for Linux, OS/2, or even Solaris. "Having made the decision to do my own interpreter, I feel much more at ease not having to rely on anyone else's external code. When it comes around to the next development cycle, I will make the Java decision again." As for embedding: "We are still working with significant chunks of an existing code base. If I did want to go off and start fresh, I would likely try doing almost everything in Java."

Embedded Java in Vampire: The Masquerade

In a recent developer update, Nihilistic's Director of Technology, Robert Huebner, stated: "I've always been rather anti-Java; all the Internet hype surrounding the language was overwhelming. But after examining the language further, it was clear that it makes an ideal scripting language for games. The embedded Java API allows us to provide our designers with a subset of the Java environment, and the JNI interface allows us to provide hooks from the Java Virtual Machine (JVM) directly into the game engine." The new Java-based system for their next game, Vampire: The Masquerade, will replace a custom compiled language, COG, that the team used in its previous title, Jedi Knight: Dark Forces 2. According to Huebner, "The JVM is a lot faster than the systems we wrote ourselves; their kernel is more heavily optimized, the available Java compilers produce much more optimized object code, and the newest JVM systems include just-in-time (JIT) compilation to native instructions as a standard feature. And since the language is so much richer than our previous C-subset, it gives the designers a much wider range of possibilities." Because Nihilistic is developing primarily for Windows, it is able to apply solutions that were not feasible for id Software's multi-platform Quake 3: Arena strategy.