‘Unit Testing’ Complex Systems

I was e-mailed by a co-worker the other week asking for advice on how to test a new feature that was implemented solely using private member functions and inaccessible objects within an existing system. It’s an interesting question that people often struggle with when trying to ‘unit test’ complex or high level systems when they are used to testing low level systems or classes where everything is accessible.

I use quotes around ‘unit testing’ because I think at times using the phrase unit testing becomes unhelpful when talking about system testing. I don’t know if it’s just the books I’ve read or the outlook of people I’ve discussed this with, but on the whole when someone says “I need to write a unit test to…” they almost always mean “I’ve just written something and now I must test the individual functions to make sure they all work”.

When testing a high level system, especially one that is being tested after the implementation is finished, people can struggle to take the processes they used to test lower level systems and apply them to more complex, interdependent objects.

Take the following unit test which checks that objects are being correctly destructed when a vector is cleared.

This is a pretty standard looking unit test. We have some input and we’re testing the behaviour of the function we’re calling and it’s affect on the data passed to it. At this point we’re working at a pretty low level, so we have the opportunity to test components almost independently. Sort of.

These functions might be low level, but they are still built up of individual (and often inaccessible) components. All these functions are calling other functions we don’t have access to and we’re obviously not testing the individual lines of code, which will be generating multiple lines of ASM. We can accept this because we’re both used to it and we simply have to. We don’t try to test the internal implementation because we know it works based on the input and generated output of the tested functions.

We have other tests which check the individual elements of this test (push_back() adding elements and clear() reducing it’s size to 0), and basing our more complex test on these allows us to concentrate on the more complex behaviour (the destruction of objects inside the vector). It’s this mindset of testing independent elements and basing more complex tests on simpler tests that needs to be expanded as the systems become more complex and more inaccessible.

Take the following use case as an example.

A title can only communicate with a the game save system using messages. It has no access to the components of the game save system at all, and can only request processes (like asking for data to be saved) and wait for a result message (like the results of a save operation). The system has the following behaviour which we want to test – when a user sends some data to be saved, if the data is the same as what was previously saved the save doesn’t take place at all.

Implementation wise, this is done completely through the use of private functions so nothing other than the internal object itself has access to the data and how it behaves. So what’s the approach here, because we certainly don’t have the opportunity to run what people would call ‘standard unit tests’ on this system as it stands.

The first question to ask is if the system itself is structured in the best possible way. For example, we have an object which is both carrying out the logic of saving but also the management of the data itself. As a result the data management aspect is hidden deep inside the system, making it both impossible to access and impossible to test but knowing the data is managed correctly is crucial to testing the whole system.

So the first approach would be to extract the data management element into it’s own structure. Having an independent object responsible for caching the data and performing any operations on the it (in this case checking if the new data is the same as the old) not only improves maintainability (under the single responsibility principle), it also allows us to test the specific data management step of any save process independently before plugging it into the system as a whole.

Now we’ve extracted what should be an independent element of the system (and directly tested the same data checks process), we can look at the rest of the system. We already have tests for the low level save API, so all we now need to test the user side behaviour.

We know the following behaviour should exist for the user
* When a user requests a save, the data, if different, is saved
* If the data is the same the system reports ‘success’ at the point the save was requested
* If the data is saved the system reports success after the save has completed

So based on this we can now test the system as a whole, knowing that we’ve extracted the independent systems into their own lower level unit tests with these tests concentrating on the high level behaviour of the save process as a whole.
* Requesting a save doesn’t return a success until the data has been saved. We can tell this via the content of the result message and the time it takes to arrive
* When requesting another save with the same data, the result should be instantaneous – also indicating success – which will only happen if saving doesn’t occur
* When following that with changed data, the save successfully completed and it doesn’t return instantaneously

Because we’ve extracted the data management structure into its own test, we can safely assume that element of the system works and it’s the high level, end user experience, that we’re testing in exactly same way in which a user would use it.

#define protected public
Opening up the system is often suggested as a way of testing the ‘internals’ of an object but the more testing deviates from how people use a system the less the tests can be relied on in a real life situation. It’s all good and well a test passing independently of everything else, but if it falls over when used by a client then testing has failed.

Changing protected or private to public (with a method of your choosing) only increases the difference between test usage or real-world usage and quite often internal structures or algorithms may well not work independently, leaving you to replicate what the object would be doing anyway which can easily lead to over-mocking your tests.

So it opens up the question of how much your system is doing and how much of that work could be extracted into more independent objects that are both easier to test and (more importantly) easier to maintain. In these cases making the system more modular or changing how the data is handled will improve both testing and code use without compromising encapsulation and use case testing.

This post was originally published on Engineering Game Development on the 14th November 2010.