Initial Experiences with Live User Test for Games

I have always been interested in Live User Test since I first heard about it, but I have never had the opportunity to try it out until now. Last year I wrote an article about my theoretical thoughts on Live User Test [1], and I thought I would follow up on that now that I have some practical experience with using this form of testing for finding and reproducing bugs.

We had problem reported by a fraction of our customers that we had not found in our internal testing, and could not reproduce once we knew about it. Either it was intermittent or environment or configuration dependent. Developers were analyzing the problem, but we also had the option of using the services of a crowd testing (live user test) company, so we jumped at the opportunity.

I had my preconceived views of that it would require a lot of time from me to set it up, but I was happily surprised when it took me only 5 minutes or so to set everything up on my end. This was because the problem we had was quite easy to find under the right circumstances, and the tests I had to request were very easy to write down.

An hour after I had sent in my request, I received a link to an excel sheet, which immediately started to get populated by tester names. I saw my test cases written in different columns, and the testers, who had already downloaded the build I had sent, immediately started testing.

All in all 150 testers from different countries with different environments, configurations, devices and operating systems executed the 4 tests I had sent them, and out of those 150 around 5% actually had the problems our customers had.

Suddenly we had a lot more information. We knew something about under which circumstances the problem occurred.

When our developers produced a fix, it was easy to send it to the 5% of the crowd testers, who had experienced this problem, to see if the fix worked for them.

After this had been verified we could send the fix out to all our customers and had finally solved the problem.

The experience I had with crowd testing confirmed my prior beliefs in how to effectively use this form of testing, when trying to find or reproduce bugs.

Small, focused, easy-to-understand test scopes with a clear objective is key to avoid unnecessary overhead, such as going through large amounts of bugs that have already been reported or are working as designed.  As I wrote in my previous article I think it is most suited for finding intermittent bugs, bugs that are dependent on environment or configuration, bugs which require a lot of simultaneous users or bugs which require certain combinations that can be hard to cover in a normal test scope.

I was positively surprised by the whole experienced, and it worked much smoother and more effectively than I had thought. I will definitely try to use live user testing more in the future, when the circumstances are right.

References

[1] Initial Thoughts on Live User Testing for Games

http://www.gamasutra.com/blogs/JohanHoberg/20141023/228441/Initial_thoughts_on_live_user_testing_for_games.php