Automated game testing is still in its infancy

Writing software is highly abstract and it’s difficult to keep the whole picture in one’s head, especially when the parts of said software change continuously. Writing tests is, for some, an obvious solution to tackle the complexity and make sure the developed software works the way it has been intended.

Video games are very complex software themselves. Most games rely on strict systems, and any broken behavior becomes very visible, very quickly. Some games are also very large, with content spanning hours of gameplay, with unique voice-over, cut scenes, puzzles, etc. That’s why maybe more than any other software, video games have to be tested over and over.

Despite this need, the game development industry is poorly equipped to test games.

A fragmented space

The game industry is very fragmented. While Unreal is the leader in the image above, many successful games have been built on different engines like GameMaker, Source, CryEngine, or even their own in-house engine. This might explain why we have yet to see the emergence of a mature ecosystem for automated game testing.

With Unity becoming more popular, the question “how can we test our game automatically” has been provided an answer by the company: the Unity Test Framework. As for the Unreal Engine they have their own testing framework called Gauntlet.

However these are fairly recent creations, and we can consider automated game testing to be in its infancy. As such it’s not difficult to consider the game industry far behind others in this regard, as there is an unfulfilled need for better tools.

Learning from the web and mobile apps industry

Quality assurance in the Web industry is a story of fragmentation too. QA teams need to test on many different operating systems, and with many different browsers. On top of that, it’s very rare to come across websites without any Javascript nowadays. This means you have to run tests with an entire browser context, or else some specific features can’t be tested. Naturally it increases the complexity of testing automation.

Despite this, web developers have overcome these barriers by building tools that are now very mature. Selenium is one of these tools, and its protocol is now an official recommended standard for browser automation.

Mobile testing has followed a similar development, with a highly fragmented space of different operating systems, hardwares, as well as native and web apps. The mobile industry came up with tools to break these barriers. For example: Appium is an open source testing framework, highly inspired by Selenium. Whereas Google and Apple provided their own native testing solutions in the form of Espresso and XCTest.

These tools have been consolidated through an effort of standardization, either by the owners of their respective platforms (like Apple providing native testing for iOS) or by the open source community itself.

Eventually this consolidation allowed the birth of a myriad of testing clients, frameworks, and secondary tools. Today these tools are even supported by many different companies — making this ecosystem vibrant and allowing it to flourish.

Our mission

For game testing there is a long way before we can reach the level of maturity that exists in the web and mobile industry.
That’s where Game Conductor comes into play. Here is the mission we’ve given ourselves:

• We want to advance the maturity of automated game testing by providing a continuous testing platform
• We aim to foster discussions around best practices and advanced techniques for QA and automation engineers
• We will also leverage our knowledge of other industries to bring relevant and known practices in the game development ecosystem

Continuous testing is the key to improve the quality of your games. It allows for faster bug fixing, the prevention of redundant work by manual testers, and provides a useful overview of your game quality.
If you are interested, follow the Game Conductor platform on LinkedIn.