A comparison of Unity test frameworks
Test Automation is slowly getting adopted by the video game industry. The Game Quality Forum 2023 clearly indicates this shift. However there is a lack of comprehensible answers to this question.
If you’re trying to understand which are the existing test automation frameworks, this article will introduce, and enumerate pros and cons of the top 3 frameworks for Unity.
Table of Contents
Unity Test Framework
Unity provides its own test framework as a package. It is a simple way to add unit, integration, and end-to-end tests to your game, as it's just a few click away via the package manager.
Pros:
- Supported by Unity
- Easily integrated into any game project
- Full access to the game internals without serialization
Cons
- Mandatory C# knowledge required
- Mandatory compilation loop
- Sub-optimal UX and DevX
- No cloud support
Links:
AltTester
AltTester is an open-source UI driven test automation tool for Unity. It is an SDK in 3 parts: a Unity package, a server, and a driver. The driver sends commands to the server which are then forwarded to the SDK embedded in your game. The SDK allows the commands to pilot the game and provide information about Unity's objects.
While it might seems complex it follows the proven architecture introduced by Selenium for testing web applications.
Pros:
- Open Source
- Supports C#, Python, or Java
- Can be used together with Appium
- User input recorder (pro version)
Cons:
- Limited access to the game internals
- Mandatory network layer
- Limited cloud support
Links:
Game Driver
GameDriver is another test framework SDK for Unity. It follows a similar architecture to AltTester where an embedded component listen for commands sent via a network layer.
The main difference with AltTester is its proprietary nature, and its ability to work without a network layer if needed.
Pros:
- Powerful query language
- Supports C#, Python
- Can work without the network layer
- User input recorder
Cons:
- Require a paid license (but free trial available)
- Proprietary SDK and query language
- Limited cloud support
Links:
Embedded vs Networked architectures and why does it matter?
The main difference between these frameworks are their architectures. While AltUnity and Game Driver are instrumenting the game through a network layer, the Unity Test Framework is embedded into the game directly.
This impacts the way a team will use the framework. For example networked solutions allow to decouple the tests from the code base. This means you can use a different programming language than C# to write your tests. This also allows a stronger separation between the game developers and the people writing automated tests. This is sometimes desirable but also has consequences on the QA process.
On the contrary, having an embedded architecture encourage to be closer to the game's code. The same programming language is used and all the internal tools and abstractions of the game can be accessed freely by the test framework. If the game developers want to shift left, this can be a great asset, since they will be able to write more powerful tests than a separate silo-ed team.
A networked solution introduces inevitably serialization which can hinder the feasibility of some tests. For example it's relatively hard to meaningfully serialize every kind of object from Unity, and would require some specific code to send them properly on the other side of the network. This will lead mechanically to less powerful tests and longer time to write tests.
Conclusion
One should look beyond the basic lists of pros and cons for each framework. Ultimately the choice of framework depends on your game teams. How your teams collaborate will have an impact on your choice. If the team writing the tests is separated from the feature team, network based solutions can help you achieve that more easily.
However, you might want to shift left and have the feature teams write automation themselves. In this case an embedded solution will be a better choice. This type of solution will allow your developers to write more sophisticated tests, from unit tests to end-to-end tests.