Unity Automation – Approaches,Problems, Solutions

1. Introduction

Hello everyone. My name is Losev Viktor, and I am a software developer in test engineering. I work in the Capgemini company. My career has turned out in such a way that, in addition to creating solutions for automation of testing for Desktop, Frontend, and Backend applications, I was lucky to be engaged in creating solutions for automation for Unity applications.

As you probably know, Unity is a cross-platform environment for the development of computer games, developed by the American company Unity Technologies. Unity allows the creation of applications on over 25 different platforms, including personal computers, game consoles, mobile devices, internet applications, and others.

Unity autotests can be very capricious, and many sleepless days and nights may be needed for debugging and maintenance.

I hope this article will help you save money, time, nerves, and energy on the winding road of Unity automation.

2. Types of Automation

Unity Test Framework supports two main types of tests: editmode and playmode.
What is the difference between them?

2.1 Editmode Tests

Tests are executed in the editor, without running the scene. They are used for unit testing.

What is important to note:

• The scene does not load.
• There is no yield return anywhere, and the method is simply void, it does not return IEnumerator as in Playmode tests.
• When running the Edit mode test, the code is compiled, but the scene is not loaded and objects are not rendered.
• Edit Mode tests are primarily intended to test your code logic, and not to evaluate the performance of the GPU and the generation of user actions.
• The Edit Mode test is a regular unit test.

If you release your product under different versions of Unity, let’s say you make some plugin, asset or package — you must run unit editmode tests on EACH version of Unity that you support. You may be unpleasantly surprised that, for example, your unit tests perfectly pass on version 2022 and fail on Unity 6.2.

And yes, run editmode tests exactly in Unity Editor. There are smart guys who tried to run through NUnit — do not do that.

If your Unity application uses backend, you do not need to write Backend API autotests as unit tests in editmode, by this you will only increase the execution time of the test suite. With such an approach you will spend time on building the whole project and loading Unity Editor. Use a separate job and repository — for C# the stack C#, NUnit, ReFit, Allure will work perfectly.

2.2 Play Mode Tests

They are launched inside the game cycle. They are suitable for integration, functional testing — interaction of objects, UI and animations.

What is necessary to pay attention to? Here everything is the opposite, not like in editmode tests, namely:

• The scene is loaded.
• Thereis not only compilation, but actually the launch of the whole application — rendering of objects, loading of assets, textures and everything else, and the most important — emulation of user actions and verification of the corresponding reactions of the application to user actions.
• Before launching a playmode test the developer can choose the simulation of the device on which supposedly this autotest is launched:
  For example, if for the mobile version of your application you can choose a simulation of iPhone or Android.

3 Open CV

For some tasks, it is more than appropriate to use computer vision technologies.

Here are 2 algorithms that are worth paying attention to:

3.1 Template Matching

Automation testing using OpenCV template matching involves using the cv2.matchTemplate() function to automatically find and locate a pre-defined template on a large image, for example, to find specific objects or interface elements in a video stream or in a series of images. This is achieved by calculating the degree of similarity between the template and each possible position on the main image, which allows you to automatically find all occurrences of the template without manual intervention.

IN SIMPLE WORDS: You have screenshots of UI buttons, the algorithm finds their coordinates on the screen, and the autotest generates user actions.

Here is a great article in English about the Template Matching algorithm

3.2 Tesseract (you can also ChatGPT instead)

Tesseract is a library for OCR, a specialized subset of CV, designed to extract text from images.

You can take a screenshot of the user interface of your application (or part of it), and convert it from an image that contains some text - into a variable of the string type.

Example:

From image

You will receive a string variable with value : “Tesseract OCR”.

Why do we need to use OPENCV in Unity automation, if autotests in playmode can generate any user actions and can get any data from any rendered controls and objects?

Of course, this is true, but there is one nuance - any playmode autotest is launched in Unity Editor. Yes, with 90% probability we can say that if the autotest passed in Unity Editor, the tested functionality will work perfectly on a real build.

But the thing is that Unity applications can be built for very different platforms

• Windows
• Mac
• Mobile Android
• Mobile IPhone
• WebGL (essentially, the unity application will work in a web browser)

From personal experience, I can say that there were the following situations - an application built for Windows works fine, but when we make a build for WebGL, everything goes wrong... in general, it doesn't work completely.

And playmode autotests will not catch this bug!

What is the solution?

We can create an autotest using Selenium Webdriver or Playwright that will open the page in the browser and, having taken a screenshot, will be able to generate user actions and check the application's reactions to user actions via OpenCV.

Such an autotest will be “external” to the application and will be launched independently of the Unity Editor, generating user actions on a real build, and not in the emulation of the Unity Editor playmode.

4 Tips for writing tests

If we are talking not just about a game, but about a 3D application that has a developed user interface (for example, CAD, or a 3D editor, a 3D printing management system, etc.), then our autotests will have to generate user actions for this UI.

And there are 2 main libraries that are used to create a user interface in Unity: UGUI and Unity Ui toolkit.

UGUI (Unity UI)

This is an old Unity interface system based on game objects (GameObject) and using standard prefabs, with a WYSIWYG editor in the scene.

UI Toolkit

This is a new, modern system that offers more flexible tools for creating user interfaces and editor extensions, using web technologies (UXML, USS) for separate definition of structure and styles, which speeds up development and debugging.

Key differences

Let's look at an example:

As we can see, the autotest does the following - it loads the scene and checks the button, which, as we have already discussed, is a separate gameObject.

If we use the UI Toolkit, then the principle works differently - the entire UI layer is one gameObject in which we are already looking for the user interface element that we want to access.

Let’s check example:

The most important modules of Unity automation are Waiters, that is, methods that wait for a user element to appear, or wait for the next frame to load, or another condition.

• 1 WaitUntil

  yield return new WaitUntil(() => GameObject.Find("Enemy") != null)
          

  Wait until gameObject “Enemy” appears.
• 2 WaitWhile

  yield return new WaitWhile(() => isLoading)
          

  Wait until the condition is met
• 3 WaitForSeconds

  yield return new WaitForSeconds(2f); 
          

  everything is simple here - wait 2 seconds
• 4 Wait ForEndFrame

  yield return new WaitForEndOfFrame();
          

  Waits for all rendering and layout calculations to complete for the current frame.
• 5 Normal return null.

  yield return null;
          

  Waiting for the next frame.

5 Waiter for UI Toolkit vs UGUI objects

Now let's think about this:

If we write autotests for UGUI, we need the following conditions before accessing a UI element: the button's gameObject is loaded, rendered, ok, let's wait until the next frame starts, and the autotest can access the UI, which is expectedly present and fully loaded.

When we write autotests for an application that uses the Unity UI Toolkit, we can have this situation:

The overall gameObject of the entire user interface is already loaded (it is called UIDocument), but the child user elements that are not separate gameObjects are not yet loaded.

So before accessing them, we should check that these child elements are available, loaded and drawn.

Let's look at an example:

6 Continuous Integration

Now let's talk about CI setup.

The first thing to note is that Unity Test Framework does not allow us to run all editmode and playmode tests in one run. Therefore, we can either have 2 jobs for each test type, or we give the user of our CI system the ability to choose which type of tests we run.

The second thing to note is that running autotests on CI (and therefore via the command line) can be done in two ways:

1 Headless run

unity-editor -runTests -projectPath -testPlatform playmode -batchmode -quit

This command has the -batchmode key, it is used to run Unity auto tests in HEADLESS mode, that is, it runs the Unity editor in batch processing mode, without a graphical interface. In this case, rendering is emulated in a console application.

2 Run without the batchmode key.

Here is an example of the command:

unity-editor -runTests -projectPath -testPlatform playmode -quit

In this case, Unity Editor is loaded with a graphical interface, everything is rendered as expected.

Therefore, for editmode tests, you should use the -batchmode key, and for playmode tests you should not use the -batchmode key, since we need to launch Unity Editor with a graphical interface and real rendering.

7 Performance testing

Certainly, the most important area for automation. Unity application development requires the developer to use code optimization techniques to improve performance, and this is possible, if we have unity application performance metrics. During application development, we ensure that the metrics of the next version of the application are not worse, and ideally better than the previous ones.

What metrics do we need?

1. RAM load monitoring.
2. CPU load monitoring.
3. Load distribution between CPU and GPU.
4. Scene loading speed.
5. Loading speed of certain objects.
6. Loading speed of Unity application data from Backend.
7. Check that FPS remains above a specified threshold (for example, 30 fps) when loading assets.

When you have a list of upstream metrics, you can clearly understand whether your application's performance is improving.

Advice - it is much better to run performance autotests that measure CPU and RAM load on a genuine Windows or MAC, most importantly, not on a virtual machine. This is not very convenient and correct from the CI point of view. Still, it is essential to understand that to get a metric for monitoring the load distribution on the CPU and GPU, the autotest must be run on a real PC with a physical video card. You mine crypto on physical farms with real video cards.

7 Conclusion

The last advice, please use this line of code in autotests:

LogAssert.ignoreFailingMessages = true;

This will allow autotests not to fail due to some random messages from Unity Engine.