Parallel Test For UI Automation With Fladle

Authors : Beyza Budak & Cihat Bostancı

1.1 What is Parallelization For UI Automation?

In our previous article, we talked about android with espresso ui automation. When the tests started to settle down slowly, we moved on to the CI/CD process, which is our next step. One of the most important elements for us while establishing this process was that we wanted to run our tests in parallel. It is called paralleling test functions to run on specified devices either by dividing them into devices or if it is desired that all tests run on different devices, at the same time on those devices. Since our test suite is large and we run on Firebase Test, we tried it to work by dividing it into devices in the first stage. The reason for this was that the package we used in Firebase Test gave 60 minutes daily on virtual devices and 30 minutes on physical devices. By using parallelization, we have achieved the goal of both running the tests in a shorter time in the live releases of the applications and not exceeding the pricing. We were able to test important functions on different devices at the same time by running the same cases on more than one device in smaller smoke packages.

1.2. Why Do We Use Fladle?

Our expectations from Android UI automation were to run parallel to the firebase when triggered from a single place and to set the features we want (Record Video, Test Shards, Environment, Reporting, Device, and Target, etc.) for the test run over the task. For these features we wanted, the most suitable Test runner choice for us was Flank. That is a massively parallel Android and iOS test runner for Firebase Test Lab. Flank is also compatible with gcloud firebase test, allowing you to see detailed reports of our tests on gcloud firebase test. We used the Fladle plugin to trigger our test cases from Gradle and send our tests to Firebase via Flank. Previously, Firebase Test Plugin was used in our project, but since we could not run parallel tests in addition to old-fashioned, and not supported anymore, consequently we implemented Fladle task for our android project. Since Fladle is an open-source development, it is a constantly updated and supported plugin. There is also a large community where we can write down the mistakes we have received and find our answers. It provides the opportunity to run tests in firebase with a single command we send from the terminal.

2.1.Fladle In Detail

As we mentioned above, Fladle is actually a Gradle task and makes our work easier with a single point on android applications. And Instead of setting gcloud firebase test functions at length, it came to us to manage it from a single place with a single task and in line with the needs we predetermined. So, what kind of features can we use in the Fladle? The answer to this question is that it is directly working on the capabilities that gcloud firebase test service offers us. The document clearly tells us which utilization and what we can trigger on firebase. In order to integrate into the CI/CD processes, which are of great importance for us and mentioned above, triggering the tests that we wrote on a single task and each release, smoke, regression, etc. Another major reason is to clearly run our tests in parallel, to take advantage of the convenience provided by the Fladle in this regard.

Basically, with a simple Gradle task, as you may understand how easy it is to set up an easy configuration on Firebase Test Lab. Fladle works as a plugin with the attributes we set through the service gcloud firebase test offers.

We can divide it into shards with the Fladle task, and we can run different tasks in the desired terminal. At the same time, you can obtain both test UI videos and test result documents for reporting.

2.2 How to Integrate Fladle?

First of all, sensitive points are clearly explained in the Fladle docs. Here we will state how we implemented our own integration and what mistakes we got in which situations. In the Beginning, do not forget to run the flankAuth command once and log in. After that, The key point here is that assemble tasks must be run before each task runs which are in below with an order.

./gradlew flankAuth
./gradlew app:assembleDebug
./gradlew app:assembleDebugAndroidTest

Here’s how our assemble tasks worked. Actually, the goal was to run our tests only in our debug apps, and the app name had to be found in between.

./gradlew app:assembleAppNameDebug
./gradlew app:assembleAppNameDebugAndroidTest

The task that is sufficient for us and that we need is as follows.

fladle {
    serviceAccountCredentials = \
        project.layout.projectDirectory.file("service credential json file")
    smartFlankGcsPath = 'gs://flank_data/results/JUnitReport.xml'
    projectId = "Firebase project name"
    localResultsDir = 'flankResults'
    useOrchestrator = true
    recordVideo = true
    performanceMetrics = false
    maxTestShards = 3
    variant = "app Name"
    configs {
        perfTests {
            devices.set([
                    [ "model": "Pixel2", "version": "28" ]
            ])
            testTargets.set([
                    "class test 1 package directory", "class test 2 package directory"
            ])
        }
        regressionTests {
            devices.set([
                    [ "model": "Nexus6P", "version": "23" ]
            ])
            testTargets.set([
                    "class test 3 package directory",
            ])
        }
    }

    environmentVariables = [
            "coverage"        : "true",
            "coverageFilePath": "/sdcard/Download/"
    ]
    directoriesToPull = [
            '/sdcard/Download'
    ]
}

Finally, we run over firebase with the commands ./gradlew runFlankperfTests and ./gradlew runFlankregressionTests from the terminal. The test names will be understood through the suffixes we have added to the flank task command of the names we have added to the Fladle Gradle task.

3.1. How We Solved Our Problems And Experiences Shared

Our most critical and unnoticed problem was to obtain the desired JSON credential file to pass with the firebase test lab. For this, we learned that the admin role is required so that Firebase can access and create authority. It clearly states the method of obtaining permission and JSON credentials file by following the steps below.

Service account credentials

1. Create a service account. Service accounts aren’t subject to spam checks or captcha prompts, which could otherwise block your CI builds. Create a service account with an Editor role in the Google Cloud Platform console.
2. Enable required APIs. After logging in using the service account: In the Google Developers Console API Library page, enable the Google Cloud Testing API and Cloud Tool Results API. To enable these APIs, type these API names into the search box at the top of the console, and then click Enable API on the overview page for that API.
3.After creating the account go to Keys, click Add Key -> Create a new Key. Select JSON for the key type. This will download the json credentials.
4.Configure the Fladle extension serviceAccountCredentials to point to the credentials.
Above instructions are based on Google instruction for authenticating with CI.

Above details of Service account credentials information were taken from this.

As you may have noticed, at first, we can give our test files in the form of a test suite and package instead of listing the directories one by one. When we applied the test suite, an error occurred due to the java virtual machine. For this reason, we collected our tests as packages and gave them in one place.

RunTests java.lang.IllegalArgumentException: Invalid argument: com.ciceksepeti.utils.TestSuite
at ftl.filter.TestFilters.parseSingleFilter(TestFilters.kt:101)
at ftl.filter.TestFilters.access$parseSingleFilter(TestFilters.kt:35)
Total run duration: 0m  5s

One of our biggest problems was the pricing issue we mentioned earlier. To overcome this and to complete our large test suites in a shorter time, we researched parallel running. In our research process, we concluded that we can do this most conveniently by using Fladle via Flank. It allows us to run our tests in parallel by choosing Smart Sharding in Fladle, or by dividing our tests into devices, by giving the number of devices we want it to run on.

When we wanted to trigger the Gradle task after completing our configurations, it was giving us the error “At least one supplied APK file has a missing or invalid signature”. This is because we needed to build apk before running the Fladle task. Therefore, before running each test, we need to run both test and apk assemblies in Gradle. Otherwise, we cannot run our tests because we do not have apks to run tests on.

./gradlew app:assembleExampleDebugAndroidTest
./gradlew app:assembleExampleDebug

If you want to see which devices and API levels firebase supports while choosing a device, you can use the “gcloud firebase test android models list” command. Here, there are device types supported by the firebase test lab on both physical and virtual devices. At first, we started to run our tests at API level 30, but we could not pass the save your password screen due to google smart lock. Because it could come out with a different language according to the device language, we had to create a resource list accordingly. For this reason, until we solve this problem, we decided to run our tests on a device with API 23, one of the devices we previously ran with the firebase test lab plugin. However, this time, we increased the API level again because the tests progressed very slowly on devices with low API levels and we had problems with page loading. At this point, it is best to test at the most used API level by looking at the statistics, if a single API level is to be controlled (which is a more effective test to see how the application works at more than one level).