Authors : Cihat Bostancı & Beyza Budak

1.1 Why Is Automation Necessary?

Today, technology, which enters every aspect of our lives, is an inevitable even in our simple daily lives. There may be negative experiences we encounter while using mobile applications and websites, which are indispensable for everyone. In order to reduce these experiences as much as possible, it is very important for clients to go through the testing phase before they are released.

Before we talk about the automation stage of these test stages, let’s briefly talk about manual testing. Manual tests are UI tests performed by humans in line with scenarios prepared by a test engineer. It takes more time than automation testing and is more likely to be mistaken. In automation, on the other hand, it tests from end to end according to completely written scenarios, it is fast and the probability of errors is very low compared to manual testing. In particular, regression automation tests run before each live are the savior of the whole test team in revealing better quality works and seeing that all areas work in harmony with each other. Automation is very important at this point, especially since store approvals of mobile apps can sometimes take even 1 day.

1.2 Which Tool Is Better Espresso or Cross Platform Tools and Why do we use Espresso?

Principally, after clearly determining all the project needs, the desired requirements, the necessary automation tool should be selected. By the way, Android and IOS applications are developing natively in the projects we are currently involved in. Although it seems to be suitable for cross platform, we are developing our automation test with native tools.

Secondly, we will be making mention of the needs and requirements of the project on the below. Besides that, although UI Automator and Appium etc. seem to be alternatives to Espresso, the table in the following made it easier for us to make this decision.

Consequently, in our opinion Espresso is much more suitable for our project.

2.1 Identifying Issues And Requirements

As we mentioned in the previous paragraph, we had to choose a tool, and our issues and requirements were at the forefront in this choice.

So, what are our issues and requirements while creating the automation infrastructure?

• Reducing the cost of testing on the manual side when the app goes live,
• Catching possible bugs in the live process faster,
• Improving the duration of regression tests to ensure that added features affect other screens (E.g. Being able to detect problems without skipping a place in big features),
• To be able to perform end to end tests where automation progresses like a user rather than a single screen test (In this way, to be able to observe that interconnected screens can work in harmony),
• It should be able to be divided into sections in order for the necessary tests to run at the necessary moments during the live process (E.g. the regression test when going live or the smoke test to be able to run if necessary to observe whether critical areas are affected),
• We also expected settings to be made so that our tests would work in different environments and for different users.

The key factor in adding these requirements was that we wanted to test the features with the whole application end-to-end. Below is an example scenario of one of the end-to-end tests we did.

2.2 Test Scenarios

First of all, we are testing a large e-commerce application, so our regression test cases had to be very detailed, to touch every point and to cover the entire application. Briefly, the screens we tested are items on the Homepage (stories, banners, widgets, see more button, favorites add — remove functions etc.), Account tab (my orders, membership information, registered cards, reminders, addresses, help, wallet, gift checks, coupons, live support etc.), Favorites tab (create -delete-share, favorite add-remove functions etc.), Categories tab (category selections, search fields (city, product, category) etc.) and we perform our tests on a wide variety of screens such as Product listings (suggestions, filter, city search, add-remove to favorites, banners, product details etc. ). You can take a look at the case we have added as an example to the cases we mentioned below.

Title: HomePage — Products similar to your recent visits on the homepage widget control

Description: After the product goes into detail, a product widget similar to the last one you visited should be displayed on the homepage.

Precondition: -

Test Steps:

1. The application opens with the live api selected
2. By clicking on the Categories tab, the first category is selected and the see all products sub-category is clicked.
3. It is observed that the product listing page appears and the first product is clicked on from the listing page.
4. After reaching the product detail, clicked on the back icon to reached the homepage.
5. It is observed that the widget appears on the screen by scrolling on the homepage.

2.3. How to Build The Infrastructure of UI Tests?

A UI Automation Test infrastructure is definitely directly related to the flow of the desired tests and the scenarios set up. In our example, the requests focused on real-time and written about features rather than mock servers and repositories. For our needs, we mostly performed the newly released features and old written features before and used our test in the test phase within the development cycle.

We derived our every written test from the Screen Test blueprint classes. Thus, we have provided the generic functions that are relevant to all our tests and the ones that should be specific standards. The main reason why we do it this way is that our android application is implemented with the Single Activity — Multiple Fragments approach. It made it easy for us to manage features such as toolbar, bottom change, navigation, getting text from resources etc. which are particularly relevant to Activity from a single point. In fact, we have tried to also manage everything directly the necessary actions about Activity in Screen Test with avoiding that becoming God Class.

Besides that, we tried to write View Actions, View Assertions and View Matchers for specific and/or generic solutions whenever necessary. Below you may see a few examples of where we need them. It has already clearly stated that we can apply this method in specialized problems, which android has suggested to us. In order to ensure maintainability, we realized through ids and scripts instead of specific text matchers. If the scenarios change, our tests will naturally have to change too. Surely, we conducted our test by approaching feature tests ( favorite, account, home feed etc. ) rather than fragment, view tests.

We have created test suite classes in order to perform the tests and at the required stage ( For example: Smoke Test Suite , Release Test Suite etc., Feature Test Suite)

3. How We Solved Our Problems And Experiences Shared

First of all, we noticed that Espresso is trying to perform actions as quick as to the views that are on the screen at that moment in the meantime. This made it easy to understand that when we click on something that is not on the screen, it throws illegal State Exception errors when we access No Matching View Exception. As a precaution, we wrote sleep codes that keep the main thread waiting. There were cases where we made the main thread wait when we had great difficulty. Mostly we tried to remove it from all our tests but a few code remained.

/**
 * For Main Thread sleeps
 */
fun sleep(millis: Long = 1000) = try {
    Thread.sleep(millis)
} catch (e: InterruptedException) {
    e.printStackTrace()
}

• However, since we do not know that API return time and when it is different, the sleep codes we set did not offer any performance and while main threads waiting, which we kept on a per-second basis were long in duration and did not occur as healthy tests. Therefore, we implemented the code that if the view is displayed on the screen, it can wait until that threshold is reached.

/**
 * Wait for view to be visible
 */
fun ViewInteraction.waitUntilVisible(timeout: Long = 4000L): ViewInteraction {
    val startTime = System.currentTimeMillis()
    val endTime = startTime + timeout
    do {
        try {
            isDisplayed()
            return this
        } catch (e: AssertionFailedError) {
            sleep(50)
        }
    } while (System.currentTimeMillis() < endTime)
    throw TimeoutException()
}

• Here, if it is not on the screen, we get errors because we waited for the second value, or because the renderer with animations tried to access our screens or directly with the pop up dialogs. Although it provides some solution, it did not work properly what we wanted, literally waiting until visible. In case of waiting until visible code is insufficient, we used the Base Robot class developed by Jake Wharton. That traverses the view by doing a breadth first search, and when it gets an error, it continues to try them, thus covering the views that come out according to the animation time, api return time, and the loss of time while rendering.
• To avoid thread sleep, and to handle synchronization with the background jobs, we should create our own Espresso Idling Resources. In this link we may understand how we could add and why we should use.
• Another function we used while writing our tests was the isDisplayedBool function. The reason we use this function is to handle the features that appear on the screen only at certain periods (such as pop-up that appears on the first login to the application) or that can be turned on or off according to the API. If it is on the screen, it returns true, if not on the screen, it returns false, and by inserting it into an if or while statement, we can direct it to the steps that should be seen in the relevant situation.

/**
 * View Interaction that checks item is displayed and return boolean
 */fun ViewInteraction.isDisplayedBool(): Boolean {
    try {
        check(matches(ViewMatchers.isDisplayed()))
        return true
    } catch (e: NoMatchingViewException) {
        return false
    }
    catch (e:AssertionFailedError) {
        return false
    }
}

• One of our problems was to take a variable number of incoming Edit Texts on a Linear Layout page and fill in all the fields. In order to achieve this, we had to write a View Action for this situation cause of no resource ids of Edit Texts and added view with manually. As in the code example below, we matched the Linear Layout and then used the children in the Linear Layout in the perform part and set the strings to all the EditTexts that matched in the children.

/**
 * View Action that takes the number of EditTexts in the Linear Layout and sets random strings to them
 */object LinearClickChildItemViewAction {
    fun clickChildViewWithId(): ViewAction {
        return object : ViewAction {
            override fun getConstraints(): BaseMatcher<View> {
                return object : BaseMatcher<View>() {
                    override fun matches(item: Any?): Boolean {
                        return isA(LinearLayout::class.java).matches(item)
                    }

                    override fun describeMismatch(item: Any?, mismatchDescription: Description?) {}
                    override fun describeTo(description: Description?) {}
                }
            }

            override fun getDescription(): String {
                return "Click on a child view with specified id."
            }

            override fun perform(uiController: UiController?, view: View) {
                val linearLayoutView: LinearLayout = view as LinearLayout
                linearLayoutView.children.forEach {
                    if (it is EditText){
                        val ed = it as EditText
                        ed.setText(getRandomString(5))
                    }
                }
            }
        }
    }
}

• Throughout the process, we had to come up with new solutions specific to situations. One of them was accessing and clicking the child of the Item View in the Recycler View. We wrote a custom view action for this. This View Action circulates the children in the itemView at the desired position in the Recycler View on the screen and clicks the desired child.

/**
 * [ChildItemViewAction] works for required then return ViewHolder view
 */
class ChildItemViewAction(val callback: (View?) -> Boolean, val position: Int, val itemId: Int) {

    var view: View? = null
    var currentPosition: Int = 0
    var isCallBackReturn = false

    /**
     * Get Child View From Recycler View
     */
    fun getChildItemView() = object : ViewAction {
        override fun getDescription(): String {
            return "Child Item View Action"
        }

        override fun getConstraints(): Matcher<View> {
            return allOf(
                isAssignableFrom(RecyclerView::class.java),
                isDisplayed()
            )
        }

        override fun perform(uiController: UiController?, view: View?) {
            val recyclerView = view as RecyclerView
            val adapter = recyclerView.adapter
            val itemCount = adapter?.itemCount ?: 0
            for (i in 0 until itemCount) {
                if (adapter?.getItemViewType(i) == itemId) {
                    if (currentPosition == position) {
                        recyclerView.scrollToPosition(i)
                        uiController?.loopMainThreadUntilIdle()
                        val itemViewHolder = recyclerView.findViewHolderForAdapterPosition(i)
                        isCallBackReturn = if (itemViewHolder != null) {
                            callback.invoke(itemViewHolder.itemView)
                        } else {
                            false
                        }
                        if (isCallBackReturn) {
                            break
                        } else {
                            uiController?.loopMainThreadUntilIdle()
                        }
                    }
                    currentPosition++
                }
            }
        }
    }
}

• In order to avoid errors for system animations both virtual or physical devices, Android recommends in the following; On your device, under Settings > Developer options, disable the following 3 settings:

1. Window animation scale

2. Transition animation scale

3. Animator duration scale

• We have different urls and test users for the different test environments. At the same time, in order to adjust the payment methods according to the test environments, we were able to solve the Test Constant Manager which is Singleton Class thus we could set all constants according to the Manager in a single point. We ran the test on what we desired values and made it independent from the environments.

References:

• https://developer.android.com/training/testing/other-components/ui-automator
• https://appium.io/docs/en/about-appium/intro/
• https://developer.android.com/training/testing/espresso
• https://academy.realm.io/posts/kau-jake-wharton-testing-robots/
• https://medium.com/androiddevelopers/android-testing-with-espressos-idling-resources-and-testing-fidelity-8b8647ed57f4