Detox: Gray Box End to End Testing Framework for Mobile Apps

Appium 

We used Appium for 2 years in general and for 8 months with React Native, and found that we invested an unreasonable portion of our time writing tests and petting the system than actually writing features.

We found that End to End testing is really hard:

Tests are flaky, we got different results on different machines, frequent failures in CI, which the only solution for was addition of sleeps, which slowed the tests down.

Tests were already slow since Apple UIAutomation tool is limited to performing one action per second, and there’s a hack which removes this cap Instruments without delays (which is already unmaintained), so after each release of a new Xcode we would have to wait for patch before upgrading.

Detox Gray box, not black box

Detox does Gray box, not Black box,

A Black Box Testing

to allows the test framework to monitor the app from the inside and actually synchronize with it.

Gray box essentially uses a piece of code that is planted in the app, it can help us see what’s going on inside.

Unlike Black box, Gray box runs in the same process, has access to memory, and can monitor the execution process. Being able to read internal memory gives it the ability to detect what’s happening inside the process: if there are network requests in flight, when the main thread is idle, other threads are idle, Animations have ended, the react native bridge is idle. It can execute on main thread, to make sure that when it performs actions nothing in the UI hierarchy changes in the meantime.

Uses EarlGrey and Espresso

The leading native gray box drivers are developed by Google — EarlGrey for iOS and Espresso for Android. These frameworks can synchronize with the application, making sure to only interact with the app when it’s idle.

The underlying synchronization mechanism used in these gray box frameworks works in the following way.

Instead of retrying an action/expectation on the UI, they will query internal resources every few milliseconds or listen to callbacks from them telling that they have now switched to idle mode. The test will not continue until all of them return yes and only then, when the app is idle, it will interact with the UI.

React Native support

Detox is built from the ground up for native mobile and has deep first-class support for React Native apps.

We found out that React Native pretty much reimplements iOS and Android, so apart from the basic synchronization support of EarlGrey and Espresso for native apps, we had to create special synchronization mechanisms for React Native as well.

Evaluating expectations on the device

Traditionally, test frameworks evaluate expectations in the test script running on the computer. Detox evaluates expectations natively directly in the tested app running on a simulator. This enables operations that were impossible before due to different scope or performance reasons.

Does not rely on WebDriver

Detox does not rely on WebDriver, since this is not the web. Detox communicates with its native driver (which extends EarlGrey and Espresso) using a JSON-based reflection mechanism, this allows a common JavaScript implementation to invoke native methods directly on the device.

How Detox Works

Let’s take a look at High level diagram, hopefully it will help us understand how Detox works.

High Level Diagram Detox Automation 

* Test Runner: Execution of an action or expectation (awaiting on a promise)

* Tester: Expectation being serialized into a nested invocation JSON

* Server: relaying a message

* Testee: Invocation of EarlGrey through method reflection

* Invocation will only execute when app is idle

* Testee: Invocation result is sent back through websocket

* Tester: resolve/reject the expectation promise

For more in depth information on how Detox works visit the docs.

https://github.com/wix/detox/blob/master/docs/Introduction.HowDetoxWorks.md#how-detox-works

UI Automation with Detox

Let’s get back to our testing pyramid.

So we now have a stable End to End testing framework. But it may still be flaky due to network and server issues.

In order to do that we would need to remove the dependency of tests in network, with expected requests and responses in a consistent well-timed manner we will create pure UI automation (UI hermetic tests).

A simple login flow test written with Detox

describe('Login flow', () => {

it('should login successfully', async () => {

await device.reloadReactNative();

await expect(element(by.id('email'))).toBeVisible();

await element(by.id('email')).typeText('john@example.com');

await element(by.id('password')).typeText('123456');

await element(by.text('Login')).tap();

await expect(element(by.text('Welcome'))).toBeVisible();

await expect(element(by.id('email'))).toNotExist();

});

});

Detox in Action

The surprising thing is that not only gray box is more stable than blackbox, it’s also much faster. No more sleeps or waitUntil, code executes the millisecond the app becomes idle. so it’s about 5–10 times faster than black box solutions.

In fact, it’s so fast it runs the full detox test project suite (79 tests) in 4 minutes.

Detox’s own test suite running on an iOS Simulator

Video Link - https://camo.githubusercontent.com/888c0d6cf1f13798062d7c7899443b6a114bd0f6/687474703a2f2f692e696d6775722e636f6d2f656f61444559702e676966

Detox’s own test suite running on an Android Emulator

- https://www.youtube.com/watch?time_continue=69&v=eb5Uqkk3uzo

Thanks for reading
Post author by Blog Admin Siddarth