yksp

let 'em test 'emselves

This project is maintained by groundupworks

A fully Python-based tool that helps you automate Android UI testing and captures everything you need from each test session.

You can see yksp in action here. Higher abstractions of UI interactions used to compose test scripts are available through AndroidViewClient by @dtmilano. This project will focus on tools to inspect, correlate, and validate the set of generated test results. Pull requests are, of course, welcome.

Getting set up

This assumes your environment has python, easy_install and git. You may confirm each installation by typing the following commands in your Terminal.

$ which python
$ which easy_install
$ which git

step one

You may skip this step if you already have the Android SDK installed.

Download the Android SDK. You may download the stand-alone package for your platform instead of the IDE bundles. After extracting the package, you need to download the following tools, as they are not included in the SDK package.

  1. Android SDK Tools
  2. Android SDK Platform-tools
  3. Android SDK Build-tools

You will download these tools by using yet another tool, which is included in the SDK package. Replace android-sdk-root with the local path to your extracted SDK root directory.

$ cd android-sdk-root/tools
$ ./android list sdk --all
$ ./android update sdk --no-ui --all --filter 1,2,3

step two

Install Pillow and AndroidViewClient.

$ cd ~
$ sudo easy_install --upgrade Pillow
$ sudo easy_install --upgrade androidviewclient

step three

Clone the yksp repo, and AndroidViewClient as a submodule.

$ git clone git://github.com/groundupworks/yksp.git
$ cd yksp
$ git submodule init
$ git submodule update

step four

Set the environment variable $YKSP_HOME, and also $ANDROID_HOME if not already set. Then configure your $PATH.

On a Mac

$ sudo nano ~/.bash_profile

On Ubuntu

$ sudo nano /etc/profile.d/yksp.sh

Add the following text, replacing android-sdk-root, yksp-root, and build-tools-version with your local paths. Log out and log back in.

export ANDROID_HOME=android-sdk-root
export YKSP_HOME=yksp-root/yksp
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/build-tools-version:$YKSP_HOME

Mine looks something like this on the Mac.

export ANDROID_HOME=/Users/benedict/Dev/android-sdk-macosx
export YKSP_HOME=/Users/benedict/Dev/projects/yksp/yksp
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/20.0.0:$YKSP_HOME

And... you're all set!

Running the example

The yksp repo has an examples/flying-photo-booth-tests folder, containing the Flying PhotoBooth APK, along with its PyUnit test scripts in a scripts folder. Connect one or more Android devices to your computer. Ensure that the device is made USB debuggable and screen lock is disabled (like, completely off... not even Swipe Unlock). This is how you would run a typical yksp test session.

$ cd $YKSP_HOME/../examples/flying-photo-booth-tests
$ yksp

Oberserve the test session logs in your Terminal, it should look as follow if everything is set up properly. Results are stored in the newly created results folder, with each test session in a folder named with its own timestamp.

$ yksp
Scanning for APK...
APK: flying-photo-booth-release.apk
Inspecting APK...
Package name: com.groundupworks.flyingphotobooth
Version name: 3.4
Version code: 13
Scanning for test scripts...
Script 1: testAbandonLinking.py
Script 2: testCapture.py
Listing devices...
Device 1: EP7309229R
[EP7309229R] Collecting device properties...
[EP7309229R] [ro.product.manufacturer]: [Sony]
[EP7309229R] [ro.product.model]: [C6502]
[EP7309229R] [ro.build.version.sdk]: [16]

[EP7309229R] Installing APK...
[EP7309229R] Preparing test case 1 of 2 [testAbandonLinking.py] on C6502...
[EP7309229R] Wiping app data...
[EP7309229R] Clearing logcat buffer...
[EP7309229R] Start printing logcat output to file...
[EP7309229R] Executing [testAbandonLinking.py] with command:
python results/2014-09-24-09-34-28/C6502-[EP7309229R]/testAbandonLinking/testAbandonLinking.py --package com.groundupworks.flyingphotobooth --serial EP7309229R --root results/2014-09-24-09-34-28/C6502-[EP7309229R]/testAbandonLinking --logs pyunit.txt --screenshots screenshots --screendumps screendumps
[EP7309229R] Logcat output saved
[EP7309229R] Downloading app data from device...
[EP7309229R] Extracting app data...

1947+0 records in
1947+0 records out
1947 bytes transferred in 0.002251 secs (864984 bytes/sec)
x apps/com.groundupworks.flyingphotobooth/_manifest
x apps/com.groundupworks.flyingphotobooth/db/webviewCookiesChromiumPrivate.db
x apps/com.groundupworks.flyingphotobooth/db/wings.db-journal
x apps/com.groundupworks.flyingphotobooth/db/wings.db
x apps/com.groundupworks.flyingphotobooth/db/webviewCookiesChromium.db
x apps/com.groundupworks.flyingphotobooth/sp/com.facebook.SharedPreferencesTokenCachingStrategy.DEFAULT_KEY.xml
x apps/com.groundupworks.flyingphotobooth/sp/com.groundupworks.flyingphotobooth_preferences.xml

[EP7309229R] App data downloaded
[EP7309229R] Wiping app data...
[EP7309229R] Preparing test case 2 of 2 [testCapture.py] on C6502...
[EP7309229R] Wiping app data...
[EP7309229R] Clearing logcat buffer...
[EP7309229R] Start printing logcat output to file...
[EP7309229R] Executing [testCapture.py] with command:
python results/2014-09-24-09-34-28/C6502-[EP7309229R]/testCapture/testCapture.py --package com.groundupworks.flyingphotobooth --serial EP7309229R --root results/2014-09-24-09-34-28/C6502-[EP7309229R]/testCapture --logs pyunit.txt --screenshots screenshots --screendumps screendumps
[EP7309229R] Logcat output saved
[EP7309229R] Downloading app data from device...
[EP7309229R] Extracting app data...

567+0 records in
567+0 records out
567 bytes transferred in 0.000659 secs (860409 bytes/sec)
x apps/com.groundupworks.flyingphotobooth/_manifest

[EP7309229R] App data downloaded
[EP7309229R] Wiping app data...
[EP7309229R] Uninstalling application...
All 2 test scripts executed on 1 device in 93 seconds

what just happened?

The test session logs should give a pretty good idea of what yksp is doing, but here is a better view.

Find APK from your root directory
Find test scripts from the scripts folder
Find connected devices and write serials to serials.txt
For each device, run in parallel:
    Collect device properties and write to device.txt
    Install APK
    For each test script:
        Make a local copy of the test script
        Wipe app data
        Start logcat
        Execute local copy of the test script:
            Connect to and wake device in YkspTestCase.setUp()
            Run each test...() method in your YkspTestCase subclass:
                For each YkspTestCase.saveScreen() call:
                    Save screenshot to the screenshots folder
                    Save view tree dump to screendumps folder 
            Kill app in YkspTestCase.tearDown()
            Write PyUnit test results to pyunit.txt
        Stop logcat and write to logcat.txt
        Dump app data to data.ab
        Extract data.ab to data folder
        Wipe app data
    Uninstall application
Report completion of test session

where are my results?

Test results are organized in a directory structure that would allow a script to easily walk the tree and generate reports from a data set involving multiple devices and test sessions.

results
    <DATETIME>
        serials.txt
        SERIAL-[MODEL]
            device.txt
            SCRIPT1
                SCRIPT1.py
                pyunit.txt
                logcat.txt
                data.ab
                data
                    ...
                screenshots
                    0-tag0.png
                    1-tag1.png
                    2-tag2.png
                screendumps
                    0-tag0.txt
                    1-tag1.txt
                    2-tag2.txt
            SCRIPT2
                SCRIPT2.py
                pyunit.txt
                logcat.txt
                data.ab
                data
                    ...
                screenshots
                    0-tag3.png
                    1-tag4.png
                screendumps
                    0-tag3.txt
                    1-tag4.txt

Creating test scripts

Test scripts define subclasses of YkspTestCase based on the PyUnit framework. You will define and implement one or more methods with names starting with test, like such def testSomething(self):. To figure out what goes in the implementation, a good way to start is to look at testCapture.py and testAbandonLinking.py in examples/flying-photo-booth-tests/scripts.

From the examples, you will find that the YkspTestCase class provides the following convenience methods, documented here.

launchApp(package=None)
refreshScreen(sleep=1)
saveScreen(tag=None, sleep=1)

It is important that refreshScreen() or saveScreen() be called after each screen transition on your device, in order for the test to pick up the updated view tree.

To send UI events to your device, we rely on AndroidViewClient by @dtmilano. You have already downloaded the documentaion to here $YKSP_HOME/../AndroidViewClient/AndroidViewClient/doc/index.html as part of the submodule.

To pass or fail test cases, aside from visual inspection of screenshots, we mostly rely on exceptions raised by the findView...() methods from AndroidViewClient, as well as the family of assert...() methods available through the PyUnit framework.

When writing test scripts, while you can find certain views by text sometimes, you will encounter UI elements like an image button, which you need to identify by a unique ID in the view tree. This is when the dump tool in AndroidViewClient becomes handy. Just manually navigate to the screen on your connected device, then type that in Terminal and pick out the ID you need from the view tree.

$ dump
android.widget.FrameLayout  
   android.widget.TextView com.groundupworks.flyingphotobooth:id/title PHOTO 1 OF 2
   android.widget.ImageButton com.groundupworks.flyingphotobooth:id/switch_button 
   android.widget.ImageButton com.groundupworks.flyingphotobooth:id/preferences_button 
   android.widget.Button com.groundupworks.flyingphotobooth:id/start_button CAPTURE

Lastly, it is important to remember that while an app data wipe is performed before and after running each test script to remove the persistent data stored, that is not the case in between test methods within the same script. Instead, the app is killed between test methods, which means in-memory states are destroyed, but disk-persisted states are carried across. For each test to have a true 'fresh start', it is recommended that you separate out your tests into different files, with only a single test method in each YkspTestCase subclass.

After all that, here is a template to get you started!

#! /usr/bin/env python
import sys
import os

try:
    sys.path.append(os.environ['YKSP_HOME'])
except:
    pass

from yksptestcase import YkspTestCase

class MyTestCase(YkspTestCase):

    def testSomething(self):
        '''
        Tests something... like geese, mustard, cabbages and kings.
        '''
        self.launchApp()
        self.saveScreen('my-start-screen', sleep=1)

        # Your implementation...


if __name__ == '__main__':
    YkspTestCase.main(sys.argv)

Run options

yksp has run options to support various use cases. You can see those by entering the following in your Terminal.

$ yksp --help

Usage:
-h, --help               OPTIONAL    print this help and exit
-l, --linear             OPTIONAL    run tests on one device after another
-a, --noapk <package>    OPTIONAL    disable APK installation and run tests using the pre-installed application with the specified package name
-d, --nodump             OPTIONAL    disable app data dump after executing each test script

--linear

By default, yksp spawns multiple processes to execute test scripts on all connected devices in parallel. You may force yksp to run the scripts on one device after another using this flag. It is useful when debugging your test scripts, since the test logs in your Terminal will look cleaner.

Note that this does not imply everything is executed in a single process, as yksp uses multiple processes to do everything it needs to do even for a single device.

--noapk

This option allows launching off a pre-installed application, specified by its package name. No APK needs to be provided, and yksp will ignore any APK in your directory and skip installation (and uninstallation) altogether. A verification step is performed on each connected device to verify that the application is already installed, and the test scripts will be executed only on the subset of devices that are verified.

This option is useful for integrating yksp with your IDE for development. In Android Studio, you can have your Run Configurations deploy the APK but not launch any Activity, and instead launch an external tool with the following configurations, where com.groundupworks.flyingphotobooth is replaced by the package name of your app, and some-directory is any directory containing a scripts folder containing your test scripts.

Program:           yksp
Parameters:        --noapk com.groundupworks.flyingphotobooth
Working directory: some-directory

--nodump

After executing each test script, yksp utilizes the adb backup command to generate the backup file data.ab. The file itself can be used to restore the device state, as it contains all the locally persisted app data, including preferences and databases. yksp also extracts the file into the data folder, which you can conveniently use to validate the state changes as a result of the test script.

Note that if the application manifest has android:allowBackup="false", a data.ab file will be dumped out containing nothing. This is the correct behaviour and the absence of files in the extracted folder can be used to validate this.

The data dump is usually fast, but can optionally be disabled with this flag.

Mildly-faq

Q: Can my PyUnit test script mock out components in my application as part of the test?

A: The short answer is no. yksp tests an APK, or a pre-installed app, as is. It cannot inject classes into your application, or switch up any build-time configurations. It doesn't mock parts of your application; it mocks you. One way around this limitation is to build the mock components into your APK, make the build-time configurations into run-time configurations, then use UI toggles or a special launch Intent to invoke a code path utilizing the mock components.

Q: How can I pass or fail my tests based on 'user-generated' events that don't result in UI implications?

A: Again, yksp mocks out the user. Within the PyUnit test method, this mock user can generate events as input, and inspect the UI elements to validate the visible part of the output. However, input events can also result in output not immediately visible through the UI, such as a state change:

The first one is irrelevant, unless there is a way to confirm it through the UI. If a human is to validate the second and third cases, it would be via a database dump, check some server state via a server API call, or in some cases, inspection of the logcat output may be sufficient. Although it is possible to do all of the above within the test case itself and do assertion, using regular Python and adb commands, the recommended way is to use the PyUnit assertions to validate only the UI.

Since yksp dumps out the locally persisted app data and the logcat output after running each test script, just make use of those! Look at the results of the many test sessions as your data set, of which the PyUnit results is only one of the many components. You can write scripts to validate state changes, compare view trees and generate image similarity indices, correlate results across different devices and test sessions, and generate reports that are more comprehensive and meaningful.

Q: Why is the name weird?

A: Maybe because it consists entirely of consonants? Although the y is debatable.