Apps ship continuously.

Testing is still mostly manual.

25 Android releases.

4.1 · 4.2 · 4.3 · 4.4 · 5.0 · 5.1 · 6

Seven of them hold about 97% of the market, across a multitude of hardware.

Seven open challenges.

The deterrents to a comprehensive, automated testing solution.

AFragmentation BTest flakiness CMobile-specific fault models DHistory-aware test cases EEvolving scripts & models FMobile-specific oracles GMulti-goal testing
A new perspective on mobile testing
CEL
Continuous · Evolutionary · Large-scale
CContinuous
EEvolutionary
LLarge-scale

Every change
runs the tests.

A change to the code, usage, APIs or devices triggers a test iteration over just its impact set.

Tests evolve
with the app.

Models, test cases and oracles adapt as the app, its usage and its devices change.

Test like
it’s the wild.

Parallel runs on physical and virtual devices, in the cloud or on premise.

CHANGES
MONITORING
Source codedevelopers’ changes
On-device usagesusers
API evolutionplatform & libraries
Marketsuser reviews + ratings
commit
TESTING ARTIFACTS
GENERATION
Multi-model
GUIUsageDomainContextFault
Test cases + oracles
tap(“Add”)
type(“Amount”, …)
assert(expected)
LARGE EXECUTION
ENGINE
Test cases runnerimpact set only
CLOUDON PREMISEOPEN SOURCE
Reportsexpressive, reproducible
A research agenda in six topics
01Multi-model representations of apps
02Goal-oriented test generation
03Open, large-scale & crowdsourced testing
04Scalable, precise automated oracles
05Mining repositories & reviews to drive testing
06Useful feedback for developers

Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing

Mario Linares-Vásquez · Kevin Moran · Denys Poshyvanyk
ICSME 2017 · IEEE International Conference on Software Maintenance and Evolution