Most automated app tests

only notice crashes.

↻ rotate
9:41
New reminder
Title
Add a titleDentist at 3 pm
Notes
9:41
New reminder
Title
Add a title
AUTOMATED TEST · implicit oracle
type “Dentist at 3 pm”
rotate device
no crash · no exception
PASS

Only about a third of failures crash.

Tests that wait for one are, in effect, blind to the rest — like the text that just vanished.

Step one: a taxonomy of how apps fail

What should an app always do?

0
reproduced, non-crashing failures · 43 apps
→
0
app-independent expected behaviors
Introducing
Magneto
Automated GUI oracles for mobile apps.

Watch every action. Check what should hold.

01 · APP EXECUTION

Any test generator drives the app

For each action: what was touched, where, plus screenshots and UI metadata before and after.

02 · TRIGGER DETECTION

Spot actions with an expected result

Rotating, pressing back, saving, picking a language or a theme.

03 · ORACLE EXECUTION

Check it with vision + language

SSIMDelta E colorOCRlanguage detectionCNN keyboard check

Five oracles, for any app.

O1
↻
When you
Rotate the screen
Then
Your input, text and images are still there.
UI metadata
O2
✓
When you
Tap Save, Done, OK
Then
What you entered is displayed.
UI metadata
O3
←
When you
Press Back
Then
The previous screen returns.
SSIM + OCR
O4
文A
When you
Pick a language
Then
Every later screen uses it.
OCR + language detection
O5
◐
When you
Switch the theme
Then
It applies everywhere, text intact.
Delta E + OCR
Exposed to 15 real failures, new to the taxonomy
0 of 15
failures revealed, with a single false positive
74.8%of 310 relevant new bug reports matched its expected behaviors
2 bugscaught in APE’s 7,382 generated actions, 0.78% average false positives
0 of 15failures revealed by OwlEyes, a UI display-issue detector

Automating GUI-based Test Oracles
for Mobile Apps

Kesina Baral · Jack Johnson · Junayed Mahmud · Sabiha Salma · Mattia Fazzini · Julia Rubin · Jeff Offutt · Kevin Moran
MSR 2024 · International Conference on Mining Software Repositories