Testing bots explore apps on their own.

Until they get stuck.

9:41
Welcome back
Log in to continue
Email
Password
Log in
Forgot password?
Trailhead
SAME SCREEN0 s
TARPIT

Random taps
can’t log in.

Log-ins, forms, ads, onboarding: screens that need the right inputs in the right order.

They’re called UI tarpits.

The previous fix steers around them — and leaves everything behind them unexplored.

01Detect
02Recognize
03Navigate

Notice the stall.

AURORA watches the screen every second. Ten seconds without a change, and it steps in.

0
seconds on one screen

Read the screen.

A silhouette of the layout plus the on-screen text go to a multimodal CLIP model trained on 21 UI design motifs.

RANKED DESIGN MOTIFS
1Log-inTARPIT
2Form
3Onboarding

Then walk through it.

One of eight motif heuristics takes over; SentenceBERT matches each field to the right preset input.

heuristic · log-in
type field ≈ “Email”
type field ≈ “Password”
tap button ≈ “Log in” → new screen
Introducing
AURORA
Recognize the screen. Then navigate it.
+0%
more method coverage than VET, the tarpit-avoiding baseline
16 of 17apps reach higher coverage
88.8%of heuristic runs get past the screen
≈81%design-motif classification accuracy
Where the explorer stalled, across 17 popular apps

Stuck, then through.

HALTS
PASSED BY AURORA
Built with Dragon Testing · now in its product

Pop-ups and forms, navigated

for thousands of automated tests.

AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding

Safwat Ali Khan · Wenyu Wang · Yiran Ren · Bin Zhu · Jiangfan Shi · Alyssa McGowan · Wing Lam · Kevin Moran
ICST 2024 · IEEE International Conference on Software Testing