The test passed.

The bug is still there.

REPRODUCTION TEST · replaying the report’s steps
✓tap “Expenses”
✓tap “Add”
✓type “12.50”
✓tap “Save”
PASSED4 steps · 0 assertions

Tools can replay a bug report’s steps.

Nothing checks for the failure itself. Writing that oracle is still done by hand.

Introducing
AndroB2O
Bug reports in. Test oracles out.
Bug report · observed behavior

“The total shows NaN after I add an expense.”

Failing screen · XML hierarchy
<FrameLayout> <TextView text="Expenses"/> <TextView desc="Total" text="NaN"/> <ListView> <TextView text="Groceries"/> <TextView text="12.50"/> </ListView> </FrameLayout>
01Find the element
02Write the assertion
03Verify

Find where it fails.

Given the report and the screen’s XML, an LLM picks the element the failure lives in. It runs three times, and must agree with itself.

RUN 1
desc="Total"
RUN 2
desc="Total"
RUN 3
desc="Total"
✓ Same element, every run

Assert the bug is gone.

It selects the element by its text or description, then writes UIAutomator assertions that the failure is absent, again agreed across runs and compiled.

failure-based oracle
UiObject2 total = device.findObject(
By.desc("Total"));
assertNotEquals("NaN", total.getText());
3 runs agreeUIAutomator API onlycompiles ✓

Fails on the bug.
Passes on the fix.

Buggy app
✕ FAILED

expected not "NaN"

Fixed app
✓ PASSED

total = "12.50"

On FBO4A, the first benchmark of its kind · 152 bug reports from 24 apps
0%
of reported failures get an oracle that catches them
2×the working oracles of the LLM baselines
10 of 16on reports the LLM likely never saw
19 of 32failures reproduced by ReBL, completed with oracles

Generating Failure-based Oracles to Support
Testing of Reported Bugs in Android Apps

Jack Johnson · Junayed Mahmud · Oscar Chaparro · Kevin Moran · Mattia Fazzini
ASE 2025 · IEEE/ACM International Conference on Automated Software Engineering