Benchmarks need bugs

that actually reproduce.

Prior datasets:
15 to 51 reports.

And the largest of them focused exclusively on crashes.

15
FUSION
24
EULER
48
YAKUSU
51
ReCDroid
Introducing
AndroR2
Bug reports, reproduced by hand. Every one.
From every Android bug on GitHub to the ones that truly reproduce

Around six person-months of manual reproduction.

Inside every entry · bug report #160

Three steps written. Fourteen actions scripted.

01 · Original report

Problem with the eraser tab

1. Click on whiteboard tab while reviewing flashcards and use the feature to write/draw anything

2. Click on the erase tab till you have undone the drawing…

3. Redraw anything by clicking on the whiteboard icon

02 · Buggy app
.apk

AnkiDroid 2.10beta3

The exact version, plus the Android configuration to run it.

03 · Reproduction script
@Test reproduce() {
find(By.text("Default"))
.click();
find(By.desc("More options"))
.click();
find(By.text(
"Enable whiteboard"))
.click();
mDevice.drag(300,300, …);

UIAutomator, excerpt

04 · Metadata
{ "id": 160, "failure_type": "gui", "s2rs": 3, "gui_actions": 14, "setup_gui_actions": 8, "rs_gui_actions": 6, "multiple_gui_action_s2rs": 1, … }
90 reproducible reports · 90 executable scripts
0
crashes
0
output errors
0
GUI errors
77need setup actions before the first written step
57have a step that takes several GUI actions

AndroR2: A Dataset of Manually-Reproduced
Bug Reports for Android Apps

Tyler Wendland · Jingyang Sun · Junayed Mahmud · S.M. Hasan Mansur · Steven Huang · Kevin Moran · Julia Rubin · Mattia Fazzini
MSR 2021 · International Conference on Mining Software Repositories · Data Showcase