Steps to reproduce are often unclear,
incomplete, or ambiguous.
0k+
open-source projects wrote GitHub a letter.
Their frustration: bug reports often arrive without steps to reproduce.
Low-quality steps lead to…
Bugs nobody can reproduceBugs that never get fixedHours of manual triage
Introducing
Euler
A quality check for steps to reproduce.
SCHEDULE #154 · STEPS TO REPRODUCE
I-S2R
Go into favorites
EMMS
I-S2R
Remove event in event details screen
VM
I-S2R
Hit BACK button to return
EM
Find every step.
A neural sequence labeler tags the sentences that describe steps. Dependency parsing breaks each one into parts.
GNUCASH #256 · CONDITIONAL SENTENCE
“When I create an entry for a purchase, the autocomplete list shows up”
ACTIONcreate
OBJECTentry
PREPfor
OBJECT 2purchase
Match each to the app.
A GUI ripper maps the app’s screens. Euler resolves every step to an event and a component, and infers the steps in between.
Launch
Main
Event details
Favorites
MISSING STEPS
tap image button
tap “Chaos Communication Camp Opening”
“Add favorites” → tap “item fav (Add to favorites)”
Flag what’s wrong.
Every step gets a quality annotation the reporter can act on.
EM
Exact match
Matches one app interaction
AS
Ambiguous
Matches several components or events
VM
Vocabulary mismatch
Matches no app interaction
MS
Missing steps
Required steps the report left out
Select event · This S2R matches multiple actions (e.g., “long click” or “click”).
Judged by external evaluators · 24 bug reports · six Android apps
0%
of existing steps
correctly identified
0%
of quality annotations
judged correct
ACM SIGSOFT Distinguished Paper Award
Assessing the Quality of the Steps to Reproduce in Bug Reports
Oscar Chaparro · Carlos Bernal-Cárdenas · Jing Lu · Kevin Moran · Andrian Marcus · Massimiliano Di Penta · Denys Poshyvanyk · Vincent Ng
ESEC/FSE 2019 · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering