Users don’t write bug reports.
They hit record.
Screen recordings are everywhere.
Bug reports. Feature requests. Crowdsourced feedback.
But a video is just pixels.
Developers re-watch it, then reproduce every tap by hand.
Introducing
V2S
Video → Replayable Scenario
REC
REPLAYING
Find every touch.
Deep learning spots the on-screen touch indicator in every frame of the recording.
Faster R-CNN
Locates each touch, frame by frame.
Opacity CNN
Tells a finger pressing from one lifting.
Classify every action.
Touches across consecutive frames become taps, long taps, and gestures.
Replay it. Automatically.
V2S writes a script that re-runs the whole scenario on a real device.
scenario · replay script
tap (184, 168)
long_tap (184, 300) 0.6 s
gesture (184, 620) → (184, 300)
tap (300, 722)
Evaluated on
≈0%
of user actions faithfully replayed
≈97%touch-detection precision
≈98%action-classification recall
96.7%of bug & crash videos fully reproduced
Case study · three industrial partners
“This solution could be integrated in the continuous integration pipeline.”
— Industry participant, on V2S
ACM SIGSOFT Distinguished Paper Award
Translating Video Recordings of Mobile App Usages into Replayable Scenarios
Carlos Bernal-Cárdenas · Nathan Cooper · Kevin Moran · Oscar Chaparro · Andrian Marcus · Denys Poshyvanyk
ICSE 2020 · IEEE/ACM International Conference on Software Engineering