Hand it a screen recording.

Get back a replay.

bug.mp4Screen recording · 30 fps
Introducing the tool
V2S
Open source. Built to be extended.

Find every touch.

Frames are normalized to 30 fps. Faster R-CNN finds each touch; an Opacity CNN reads press vs. lift.

Turn touches into actions.

Grouped across frames, split apart when they overlap, labeled Tap, Long Tap, or Gesture.

Write the script. Replay it.

sendevent commands, 33 ms per frame, run on the device by a modified RERAN binary.

Swap any part.

Every phase and component can be extended or substituted, say, with your own detection model.

CONFIG.JSON
{
"video": "bug.mp4",
"detector": "frcnn",
"opacity": "cnn",
"device": "Nexus 5"
}
PHASE 1

Touch detection

FrameExtractor
TouchDetectorFRCNN
OpacityDetector
YourDetector
PHASE 2

Action classification

GUIActionClassifier
group · segment · label
PHASE 3

Scenario generation

Action2EventConverter
Translator
REC
detection_full.json
detected_actions.json
send_events.log
0
videos
0
GUI actions
≈0%
of sequential actions reproduced

All three industry developers interviewed saw it as a potentially useful tool for testing and debugging.

Formal Tool Demonstration

V2S: A Tool for Translating Video Recordings of Mobile App Usages into Replayable Scenarios

Madeleine Havranek · Carlos Bernal-Cárdenas · Nathan Cooper · Oscar Chaparro · Denys Poshyvanyk · Kevin Moran
ICSE 2021 · IEEE/ACM International Conference on Software Engineering, Tool Demonstrations