Many apps need two fingers.

Pinch. Zoom. Rotate.

Two fingers. One gesture.

To replay it from video, you have to untangle which touch belongs to which finger, frame by frame.

Then there are hybrid apps.

Native screens mixed with web content, and few GUI testing tools that support them.

Introducing
V2S+
Every finger, replayed from video.

Pure computer vision. Native & hybrid apps.

VIDEO
REPLAYING
01Track each finger
02Count the fingers
03Replay them together

Untangle every finger.

Touches link frame to frame by distance. When that’s ambiguous, a fading touch marks a finger lifting.

Count the fingers.

Actions that overlap in time merge into one multi-finger gesture, from 1 to 10 fingers.

FRAMES →

Replay them together.

Each finger gets its own input slot in a sendevent script, replayed on the device.

two-finger gesture · sendevent script
[0.000] 0003 0039 00000000 // fing1 action ID
[0.000] 0003 0035 00000188 // fing1 x-coord
[0.000] 0003 002f 00000001 // switch to fing2
[0.000] 0003 0035 00000337 // fing2 x-coord
[0.033] 0003 002f 00000000 // switch fing1
Evaluated on
0
videos
0
GUI actions
0
native & hybrid apps
0%
of consecutive actions replayed in multi-finger scenarios
≈97.9%multi-finger precision & recall
≈90.2%of sequential actions matched · native apps
≈83%of sequential actions matched · hybrid apps

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios for Native & Hybrid Apps

Carlos Bernal Cárdenas · Nathan Cooper · Madeleine Havranek · Kevin Moran · Oscar Chaparro · Denys Poshyvanyk · Andrian Marcus
TSE 2022 · IEEE Transactions on Software Engineering