Every release, the design moves on.

The code has to follow.

CURRENT RELEASEscreenshot + UI hierarchy
EVOLVED RELEASEupdated metadata, implemented
NEW MOCK-UPan image, nothing more

Developers spot every change by eye.

Then implement each one in the GUI code, by hand.

And a mock-up has no code behind it.

No UI hierarchy. No metadata. Just pixels.

Introducing
GuiEvo
Old release + new mock-up → evolved GUI
01Match components
02Classify changes
03Generate metadata

Find what survived.

Each component is cropped from the old screen and searched for on the mock-up.

Recurring

Found again with edge maps, SSIM, and multi-scale template matching.

Removed

No region of the mock-up matches it closely enough.

Added

Regions of the mock-up that don’t exist on the old screen.

Name every change.

Ten change types, across text, layout, and resources.

TEXT

TCcontent
TXCcolor

LAYOUT

VTmoved ↕
HTmoved ↔
VSheight
HSwidth

RESOURCE

CCcolor
CAadded
CRremoved
ICimage

Evolve the hierarchy.

GuiEvo updates the old UI hierarchy into metadata for the mock-up, ready to implement by hand or with GPT-4.

mock-up · updated uiautomator xml
~ <node class="TextView" text="My list"/> TC
~ <node class="ImageView"/> IC
+ <node class="LinearLayout"/> CA
~ <node bounds="[20,412][348,482]"/> VT
− <node bounds="[20,520][348,590]"/> CR
~ <node class="Button" bounds="[20,690][348,754]"/>
Evaluated on real Android apps
0
screen + metadata pairs captured
0
apps examined by hand
0
screen pairs with real GUI changes
0%
app-wise accuracy identifying real GUI changes
88%app-wise F1-score
5.9avg. tree edit distance to the true hierarchy
0.99avg. SSIM, recompiled screen vs. mock-up

Closer to the mock-up.

Tree edit distance to the mock-up’s UI hierarchy, five recompiled apps. Lower is better.

GuiEvoimplemented by hand
0
GuiEvoimplemented by GPT-4
0
REMAUI
0
ReDraw
0

GuiEvo: Automated Evolution of Mobile App UIs

Sabiha Salma · SM Hasan Mansur · Yule Zhang · Kevin Moran
MSR 2024 · International Conference on Mining Software Repositories