Designers draw every screen.

Developers redraw it.

activity_signup.xml · written by hand
<LinearLayout
android:layout_width="match_parent"
android:orientation="vertical">
<ImageView android:id="@+id/hero"
android:layout_height="200dp" />
<TextView android:text="Create your

A mock-up is just pixels.

Every component, layout and style has to be rebuilt in code.

“If you are trying to create a new activity with all its components, this can take hours.”

— Front-end Android developer, Google

Introducing
ReDraw
Mock-up → Prototype app
MOCK-UP
PROTOTYPE APP
01Detect
02Classify
03Assemble

Find every component.

Computer vision — or the mock-up’s own metadata — boxes each atomic element of the design.

edges → dilate
→ contours

From pixels

Canny edges, merged and traced into bounding boxes.

{ x: 20, y: 556,
  w: 328, h: 58 }

From Sketch

Exported mock-up metadata parsed for exact boxes.

Name every component.

A CNN labels each crop with its Android type — trained on GUIs mined automatically from Google Play.

0
apps explored
→
0
unique screens
→
0
labeled components
Screenshots + runtime GUI metadata = training data for 15 component types, no hand-labeling.

Assemble the app.

A KNN finds look-alike screens from real apps and borrows their containers. Then ReDraw writes the code.

res/layout · generated
<LinearLayout android:orientation="vertical">
<Button android:text="Sign up with Google"
style="@style/Style65"/>
<Button android:text="Sign up with Facebook"
style="@style/Style66"/>
</LinearLayout>
Classifying GUI-components
0%
top-1 precision for ReDraw’s CNN
ReDraw CNN
0%
BOVW baselinefeature descriptor + SVM
0%
Generated apps · versus REMAUI & pix2code

Closer to how real apps are built.

83 appsprototyped from their screens for the hierarchy and visual studies
< 10 editsin general, to exactly match the real GUI-hierarchy
Beats pix2codeon visual similarity to the target, significantly
Interviews · Google · Huawei · Facebook

“A developer could generate the initial view, clean up the layouts, and have a working app.”

— Front-end Android developer, Google, on ReDraw

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Kevin Moran · Carlos Bernal-Cárdenas · Michael Curcio · Richard Bonett · Denys Poshyvanyk
TSE 2018 · IEEE Transactions on Software Engineering · Journal-First at ESEC/FSE 2018