Automation learns from code.

And from natural language.

Rarely from the UI.

CODE

public void onCreate(Bundle b) {
  setContentView(R.layout.shop);
  grid = findViewById(R.id.grid);
  grid.setAdapter(new Tiles());
}

NATURAL LANGUAGE

Bug report: app freezes after I pick a category and open the cart.

Requirement: users can browse items by category.

USER INTERFACE

Every screen encodes
what software does.

Functionality, rendered as pixels — latent patterns that reveal an app’s use cases.

Now deep learning
can read it.

Advances in computer vision and NLP, plus large-scale datasets of UIs.

A position paper

Learn from the UI.

Patterns in interfaces → software engineering automation

CHANNEL 01Structural

CHANNEL 02Functional

What a UI is made of.

The hierarchy, its components and their programmatic types, plus colors, fonts, and images.

<node class="android.widget.ImageView"

bounds="[24,176][178,306]" clickable="true"

text="" content-desc="New in"/>

Fuels reverse engineering of UIs into code.

What a UI is for.

The use cases and affordances it supports, for a whole screen or a single element.

SCREEN

The screen lets the user browse shopping categories.

ELEMENT

1The top-left icon opens the menu.

ELEMENT

2The top-right icon opens the shopping cart.

Guides UI testing by the use cases a screen affords.

New frontiers

One screen. Many SE tasks.

01

Ground SE tasks in UIs

Bridge the lexical gap between code and language: traceability, fault localization, feature location.

03

UI-centric program synthesis

Map feature descriptions to screens, then to “program templates” for the logic behind them.

02

Usage-based UI testing

Screen understanding, aligned with real usage data, to generate tests that act like users.

04

Accessibility dev tools

Build accessibility in during design and development, not after.

The challenges · a call to action

Toward the next decade of
UI-centric developer tools.

01

Multimodal datasets of code, UIs, and language

02

Multimodal models built for UI understanding

03

Linking UIs to real usage traces and videos

04

Generalizing across desktop, web, and mobile

Position paper

Learning Patterns from User Interfaces to Support Software Engineering Tasks

Kevin Moran

CompUI@CHI 2022 · CHI 2022 Workshop on Computational Approaches for Understanding, Generating, and Adapting User Interfaces