A screen shows what software does.
Can a model say it?
Doc tools translate code into words.
But code and English rarely share words.
The GUI could bridge the gap.
CODE
void onClick(View v) {
if (v.getId() == R.id.ic_cart)
startActivity(cartIntent);
}
DOCUMENTATION
“Opens the shopping cart.”
LEXICAL GAP
An empirical study — and a new dataset
Clarity
Comprehending visuaL semAntics to pRedict applIcation functionalTY
Screenshot → functional description
Describe every screen.
Crowd workers wrote one high-level caption and up to four low-level ones per screen. Authors vetted them all.
HIGH-LEVEL
“The screen allows the user to look at clothing categories.”
LOW-LEVEL
1The top left icon allows the user to access the menu
2The heart icon to the left of the shopping cart allows the user to view favorites
3The top right icon allows the user to access the shopping cart
4The center list of categories allows the user to make a selection
Measure how natural it is.
n-gram language models score how predictable the captions are, against other corpora.
1Java source coderawmost natural
2Clarity captionshigh- & low-level≈5 bits
LESS NATURAL THAN CLARITY
Java w/o syntax tokensStack OverflowGutenberg books
Cross-entropy · lower means more predictable
Teach models to caption.
Four encoder–decoder models in 15+ configurations. Three read pixels; one reads GUI metadata.
→
ENCODERCNN
→
DECODERLSTM · RNN
→
OUTPUTone word at a time
CAPTIONthe top right icon allows the user to access the shopping cart
im2txtNeuralTalk2Show, Attend & Tellseq2seq · GUI metadata
The Clarity dataset
0
composite BLEU · Show, Attend & Tell
Edging out NeuralTalk2 trained on Flickr8K (34.0), a photo dataset of similar size.
GUI pre-trainingim2txt’s best encoders were tuned on GUI images, not ImageNet
Pixels > metadataseq2seq on GUI metadata performed poorly across caption types
Humans ≠ BLEU220 raters favored im2txt, though NeuralTalk2 scored higher BLEU in two configs
Lesson learned
“GUI screenshots could be used to augment approaches for automated documentation.”
— Discussion & Learned Lessons, Lesson 1
An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
Kevin Moran · Ali Yachnes · George Purnell · Junayed Mahmud · Michele Tufano · Carlos Bernal-Cárdenas · Denys Poshyvanyk · Zach H-Doubler
SANER 2022 · IEEE International Conference on Software Analysis, Evolution and Reengineering