Documenting code is tedious.

So models translate it.

Code in. English out.

Neural models “translate” a method into a natural-language summary.

Java method
public void setReadTimeout(int timeout) {
if (0 > timeout) {
this.readTimeout = timeout;
}
}
→
NeuralCodeSum · generated summary
/** sets the number of milliseconds to wait for a response before timing out */

Same meaning. Few shared words.

Human
Model

Word-overlap metrics like BLEU won’t score this highly, and can’t say what a model gets wrong.

A comparative study

What do these models
actually get wrong?

CodeBERTNeuralCodeSumcode2seq
Java method
public void setReadTimeout(int timeout) {
if (0 > timeout) {
this.readTimeout = timeout;
} }
Human-written
sets the timeout value in milliseconds for reading from the input stream
CodeBERTFocusing only on method name
sets the read timeout
NeuralCodeSumConsistent with specific info
sets the number of milliseconds to wait for a response before timing out
code2seqConsistent but missing specific info
sets the timeout to read from the server
01Generate summaries
02Score with metrics
03Label every error

Three models.

Each is trained on 400,000 Java methods from Funcom, split by project.

PRE-TRAINED

CodeBERT

Bimodal transformer, 125M parameters.

SEQUENCE

NeuralCodeSum

Transformer with a copy mechanism.

STRUCTURE

code2seq

Encodes paths through the AST.

Score them the usual way.

Each summary is compared with the human-written comment.

SMOOTHED

BLEU-4

Precision of matching n-grams.

+ SYNONYMS

METEOR

Unigram matches, stems and synonyms.

SEQUENCE

ROUGE-L

Longest common subsequence.

Then read every one.

Annotators open-coded 444 predictions (148 methods × 3 models) into a taxonomy of errors.

Consistent with ground truth 162 Missing information 148 Incorrect construction 110 Incorrect semantic info 52 Over-generalization 49 Extraneous info 9 Missing context 5
Smoothed BLEU-4 · Funcom test set

CodeBERT comes out on top.

CodeBERT
0
NeuralCodeSum
0
code2seq
0
30.34METEOR, best of three
35.65ROUGE-L, best of three
95% CIsignificant lead over NeuralCodeSum
0%
of labels: Missing Information, the most common error
≈30%consistent with the ground truth
≈21%incorrect construction
≈13%salient identifiers missed or misused

Each model fails its own way.

CodeBERT

Leans on the method name.

setReadTimeout(…)
sets the read timeout
≈10% of its labels: focusing only on method name
NeuralCodeSum

Can drift off topic.

numRulesTipText()
returns the number of labels in the rule set
11 summaries unrelated to the code. CodeBERT: 0
code2seq

Repeats itself.

getSchema()
returns the schema schema
≈19% of its labels: repetition, its most common
The takeaway

Measure meaning,
not just words.

The study calls for metrics built for code, and models that stop leaving information out.

Code to Comment Translation: A Comparative Study on Model Effectiveness & Errors

Junayed Mahmud · Fahim Faisal · Raihan Islam Arnob · Antonios Anastasopoulos · Kevin Moran
NLP4Prog 2021 · Workshop on Natural Language Processing for Programming (ACL-IJCNLP 2021)