Deep learning came to SE slowly.

Then all at once.

9→9
DL4SE papers per year, 2016 → 2017

Researchers needed a map.

What’s been done. What works. What can be improved.

A systematic literature review
DL4SE
A decade of deep learning in SE research · 2009–2019
0
papers
01Search
02Filter
03Dissect

Cast a wide net.

Ten years of the top SE, PL and AI venues, across four databases, with one deliberately broad query.

search string
"Deep" OR "Learning" OR "Neural"

Filter by hand.

Inclusion and exclusion criteria, snowballing, and author consensus on every call.

1,771
returned by the search string
1,217
pass the inclusion criteria
128
implement and evaluate DL for an SE task

Dissect every paper.

Each study is coded against the components of learning, one research question each.

RQ1
Target function
Which SE tasks?
RQ2
Training data
What data, prepared how?
RQ3
Algorithm & hypotheses
Which models, trained how?
RQ4
Final hypothesis
How well do they perform?
RQ5
All of the above
Can it be reproduced?
The review spans
0
papers
0
SE tasks
0
years of research
Finding · source code and RNNs dominate

What models learn from

Types of SE data used

What they’re built on

DL architectures used
As many as
half
of studies run no exploratory data analysis.

The step that guards against data snooping and bias.

Finding · reproducibility
Only 0
of 128 studies could be replicated.
63reproducible from the paper’s description
55missing data-filtering details
45missing hyperparameter details
The result

A roadmap for DL in SE.

Five steps, and what to report at each one.

1Explore the data
→
2Engineer features
→
3Pick an architecture
→
4Check learning principles
→
5Check generalizability

A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research

Cody Watson · Nathan Cooper · David N. Palacio · Kevin Moran · Denys Poshyvanyk
TOSEM 2022 · ACM Transactions on Software Engineering and Methodology