A bug gets reported.

Who should fix it?

Mozilla Core alone: 0 developers.

And 105 components a report could belong to.

Triage by hand takes months.

On Mozilla, assigning a bug to a developer takes about 180 days.

A comparative study

Can transformers triage bugs?

ALBERTBERTCodeBERT DeBERTaDistilBERTRoBERTa vs. TF-IDF + SVM
01Fine-tune
02Test on the future
03Rank who & where

Read the whole report.

Six pre-trained transformers are fine-tuned to classify a report from its summary and description.

[CLS]editorfreezeswhenrenaming…
→
TRANSFORMER LAYERS · [CLS] EMBEDDING
SOFTMAX

Tested on the future.

Reports are sorted by time into 11 folds: train on folds 1…n, test on fold n + 1. Ten runs, averaged.

■ TRAINRUN 1 / 10OLDEST → NEWEST

Rank who. Name where.

Each model ranks developers and picks the component.

DEVELOPERS · RANKED
TOP@1 · TOP@5 · TOP@10 · MRR
COMPONENT
UI
PRECISION · RECALL · F1
Four open-source products · Eclipse JDT & Platform, Mozilla Core & SeaMonkey
0
bug reports
0
years of history
0
developers
0
components
Finding 1

DeBERTa is the strongest transformer.

Significantly better than the other five, on developers and components alike.

DeBERTaOther six modelsComponent-assignment F1 · higher is better
Finding 2

But a tuned TF-IDF + SVM still wins.

Best developer assignment on both Mozilla datasets, by a statistically significant margin.

Mozilla Core · MRR
TF-IDF + SVM
0
DeBERTa
0
Mozilla SeaMonkey · MRR
TF-IDF + SVM
0
DeBERTa
0
Finding 3

Every model gets bugs right that the others miss.

SVM is the most orthogonal; DeBERTa, among transformers. A case for ensembles.

0developer assignments
only ALBERT gets right
Finding 4

Look-alike components fool every model.

In Eclipse JDT, all seven models misassign the same 1,184 reports.

UI94.92% similarCore
0UI ↔ Core mix-ups,
summed over all seven models

A Comparative Study of Transformer-based Neural Text Representation Techniques on Bug Triaging

Atish Kumar Dipongkor · Kevin Moran
ASE 2023 · IEEE/ACM International Conference on Automated Software Engineering