Real bugs are hard to collect.
So research plants fake ones.
A mutant: one tiny, automatic change.
Inventory.java
if (index <<= size) {
return items[index];
}
CONDITIONALS BOUNDARY
Prioritization is scored on thousands.
The assumption: rank well on mutants, rank well on real bugs.
But are they honest stand-ins?
The first study of its kind
Mutants vs. real faults
8 TCP techniques357 real faults5 Java systems · Defects4J35k+ mutants
REAL FAULTClosure · bug 2
+
MUTANTrandom, killable
REAL FAULTLang · bug 37
+
MUTANTrandom, killable
REAL FAULT… all 357
+
MUTANTrandom, killable
× 100 random draws
Start from real bugs.
357 Defects4J faults, each with a buggy version, its fix, and a test that reveals it.
Add matched mutants.
One random mutant per faulty version, drawn 100 times — with and without trivial and subsumed mutants.
Rank both ways.
Score all eight techniques on each fault set, then ask: do the leaderboards agree?
Finding 1 · absolute scores
Mutants flatter.
≈0%
average overestimate of APFD(c) on the full mutant set
≈3%underestimate once trivial and subsumed mutants are removed
Finding 2 · average APFD, ranked
The leaderboard flips.
Real faults
Mutants
Finding 3 · do mutant rankings predict real ones? (Kendall τb)
It depends on the yardstick.
APFD · very weak
−1
0
+1
0.000
APFDc · medium to strong
−1
0
+1
0.000
What it means for testing research
The best technique on mutants may not be the best on real faults.
Agreement varies by program and mutation operator — so operators should be tailored to the domain.
Nominated for IEEE TCSE Distinguished Paper Award
Assessing Test Case Prioritization on Real Faults and Mutants
Qi Luo · Kevin Moran · Denys Poshyvanyk · Massimiliano Di Penta
ICSME 2018 · IEEE International Conference on Software Maintenance and Evolution