Tests don’t all cost the same.
So count the seconds.
The classic score, APFD, counts tests.
But real tests run for very different times.
APFDc weighs every test by its run time.
TESTS RUN →
SECONDS ELAPSED →
High-coverage tests, which dynamic techniques favor, tend to cost more time.
An extensive study on GitHub projects
Static vs. dynamic,
on the clock.
58 Java programs714 KLoC4 static · 4 dynamicAPFD + APFDcclass & method level
Then, try to break the verdict
Four things that might change it.
Program size
Bigger programs?
29 smaller vs. 29 larger systems.
Software evolution
Older versions?
Prioritize on an earlier version. 66 versions of 12 programs.
Fault quantity
More faults?
1 to 10 mutants per faulty version.
Fault type
Different faults?
Mutants from one operator at a time.
Test-method level · average over 58 programs
Count tests: dynamic leads.
Count seconds: static leads.
StaticDynamicAPFD
Under APFDc, no technique is significantly better.
Size · evolution · fault count · fault type
None of them flips the ranking.
The lesson
One metric can mislead.
Score prioritization by time, too.
APFD and APFDc can disagree, so new techniques should be judged on both.
How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects
Qi Luo · Kevin Moran · Lingming Zhang · Denys Poshyvanyk
TSE 2018 · IEEE Transactions on Software Engineering