Every change reruns the tests.

Order is everything.

In the default order, the fault hides until test 13.

Prioritize, and it surfaces at test 2.

Test case prioritization reorders a suite to catch regressions sooner.

fault found · test 13 of 16
fault found · test 2 of 16

Two ways to pick the order.

Reads the code

Static

  • Call graphs
  • String distance
  • Topic models
Runs the code

Dynamic

  • Greedy total & additional
  • Adaptive random
  • Search-based (genetic)

Static needs no execution. Dynamic needs coverage from past runs.

The first large-scale comparison
Static vs. Dynamic
100 versions × 5 mutants = 500 faults
TEST-CLASS LEVEL
TEST-METHOD LEVEL
% OF SUITE RUN → % FAULTS FOUND →
APFD
01Seed faults
02Prioritize
03Score

Plant the bugs.

Mutation testing builds 100 faulty versions of every program, five mutants each.

Reorder the suite.

All eight techniques rank the tests twice: whole test classes, then individual test methods.

Score the speed.

APFD is the area under the fault-detection curve. Find faults sooner, score higher.

Finding 1 · prioritizing test classes

Static edges ahead.

StaticCall-graph, additional
0.000
DynamicGreedy additional
0.000
Finding 2 · prioritizing test methods

Dynamic pulls away.

DynamicGreedy additional
0.000
StaticTopic model (Mallet)
0.000
Average APFD across 30 programs · best technique of each kind · bars on a 0–1 scale
Finding 3 · what they catch

They find different faults.

Static
Dynamic
≈25–30%

of detected faults shared by the top 10% of prioritized tests

The lesson

Don’t pick a side.

Combine them.

Static and dynamic information could be combined into more effective prioritization.

A Large-scale Empirical Comparison of Static and Dynamic Test Case Prioritization Techniques

Qi Luo · Kevin Moran · Denys Poshyvanyk
FSE 2016 · ACM SIGSOFT International Symposium on the Foundations of Software Engineering