Your tests pass.

Do they catch bugs?

Plant a bug. See if tests notice.

Mutation testing seeds small faults — mutants — to measure a test suite.

NoteListActivity.java
Intent i = new Intent(this, Detail.class);Intent i = null;
startActivity(i);
MUTANT · NullIntent testOpenNote() FAILED → mutant killed

But Android bugs aren’t just Java bugs.

Java mutation tools have no Android operators. Research had proposed just eight.

Intents Manifest permissions findViewById Activity lifecycle GPS location DB cursors String resources

So we started from real bugs.

0
software artifacts, analyzed by hand

Bug reports · bug-fixing commits · Stack Overflow · exception docs · prior studies

0 fault types. 0 categories.

More than a third of the bugs — 35% — are specific to Android.

Introducing
MDroid+
38 operators · born from real Android bugs
NoteListActivity.java
Button save = findViewById(R.id.save);
save.setOnClickListener(v -> {
Intent i = new Intent(this, Detail.class); Intent i = null;
i.putExtra("noteId", note.id);
startActivity(i);
});
AndroidManifest.xml
<uses-permission …CAMERA"/>
<activity android:name=".Detail"/>
res/values/strings.xml
<string name="save">Save</string>
01Profile the app
02Seed a fault
03Build mutants

Find where bugs live.

Static analysis of code and XML builds a Potential Failure Profile: every spot an operator applies.

FindViewByIdReturnsNullJava · AST
NullIntentJava · AST
InvalidKeyIntentPutExtraJava · AST
MissingPermissionManifestManifest · XML
WrongStringResourceResources · XML

One fault per location.

Each location gets its operator’s transformation, in its own clone of the app.

NullIntentNoteListActivity.java
- Intent i = new Intent(this, Detail.class);
+ Intent i = null;
MissingPermissionManifestAndroidManifest.xml
- <uses-permission android:name="…CAMERA"/>

One APK per mutant.

Clones are compiled into individual Android packages, ready for your tests.

Real bugs from 726 held-out artifacts · 7 mutation tools compared

Bug types covered

0%
0%

Bug instances covered

0%
0%
Mutants generated for 55 open-source apps
0.00%
stillborn mutants, on average
vs 1.8% for Major · 53.9% for muDroid
0.00%
trivial mutants — fewest of four tools
vs 5.4% Major · 7.2% PIT · 11.8% muDroid

Enabling Mutation Testing for Android Apps

Mario Linares-Vásquez · Gabriele Bavota · Michele Tufano · Kevin Moran · Massimiliano Di Penta · Christopher Vendome · Carlos Bernal-Cárdenas · Denys Poshyvanyk
ESEC/FSE 2017
Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering