data-leak analysis · notes.apk
$ analyze notes.apk --detect leaks
✓ No private data leaks found.

No leaks found.

Should you believe it?

Static analyzers cut corners.

They trade soundness for precision — they’re “soundy.” Which corners they cut is rarely written down.

And the blind spots spread.

FlowDroid alone has been extended by at least 13 research tools.

FlowDroid

So plant leaks — on purpose — and see what slips through.

Introducing
µSE
Mutation-based Soundness Evaluation
NotesActivity.java · mutated
public class NotesActivity extends Activity {
void onStart() {
String leak0 = Calendar…getTimeZone()…;
}
void onResume() {
Log.d("leak-0", leak0);
}
button.setOnClickListener(v -> {
Log.d("leak-1", Calendar…getTimeZone()…);
});
}
public class NotesFragment extends Fragment {
View onCreateView(…) {
Log.d("leak-2", Calendar…getTimeZone()…);
}
}
01Specify
02Seed
03Filter
04Analyze

Describe the threat once.

A security operator captures what a class of tools claims to catch — here, a data leak.

SECURITY OPERATOR · DATA LEAK
String dataLeak## = Calendar.getInstance()
.getTimeZone().getDisplayName();
android.util.Log.d("leak-##", dataLeak##);

Hide it everywhere.

Mutation schemes seed leaks wherever Android makes analysis hard.

Activity & Fragment lifecycles Async callbacks Intents & receivers XML resources Every method (reachability) Source and sink split apart Deliberately complex paths

Keep only leaks that run.

A dynamic execution engine drives each mutated app and discards leaks that never execute.

0
leaks injected into 7 apps
0
confirmed executable · over 73% filtered

Every miss is a lead.

Run the tool. Each leak it misses is traced and reduced to a minimal app that proves the flaw.

Missed leak→ Call chain→ Minimal app→ Confirmed flaw
Executable leaks each tool missed · out of 2,026
0%
0%
0%
0
undocumented flaws found in FlowDroid
All 13also in IccTA & DidFail, which build on it
4classes of flaws, from missing callbacks to async flows
< 1 hourof analyst time per flaw, worst case
One flaw, found and fixed

FlowDroid was blind to Fragments.

Fragments are in over 90% of top Google Play apps. Our patch was accepted upstream.

Discovering Flaws in Security-Focused Static Analysis Tools for Android using Systematic Mutation

Richard Bonett · Kaushal Kafle · Kevin Moran · Adwait Nadkarni · Denys Poshyvanyk
USENIX Security 2018 · 27th USENIX Security Symposium