Who tests the security analyzers?

Benchmarks go stale. Unsound shortcuts stay hidden.

Introducing
µSE
Mutation testing for security analyzers

Point it at an app.

One properties file. Bring your own source and sink APIs if you like.

configuration.properties
lib4ast: libs4ast/
appSrc: /tmp/AppFoo/src/
appName: AppFoo
output: /tmp/mutants/
operatorType: SCOPESINK
// optional: custom source & sink
source: var%d = api.method_value()

Choose where the leaks hide.

Four mutation schemes, each probing a different blind spot.

Reachability

A leak at every reachable spot — methods, inner classes, class fields.

Complex-reachability

Extra hops between source and sink, to outlast shallow analyses.

TaintSink

Source in one callback, sink in the next — onStart, then onResume.

ScopeSink

Data shared across visibility scopes, leaked in parent and child.

sensitive sourcepublic sinkextra hop

One command. Only useful mutants.

From the CLI or the GUI — with filters that drop mutants that would waste your time.

$ java -jar muse.jar configuration.properties
SYNTAX

Auto try-catch

APIs that throw are wrapped so mutants stay valid.

COMPILE

Compilability filter

The JDK compiler weeds out mutants that won’t build.

EXECUTE

Executability filter

Optional: an engine like CrashScope drops leaks that never run.

Then let the analyzers try.

Check each tool’s report against µSE’s log. Every missed leak points to a soundness issue.

In file: BMIMain.java · Mutation Scheme: TAINTSINK
mutation-3-0: BMIMain.onCreateFLAGGED
mutation-2-0: BMIMain.onCreateMISSED
mutation-1-0: BMIMain.onCreateFLAGGED
mutation-0-0: BMIMain.onCreateMISSED
Found with µSE in Android data-leak detectors
0
previously undocumented flaws
13in FlowDroid — and the tools built on it
12 morein Argus and HornDroid
Open sourceCLI, GUI, and documentation

Mutation-based Evaluation of Security-focused Static Analysis Tools for Android

Amit Seal Ami · Kaushal Kafle · Kevin Moran · Adwait Nadkarni · Denys Poshyvanyk
ICSE 2021 · IEEE/ACM International Conference on Software Engineering · Demonstrations