A survey of open-source Android contributors
RQ1How they design tests
RQ2What generated tests should be
RQ3Which tools they use
RQ4Whether coverage matters
35 of 102 do more than 90% of their testing by hand.
62 of 102 have no experience with Monkey or random testers.
82 of 102 chose expected outputs or steps grouped by use case.
65 of 102 don’t use coverage, or don’t find it useful.
Finding 01
Testing by hand
still wins.
0%
average share of testing effort spent on manual testing
Finding 02
Random testing
barely registers.
≈0%
have no experience with Monkey or random testing tools
Only 3 of 102 even listed Monkey among the tools they use.
Finding 03
Tests should read
like use cases.
≈0%
want expected outputs, or steps grouped by use case or feature
Natural language was the favorite format. Raw ADB commands, the least.
Finding 04
Coverage isn’t quality.
≈0%
don’t use code coverage, or don’t find it a useful measure
Many prefer fault detection or feature coverage instead.