A test runs your code.
The assert judges it.
Test generators can write the test.
But their asserts are often too weak to catch the bug.
NamedValueTest.java · generated
@Test
void testName() {
NamedValue namedValue = new NamedValue();
namedValue.setName(NamedValueTest.NAME);
assertNotNull(namedValue);
}
PASSES · CHECKS ALMOST NOTHING
Introducing
ATLAS
Learning to write the assert a developer would.
AuTomatic Learning of Assert Statements
TEST METHOD + FOCAL METHOD
void testName() {
NamedValue namedValue = new NamedValue();
namedValue.setName(NamedValueTest.NAME);
<AssertPlaceHolder>
}
// focal method
getName() { return name; }
void METHOD_0() {
IDENT_2 IDENT_3 = new METHOD_1();
IDENT_3.setName(IDENT_4.IDENT_5);
<AssertPlaceHolder>
}
// focal method
getName() { return name; }
Learn from developers.
9k+ GitHub projects that use JUnit 4, mined for the asserts developers actually write.
Find the focal method.
The method under test — the last call before the assert — is appended to the test as context.
Abstract the code.
Rare identifiers become typed tokens. The 1,000 most common idioms stay as they are.
Translate.
An RNN encoder–decoder with attention writes the assert token by token, then maps it back to real names.
Encoder
bi-LSTM
Decoder
LSTM
ATTENTION
0%
of asserts predicted exactly as the developer wrote them · top-1
49.69%exact match within the top 5
0.14 sfor five suggestions, one consumer GPU
Exact matches at top-1, vs. always guessing the most common asserts
10.9× more than the baseline.
Where it fits
A partner for test generators.
Code completion for developers.
Generated tests get meaningful asserts. Developers get suggestions as they type.
On Learning Meaningful Assert Statements for Unit Test Cases
Cody Watson · Michele Tufano · Kevin Moran · Gabriele Bavota · Denys Poshyvanyk
ICSE 2020 · IEEE/ACM International Conference on Software Engineering