Add retries to fetch_user #482
The checks pass.
An agent wrote the change, opened the PR, and every test went green.
But were the new lines ever run?
A green build only says the tests passed — not that they touched the agent’s code.
Rebuild every PR.
Each patch is reconstructed from its base and head commits, test files split from code.
git diff base..headtest-only patch
Run the whole suite.
Line coverage with JaCoCo and pytest-cov marks each added line covered or missed.
Diff coverage =covered added linesall added executable lines
Pull the agent’s tests.
Reverse-apply its test changes and rerun: the difference is what the agent’s own tests add.
Finding 4 · lines inside try / catch blocks
Error handling is the blind spot.
An empirical study of agentic pull requests
How well-tested is
agent-written code?
4,882 PRs5 coding agentsJava + PythonAIDev dataset
Finding 1 · PRs that change code
Half ship without tests.
0%
include no test changes
Finding 2 · agent-changed lines run by the project’s existing tests
The safety net has holes.
64.8%of Python PRs: not one changed line
executed by any existing test
Finding 3 · PRs where the agent also wrote tests
Agent tests help — rarely.
Java
0%
of these PRs gain any coverage from the agent’s tests
+15.6 pp on average
Python
0%
of these PRs gain any coverage from the agent’s tests
+9.6 pp on average