Students have a new study partner.
Does it know testing?
CHATGPT · RESEARCH PREVIEW
TEXTBOOK QUESTION
Suppose that coverage criterion C1 subsumes coverage criterion C2. Further, suppose that test set T1 satisfies C1 on program P, and test set T2 satisfies C2, also on P. Does T1 necessarily satisfy C2? Explain.Ammann & Offutt, Introduction to Software Testing, p. 114
T1 may or may not satisfy C2. The fact that T1 satisfies C1, which subsumes C2, means that C1 is a more comprehensive criterion that includes all of the requirements of C2. However, it does not guarantee that T1 will also satisfy C2. …
ANSWERINCORRECT
EXPLANATIONPARTIALLY CORRECT
T1 does satisfy C2 — that is what subsumption means.
Fluent. Plausible. Wrong.
So how often is it right?
An empirical study
ChatGPT takes
the testing course.
31 textbook questions5 chapters3 runs eachanswers & explanations graded
The method
Ask. Repeat. Grade.
Two ways to ask
Sub-questions in one shared chat, or each in a fresh one.
Three runs each
Every question asked three times, to see how answers drift.
CorrectPartialIncorrect
answer · explanation · confidence
Graded twice
Two authors label every answer and explanation; a third settles disagreements.
0%
of answers correct or partially correct
53.0%explanations correct or partially correct
49.4% vs 34.6%fully correct · shared vs. separate chats
After each answer: “How confident are you that your previous response is correct?”
Confidence says little.
CORRECT OR PARTIALINCORRECT
When it was “highly confident,” its answers were incorrect about half the time.
0
of answer–explanation pairs disagree on correctness
0
of questions flip between right and wrong across runs
0
of textbook questions answerable in text at all
Promises & perils
Promise
A guide for lab exercises.
With carefully designed activities and added question context, it could walk students through practice problems.
Peril
A poor judge of its own correctness.
Its confidence has little bearing on whether it is right, and assessments may need to detect its use.
ChatGPT and Software Testing Education: Promises & Perils
Sajed Jalil · Suzzana Rafi · Thomas D. LaToza · Kevin Moran · Wing Lam
TestEd 2023 · Software Testing Education Workshop, co-located with ICST