Code models write whole programs.

Then miscount a parenthesis.

#print the string 742
print(str(str(str(742))))
✕ Needs four. The model closed three.

Four closing parens? Every GPT-2 model scores 0%.

So what goes wrong inside?

Seven language models, 124M to 7B parameters, opened up component by component.

INSIDE THE MODEL · ATTENTION HEADS & FF NEURONS
TOP-10 × α · α ∈ [1.1, 2.0]

Every component casts its own vote.

Heads and neurons each push the logits toward a closing token, read out with the logit lens.

A few are sound. Most are noise.

Sound mechanisms generalize across inputs. Faulty ones promote the wrong tokens too.

Failure by interference.

Errors happen when faulty mechanisms overshadow the sound ones.

One head beats the whole model.

CodeLlama-7b’s head L30H0 is almost 100% accurate on every sub-task. The full model averages 96.00%.

01Rank
02Steer

Rank by reliability.

First by how many sub-tasks a component generalizes to, then by how well it promotes the right token.

Turn up the sound.

Scale the top-k sound components’ activations by α before they enter the residual stream.

SUMMED LOGITS · NEXT TOKEN
)
))
)))
))))
print(str(str(str(742 …→ ))) ✕→ )))) ✓
Introducing
RaSteer
Rank the reliable. Steer toward them.
0%→≈0%
GPT-2 XL on three- and four-paren tasks, steering only its top-10 heads
+5.49%on HumanEval for Llama2-7b. General coding is preserved.
+20.25%on arithmetic for Pythia-6.9b. It works beyond parentheses.
Beats circuitsOutperforms a circuit-discovery baseline across models and tasks.

Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones

Daking Rai · Samuel Miller · Kevin Moran · Ziyu Yao
NeurIPS 2025 · Conference on Neural Information Processing Systems