Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Ninghao Zhang; Bin Zhu; Shijie Zhou; Jingjing Chen

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Ninghao Zhang^1* Bin Zhu^2*† Shijie Zhou³ Jingjing Chen³

¹ Tsinghua University ² Singapore Management University ³ Fudan University

^* Equal contribution ^† Corresponding author & project lead

ECCV 2026

Paper Video Code coming soon

Abstract

We study a reliability failure in modern Vision-Language-Action models: under out-of-distribution contradictory instructions, policies often ignore instruction semantics and execute actions supported by visual priors. We introduce ICBench, a controlled diagnostic benchmark built from LIBERO, and propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time intervention that shifts attention back toward instruction tokens without retraining or architecture changes.

Linguistic blindness is the tendency of VLA policies to prioritize visual priors over instruction semantics, executing a plausible action even when the instruction is incompatible with the scene.

Normal instructions succeed while contradictory instructions produce visually plausible but instruction-inconsistent executions. — Existing policies can execute the same visually plausible trajectory even when the modified instruction is logically incompatible with the scene.

Method

Instruction-Guided Attention Recalibration

IGAR detects sink tokens through hidden-state spikes, selects grounding heads with cross-modal imbalance, and redistributes attention toward non-sink instruction tokens. The intervention operates entirely inside the forward pass.

Overview of the IGAR framework with sink token detection, grounding head selection, and attention redistribution. — IGAR is a plug-and-play inference-time module for restoring language influence during action generation.

Experimental Results

We first test whether IGAR restores language-sensitive behavior, then verify that it preserves normal task performance and examine why the intervention works.

1. IGAR restores linguistic grounding under contradictory instructions

Under contradiction, a lower Success Rate (SR) means fewer instruction-inconsistent executions, while a higher Linguistic Grounding Score (LGS) indicates stronger language influence.

Suite	Type	π₀		π_0.5		OpenVLA-OFT
Suite	Type	SR ↓	LGS ↑	SR ↓	LGS ↑	SR ↓	LGS ↑
Spatial	V1	76.4	20.4	95.8	1.6	86.4	11.2
	V2	84.2	12.6	93.6	3.8	86.2	11.4
	V3	80.4	16.4	94.8	2.6	90.2	7.4
	V4	76.2	20.6	99.6	-2.2	88.4	9.2
Object	V1	88.2	10.6	94.2	4.2	88.4	10.0
	V2	86.4	12.4	90.4	8.0	84.2	14.2
	V3	93.6	5.2	87.6	10.8	88.2	10.2
	V4	90.4	8.4	88.4	10.0	82.4	16.0
Goal	V1	46.4	49.4	90.2	7.4	66.4	31.6
	V2	40.2	55.6	82.8	14.8	70.2	27.8
	V3	46.2	49.6	92.4	5.2	64.2	33.8
	V4	36.4	59.4	96.2	1.4	58.4	39.6

IGAR mitigation on ICBench. Green cells mark LGS ≥ 10. The largest gain appears on π₀ in the Goal suite, reaching 59.4 LGS for V4.

2. Normal task performance is preserved

Applying IGAR to valid instructions changes average success rates only marginally across all three VLA backbones.

Suite	π₀	π₀ + IGAR	π_0.5	π_0.5 + IGAR	OpenVLA-OFT	OFT + IGAR
Spatial	96.8	96.4 (-0.4)	97.4	98.2 (+0.8)	97.6	97.6 (+0.0)
Object	98.8	98.2 (-0.6)	98.4	94.4 (-4.0)	98.4	99.8 (+1.4)
Goal	95.8	95.6 (-0.2)	97.6	96.4 (-1.2)	98.0	98.2 (+0.2)
Average	97.1	96.7 (-0.4)	97.8	96.3 (-1.5)	98.0	98.5 (+0.5)

Success Rate (%) under normal instructions. IGAR preserves the original policy behavior while adding only 7.0 ms of per-step latency for OpenVLA-OFT.

3. IGAR redirects attention toward language-relevant regions

Attention maps with and without IGAR under normal and contradictory instructions. — The baseline follows visually salient regions regardless of instruction semantics, while IGAR restores attention to the instructed object and spatial relation.

4. The effect transfers to a real robot

Both policies complete the task when the instruction matches the scene.
The baseline still acts under contradiction, producing a visually plausible but invalid success.
IGAR interrupts the inconsistent execution through hovering or empty grasp attempts.

Real-world Franka experiments comparing the baseline policy with IGAR. — Real-world evaluation on the task “place the blue cube into the open drawer.”

5. IGAR remains stable around its selected configuration

Sensitivity curves for text-sink decay, visual-sink bound, and intervened layers. — The selected decay factor, head-selection bound, and intervention depth lie in stable high-performing regions for linguistic grounding.

Video

The project video presents the ICBench setting, the IGAR mechanism, quantitative findings, simulated rollouts, and real-world behavior.

ICBench Rollout Gallery

Under valid instructions, IGAR preserves the baseline policy's real success. When the ICBench instruction is incompatible with the unchanged scene, a language-grounded policy should abstain: IGAR turns the baseline's fake success into a deserved failure.

Choose a task

BibTeX

@inproceedings{zhang2026restoring,
  title={Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration},
  author={Ninghao Zhang and Bin Zhu and Shijie Zhou and Jingjing Chen},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026},
  url={https://arxiv.org/abs/2603.06001}
}