Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance.
Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation.
To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance.
Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures.
Overview of ViFailback Framework. Left: We collect real-world manipulation trajectories via teleoperation and policy rollout, then use our high-efficiency, visual-symbol-based annotation framework to generate VQA pairs for the dataset. Middle: Our dataset comprises 58,126 VQA pairs from 5,202 real-world trajectories. We extract ViFailback-Bench (Lite and Hard) from this dataset to evaluate VLM failure diagnosis and correction capabilities. Right: We fine-tune Qwen3-VL-8B on our VQA pairs to obtain ViFailback-8B. This model is deployed as an external supervisor to assist the policy in recovering from failures.
An overview of ViFailback-Bench. Left: The Lite benchmark uses close-ended VQA to test VLM failure diagnosis (e.g., detection, localization) and low-level correction guidance. Right: The Hard benchmark uses open-ended VQA to test failure reason and high-level/CoT-based guidance.
Comparison of overall model performance on ViFailback-Bench. All metrics are reported as accuracy (%).
Demos that showcase the VLA rollout with ViFailback-8B assisting in failure correction using visual symbols.
VSF Method: Incorporating Visual Symbols-Following Dataset
PMC Method: Point-based Motion Control
@article{zeng2025vifailback,
author = {Xianchao Zeng and Xinyu Zhou and Youcheng Li and Jiayou Shi and Tianle Li and Liangming Chen and Lei Ren and Yong-Lu Li},
title = {Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols},
year = {2025},
eprint = {2512.02787},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2512.02787},
}