Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models

Created by
  • Haebom

Author

Yuhan Hao, Zhengning Li, Lei Sun, Weilong Wang, Naixin Yi, Sheng Song, Caihong Qin, Mofan Zhou, Yifei Zhan, Xianpeng Lang

Outline

This paper introduces DriveAction, a novel benchmark for evaluating Vision-Language-Action (VLA) models in autonomous driving. DriveAction is designed to overcome the limitations of existing benchmarks, which include a lack of diverse scenarios, reliable action-level annotations, and evaluation protocols that are tailored to human preferences. DriveAction is based on real-world autonomous driving data, containing 16,185 QA pairs and 2,610 driving scenarios. It uses high-level discrete action labels directly collected from drivers' actual driving behavior. Furthermore, it implements an action-based, tree-structured evaluation framework to ensure a clear link between vision, language, and actions. Experimental results demonstrate that state-of-the-art VLMs require both visual and language guidance for accurate action prediction, with accuracy degrading by 3.3% without visual input, 4.1% without language input, and 8.0% without both.

Takeaways, Limitations

Takeaways:
DriveAction, a new benchmark for evaluating autonomous driving VLA models, is presented.
Utilizing real-world driving data to ensure a wide range of representative scenarios.
Using high-level action labels collected directly from the driver's actual driving behavior.
Implementing an action-based tree-structured evaluation framework for clear connections between vision, language, and action.
Suggesting model improvement directions through performance analysis of the latest VLMs.
Limitations:
There is no specific mention of Limitations in the paper.
👍