This is a page that curates AI-related papers published worldwide. All content here is summarized using Google Gemini and operated on a non-profit basis. Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta
Outline
This paper proposes a novel evaluation method, "pseudo-simulation," to address the limitations of existing paradigms for evaluating autonomous vehicles (AVs). Existing real-world evaluations suffer from safety concerns and lack of reproducibility, while closed-loop simulations suffer from lack of realism and high computational costs. Open-loop evaluations, while efficient and data-driven, tend to overlook accumulated errors. Similar to open-loop evaluations, pseudo-simulation uses a real-world dataset but adds synthetic observations generated using 3D Gaussian splatting. Using a proximity-based weighting method, which assigns higher weights to synthetic observations that best match the expected behavior of the AV, we evaluate error recovery and causal confusion mitigation. We demonstrate a higher correlation with closed-loop simulations ($R^2=0.8$) than the best existing open-loop method ($R^2=0.7$), and provide a public leaderboard and code ( https://github.com/autonomousvision/navsim) for benchmarking the new methodology .
Takeaways, Limitations
•
Takeaways:
◦
Presenting pseudo-simulation, a new paradigm for evaluating autonomous vehicles.
◦
Combines the advantages of closed-loop simulation (evaluating error recovery and causal confounding mitigation) with those of open-loop simulation (efficiency, data-driven).
◦
It is more efficient than closed-loop simulations while showing similar performance ($R^2=0.8$).
◦
Encouraging research community engagement through public leaderboards and code contributions
•
Limitations:
◦
Further validation is needed on the accuracy and generalization performance of synthetic observation generation using 3D Gaussian Splatting.
◦
Optimization of proximity-based weighting methods and their applicability to various situations need to be reviewed.
◦
A perfect match with real-world road conditions can be difficult. Further research is needed to determine the realism of synthetic data.
