Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions

Created by
  • Haebom

Author

Maxim Popov, Regina Kurkova, Mikhail Iumanov, Jaafar Mahmoud, Sergey Kolyubin

Outline

OSMa-Bench is a dynamically configurable and automated Open Semantic Mapping (OSM) evaluation pipeline based on LLM/LVLM. This paper focuses on evaluating state-of-the-art semantic mapping algorithms under various indoor lighting conditions, introducing a novel dataset containing simulated RGB-D sequences and ground truth 3D reconstructions. We evaluate the semantic fidelity of object recognition and segmentation using leading models such as ConceptGraphs, BBQ, and OpenScene, and analyze the model's ability to interpret semantic structures using a scene graph evaluation method. The experimental results provide insight into the model's robustness and suggest future research directions for developing resilient and adaptive robotic systems.

Takeaways, Limitations

Takeaways:
We present a new benchmark (OSMa-Bench) that allows for systematic evaluation of the performance of the OSM algorithm under various lighting conditions.
Rigorous performance analysis possible with simulated RGB-D datasets and correct 3D reconstructions.
Evaluating the model's ability to interpret semantic structures using a scene graph evaluation method.
Analyze the strengths and weaknesses of cutting-edge OSM models to suggest future research directions.
Limitations:
Evaluations based on simulation data may not fully reflect performance in real environments.
The models being evaluated may be limited.
Further review is needed on the objectivity and generalizability of scene graph evaluation methods.
👍