OSMa-Bench is a dynamically configurable and automated Open Semantic Mapping (OSM) evaluation pipeline based on LLM/LVLM. This paper focuses on evaluating state-of-the-art semantic mapping algorithms under various indoor lighting conditions, introducing a novel dataset containing simulated RGB-D sequences and ground truth 3D reconstructions. We evaluate the semantic fidelity of object recognition and segmentation using leading models such as ConceptGraphs, BBQ, and OpenScene, and analyze the model's ability to interpret semantic structures using a scene graph evaluation method. The experimental results provide insight into the model's robustness and suggest future research directions for developing resilient and adaptive robotic systems.