OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions

Maxim Popov · Regina Kurkova · Mikhail Iumanov · Jaafar Mahmoud . Sergey Kolyubin

IROS 2025

Paper | ArXiv | Video | HuggingFace |Project Page

Table of Contents

1. TODO
2. Data Preparation
3. Semantic Segmentation Evaluation
4. Citation

TODO

• Release Habitat Data Generator
• Release semantic segmentation evaluation module
• Release Visual Question Answering module
• Release Augmented ReplicaCAD dataset
• Release OSMa-Bench dataset

Data Preparation

We offer HaDaGe (Habitat Data Generator), a Habitat-based data generator that supports multiple scene datasets. Currently, it is compatible with Replica, ReplicaCAD, and HM3D datasets, and it can be easily adapted to any dataset supported by Habitat.

We have enhanced the ReplicaCAD dataset with improved semantic annotations and published it as the Augmented ReplicaCAD as part of our benchmark.

Please follow the readme of these packages if you want to prepare the data yourself.

Using HaDaGe, we generated the OSMa-Bench dataset, which builds upon two base datasets:

• Augmented ReplicaCAD: 22 scenes with 4 lighting configurations and a velocity modifier.
• Habitat Matterport 3D (HM3D): 8 scenes with 2 lighting configurations and a velocity modifier.

The pre-generated OSMa-Bench dataset is ready to use with our benchmark for convenience:

git xet install
git clone https://huggingface.co/datasets/warmhammer/OSMa-Bench_dataset -b compressed
unzip OSMa-Bench_dataset/data.zip

Semantic Segmentation Evaluation

This repository provides semantic segmentation (semseg) evaluation tools. The semseg/ directory contains the full evaluation pipeline, including scripts, utilities, and configuration files.

The directory also includes benchmarked methods' repositories, benchmarking results, and Docker setup files with running scripts.

For detailed instructions on setup, execution, and workflow, please refer to the README inside the semseg/ directory.

Citation

Using OSMa-Bench in your research? Please cite:

@inproceedings{popov2025osmabench,
    title={OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions},
    author={Popov, Maxim and Kurkova, Regina and Iumanov, Mikhail and Mahmoud, Jaafar and Kolyubin, Sergey},
    booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    year={2025}
}