RAPTR: Radar-based 3D Pose Estimation using Transformer [NeurIPS 2025]

PyTorch implementation of RAPTR (RAdar Pose esTimation using tRansformer), a radar-based pipeline that takes multi-view radar heatmaps as input and estimates 3D human poses. The RAPTR model is trained under Weak Supervision with 3D bounding box (BBox) and 2D keypoint labels.

[Paper] [Code] [Dataset] [BibTeX]

Prerequisites

• uv — a fast Python package manager for creating and managing virtual environments:

  pip install uv

Environment setup [Linux]

Create and populate a virtual environment with all required dependencies:

uv venv --python 3.11
uv sync
uv run mim install mmcv

Activate the virtual environment:

source .venv/bin/activate

Tip: For GPU acceleration, ensure that the appropriate CUDA Toolkit is installed and compatible with your versions of PyTorch and MMCV.

• Linux: Most dependencies (including CUDA-enabled PyTorch and MMCV) will be installed automatically if you follow the above steps. Tested on Ubuntu 24.04.3 LTS and Ubuntu 24.04.2 LTS.
• Windows: You may need to manually install the CUDA-enabled version of MMCV. See MMCV Installation for details. Note: we have not tested this setup on Windows.

Download and organize datasets

Download and prepare two indoor radar datasets - HIBER and MMVR - which provide multi-view radar heatmaps along with 2D (HIBER & MMVR) and 3D (HIBER) pose annotations for training and evaluation.

1. HIBER (MIT License)

• Refer to the official HIBER repository for dataset access instructions.
• After downloading, place the dataset in a directory named HIBER while maintaining the original folder structure:

  HIBER/
  ├── test/
  ├── train/
  └── val/
  

2. MMVR (CC BY-SA 4.0 License)

• Download the full MMVR dataset from Zenodo:

  • P1.zip
  • P2_00.zip, P2_01.zip, P2_02.zip

• Create a directory (e.g. ./MMVR) and extract the archives:

  • unzip P1.zip -d ./MMVR/P1
  • unzip P2_00.zip -d ./MMVR/P2 && unzip P2_01.zip -d ./MMVR/P2 && unzip P2_02.zip -d ./MMVR/P2

• Ensure the final directory layout matches the structure below:

  MMVR/
  ├── P1/
  │   ├── README.md
  │   ├── data_split.npz
  │   ├── load_sample.ipynb
  │   ├── d1s1/
  │   │   ├── 000/
  │   │   │   ├── 00000_meta.npz
  │   │   │   ├── 00000_radar.npz
  │   │   │   ├── 00000_bbox.npz
  │   │   │   ├── 00000_pose.npz
  │   │   │   └── 00000_mask.npz
  │   │   ├── 001/
  │   │   └── ...
  │   ├── ...
  │   └── d4s1/
  └── P2/
      ├── d5s1/
      ├── ...
      └── d9s6/
  

3. Set the dataset root directory

Update the root_dir field in configs/data/default.yaml to point to the dataset root. Place both radar datasets directly under this directory:

<root_dir>/
├── HIBER/
│   ├── test/
│   ├── train/
│   └── val/
└── MMVR/
    ├── P1/
    └── P2/

Training

1. Training jobs

To train RAPTR, run the following command:

python train.py data.protocol={'WALK','MULTI','P1S1','P1S2','P2S1','P2S2'}

• For HIBER, use WALK and MULTI.
• For MMVR, use P1S1, P1S2, P2S1, or P2S2.

You can customize additional hyperparameters such as learning rate, batch size, number of epochs, and early stopping patience in configs/train.yaml.

2. Logging

During training, the following are saved under logs/train/runs/<training_initiation_date>:

• Full configuration information
• Model checkpoints
• TensorBoard event files
• Training progress plots and visualizations

3. Loss function configuration

You may configure loss terms for the pose and joint decoders in configs/train.yaml:

model:
  enable_loss_template_pose: True      # 3D Template (T3D) Loss at pose decoder
  enable_loss_kpt_pose: False          # 2D Keypoint (K2D) Loss at pose decoder
  enable_loss_gravity_pose: False      # 3D Gravity (G3D) Loss at pose decoder
  enable_loss_kpt_joint: True          # 2D Keypoint (K2D) Loss at joint decoder
  enable_loss_gravity_joint: True      # 3D Gravity (G3D) Loss at joint decoder

For RAPTR, we use T3D loss at the pose decoder and K2D+G3D at the joint decoder (see Table 3 in our paper). You can adjust these flags to control which loss terms supervise the pose and joint decoders. Note: Certain combinations of loss flags may require changing the random seed to ensure numerical stability.

Evaluation

You can evaluate either your own trained RAPTR models or the provided pretrained checkpoints.

1. Evaluate your own models

Run the following command using your chosen data protocol and the corresponding saved checkpoint:

python test.py data.protocol={'WALK','MULTI','P1S1','P1S2','P2S1','P2S2'} ckpt_path=<ckpt_path>

• ckpt_path should follow the format: logs/train/runs/<training_initiation_date>/checkpoints/best.ckpt
• Evaluation results, including visualizations, are saved to: logs/test/runs/<training_initiation_date>/eval_figs/

2. Evaluate pretrained checkpoints

Pretrained RAPTR checkpoints (for HIBER) are available in the logs/pretrained/ directory.

To evaluate a pretrained model, run:

python test.py data.protocol={'WALK','MULTI'} ckpt_path=<ckpt_path> data.batch_size=<16 or 32>

• ckpt_path should follow the format: logs/pretrained/<exp_name>/checkpoints/best.ckpt where <exp_name> is one of: multi_1, multi_2, multi_3, walk_1, walk_2, or walk_3.
• Results, including visualizations, are saved to: logs/test/runs/<exp_name>/eval_figs/

You should be able to reproduce the results in the following table by averaging the evaluation performance from the three checkpoints corresponding to each HIBER split:

Dataset	Split	MPJPE (cm)	saved ckpt	batch size
HIBER	MULTI	18.99 $\pm$ 0.16	1, 2, 3	32
HIBER	WALK	22.32 $\pm$ 0.06	1, 2, 3	16

Contributing

See CONTRIBUTING.md for our policy on contributions.

Citation

If you use this repository, please consider citing our paper:

@misc{RAPTR25,
   title       = {{RAPTR}: Radar-based {3D} Pose Estimation using Transformer},
   author      = {Sorachi Kato and Ryoma Yataka and Pu (Perry) Wang and Pedro Miraldo and Takuya Fujihashi and Petros Boufounos},
   year        = {2025},
   eprint      = {2511.08387},
   archivePrefix={arXiv},
   primaryClass= {cs.CV},
   url         = {http://arxiv.org/abs/2511.08387},
}

@inproceedings{RAPTR25_neurIPS,
   title       = {{RAPTR}: Radar-based {3D} Pose Estimation using Transformer},
   author      = {Sorachi Kato and Ryoma Yataka and Pu (Perry) Wang and Pedro Miraldo and Takuya Fuj)ihashi and Petros Boufounos},
   year        = {2025},
   booktitle   = {The Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS)},
   url         = {https://openreview.net/pdf?id=4pUumnQxDG},
}

License

Released under the AGPL-3.0-or-later License, as found in LICENSE.md.

All files, except as noted below:

Copyright (C) 2025 Mitsubishi Electric Research Laboratories (MERL)

SPDX-License-Identifier: AGPL-3.0-or-later

The following files:

• ./src/assigner/hungarian_assigner.py
• ./src/assigner/match_cost.py
• ./src/loss/oks_loss.py
• ./src/loss/focal_loss.py
• ./src/loss/mae_loss.py
• ./src/networks/positional_embedding.py
• ./src/networks/raptr.py
• ./src/networks/transformer.py

were adapted from here (license included in LICENSES/Apache-2.0.txt), with the following copyrights:

Copyright (C) 2025 Mitsubishi Electric Research Laboratories (MERL)
Copyright 2022-2023 Hikvision Research Institute.

SPDX-License-Identifier: AGPL-3.0-or-later
SPDX-License-Identifier: Apache-2.0

Parts of the following files:

• ./src/utils/loss_utils.py
• ./src/utils/tensor_utils.py

were adapted from here (license included in LICENSES/Apache-2.0.txt), with the following copyrights:

Copyright (C) 2025 Mitsubishi Electric Research Laboratories (MERL)
Copyright 2018-2023 OpenMMLab.

SPDX-License-Identifier: AGPL-3.0-or-later
SPDX-License-Identifier: Apache-2.0

Parts of the following files:

• ./src/networks/camera_params.py
• ./src/utils/coordinate_transform_utils.py

were adapted from here (license included in LICENSES/MIT.txt), with the following copyrights:

Copyright (C) 2025 Mitsubishi Electric Research Laboratories (MERL)
Copyright (c) 2022 wuzhiwyyx

SPDX-License-Identifier: AGPL-3.0-or-later
SPDX-License-Identifier: MIT