Geometry-biased Transformers for Novel View Synthesis

Geometry-biased Transformers for Novel View Synthesis

Paper    Web    Demo

Environment Setup

For detailed instructions refer to SETUP.md

Dataset

Follow instructions from the official CO3D repository to download the dataset in this format.

Training

Commands

Train GBT model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt.yaml

Train GBT-nb (no geometric bias) model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt_nb.yaml

Note: Modify yaml config files with appropriate num_pixel_queries that can fit on the GPU.

Inference

Checkpoints

Download pre-trained checkpoints from this link. Extract contents inside the repository base directory. Alternatively, run the following commands from terminal.

pip install gdown
gdown 1eHeNba_qlsM-7iEiIlZw9XH9-VXqem7T
unzip runs.zip
rm runs.zip

Verify that the extracted checkpoints are of the following structure.

gbt/runs/co3dv2/cat_agnostic/
|-- gbt
|   `-- latest.pt
`-- gbt_nb
    `-- latest.pt

Commands

Run GBT model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt.yaml --dataset-path /path/to/co3d/dataset --category donut

Run GBT-nb (no geometric bias) model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt_nb.yaml --dataset-path /path/to/co3d/dataset --category donut

Output

The inference script computes average psnr and lpips metrics for objects of the specified category, and also saves individual rotating gifs for qualitative analysis.

runs/co3dv2/cat_agnostic/gbt/infer/num_views=3/donut/
|-- 198_21296_42378.gif
|-- 290_30761_58510.gif
|-- ...
`-- metrics.txt