Basic 2D/3D Vision Tasks Evaluation

Performance measures for evaluating 2D/3D multi-object detection, segmentation and tracking.

Implemented Performance Measures

The following performance measures are currently implemented:

Measures	Sub-Measures	Paper	Code	Notes
OSPA(2) (OSPATrack) metric	OSPA2, OSPA2_CARD, OSPA2_LOC	paper	code	A metric for multi-object tracking
OSPA (OSPADetection) metric	OSPA, OSPA_CARD, OSPA_LOC	paper	code	A metric for multi-object detection and segmentation
HOTA	HOTA, DetA, AssA, LocA, DetPr, DetRe, AssPr, AssRe	paper	code
CLEARMOT	MOTA, MOTP, MT, ML, Frag, etc.	paper	code
Identity	IDF1, IDP, IDR	paper	code
VACE	ATA, SFDA	paper	code
Track mAP	Track mAP	paper	code	Requires confidence scores
J & F	J&F, J, F	paper	code	Only for Seg Masks
ID Euclidean	ID Euclidean	paper	code

Note:

• Generalized Intersection over Union (GIOU) is included as a base metric for bounding box evaluation. Use the 'GIOU' flag in the evaluation scripts to choose the base metric. Note that the GIOU score is normalized to values between 0 and 1, same as the IOU score.
• OSPA(2) and OSPA metrics are given in terms of distances (errors), hence the lower the better.
• OSPA(2) and OSPA are the main distances (errors). OSPA2_CARD and OSPA_CARD are the cardinality error components, and OSPA2_LOC and OSPA_LOC are the localization error components.
• By definitions:
  • OSPA2 = OSPA2_CARD + OSPA2_LOC,
  • OSPA = OSPA_CARD + OSPA_LOC.

Implemented Benchmarks

The following benchmarks are currently implemented:

Benchmark	Sub-benchmarks	Type	Website	Code	Data Format
Curtin Multi-Camera	CMC1/2/3/4/5	3D BBox	website	code	format
RobMOTS	Combination of 8 benchmarks	Seg Masks	website	code	format
Open World Tracking	TAO-OW	OpenWorld / Seg Masks	website	code	format
MOTChallenge	MOT15/16/17/20	2D BBox	website	code	format
KITTI Tracking		2D BBox	website	code	format
BDD-100k		2D BBox	website	code	format
TAO		2D BBox	website	code	format
MOTS	KITTI-MOTS, MOTS-Challenge	Seg Mask	website	code and code	format
DAVIS	Unsupervised	Seg Mask	website	code	format
YouTube-VIS		Seg Mask	website	code	format
Head Tracking Challenge		2D BBox	website	code	format
PersonPath22		2D BBox	website	code	format
BURST	{Common, Long-tail, Open-world} Class-guided, {Point, Box, Mask} Exemplar-guided	Seg Mask	website	format

Evaluating Multi-Object Detection and Segmentation

OSPA metric (not OSPA(2)) can be used for multi-object detection/segmentation, and it can be done by:

• First, convert your data to MOTChallenge format (2D bounding box or segmentation mask), treating each image as a frame.
• In each image, assign the detected objects with distinct IDs to ensure the data pre-processing steps can be done correctly. Note that, the IDs can be reused for different images as long as in each image they are distinct.
• The OSPA metric does not use object IDs in the evaluation, which means, the distinct IDs can be generated randomly.
• Finally, use the OSPA metric (not OSPA(2)) to evaluate.

Running the Code

The code can be run in one of two ways:

• From the terminal via one of the scripts here. See each script for instructions and arguments.
• Directly by importing this package into your code, see the same scripts above for how.

Outputs

By default the code prints results to the screen, saves results out as both a summary txt file and a detailed results csv file, and outputs plots of the results. All outputs are by default saved to the 'tracker' folder for each tracker.

Quick Test on Supported Benchmarks

Ground-truth data, meta-data and example trackers for all currently supported benchmarks are provided in the following link. You can download this here: data.zip (~150mb).

The data for RobMOTS is separate and can be found here: rob_mots_train_data.zip (~750mb).

The data for PersonPath22 is separate and can be found here: person_path_22_data.zip (~3mb).

The easiest way to begin is to extract this zip into the repository root folder such that the file paths look like: VisionEvaluation/data/gt/...

This then corresponds to the default paths in the code. You can now run each of the scripts here without providing any arguments and they will by default evaluate all trackers present in the supplied file structure. To evaluate your own tracking results, simply copy your files as a new tracker folder into the file structure at the same level as the example trackers (MPNTrack, CIWT, track_rcnn, qdtrack, ags, Tracktor++, STEm_Seg), ensuring the same file structure for your trackers as in the example.

If your ground truth and tracker files are located somewhere else, you can use the script arguments to point the code toward your data.

To ensure your tracker outputs data in the correct format, check out the format guides for each of the supported benchmarks here, or check out the example trackers provided.

Evaluation on Custom Benchmark

To evaluate on a custom benchmark, there are two options:

• Write custom dataset code.
• Convert your current dataset and trackers to the same format as an already implemented benchmark.

To convert formats, check out the format specifications defined here.

By default, it is recommended to use the MOTChallenge format, although any implemented format should work. Note that for many cases you will want to use the argument --DO_PREPROC False unless you want to run preprocessing to remove distractor objects.

Requirements

Code tested on Python 3.7.

• Minimum requirements: numpy, scipy
• For plotting: matplotlib
• For segmentation datasets (KITTI MOTS, MOTS-Challenge, DAVIS, YouTube-VIS): pycocotools
• For DAVIS dataset: Pillow
• For J & F metric: opencv_python, scikit_image
• For simples test-cases for metrics: pytest

use pip3 -r install requirements.txt to install all possible requirements.

use pip3 -r install minimum_requirments.txt to only install the minimum if you don't need the extra functionality as listed above.

Acknowledgement

Parts of the codes and most of the information in this README file are released under the 'TrackEval' repo by Jonathon Luiten and Arne Hoffhues at https://github.com/JonathonLuiten/TrackEval. Please include the following citation in your research if you use the code.

@misc{luiten2020trackeval,
  author =       {Jonathon Luiten, Arne Hoffhues},
  title =        {TrackEval},
  howpublished = {\url{https://github.com/JonathonLuiten/TrackEval}},
  year =         {2020}
}

The OSPA(2) and OSPA metrics (and other performance criteria) for multi-object tracking, detection and segmentation evaluation are discussed in the following paper:

@ARTICLE{9976259,
  author={Nguyen, Tran Thien Dat and Rezatofighi, Hamid and Vo, Ba-Ngu and Vo, Ba-Tuong and Savarese, Silvio and Reid, Ian},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={How Trustworthy are Performance Evaluations for Basic Vision Tasks?}, 
  year={2023},
  volume={45},
  number={7},
  pages={8538-8552},
  doi={10.1109/TPAMI.2022.3227571}}

If you use any performance measures and benchmarks, please cite the relevant papers.

Contact

Contributor: Tran Thien Dat Nguyen. Email: tranthiendat.nguyen@gmail.com