New video tasks for object detection and tracking

[austinmw]

🚀 Feature

I'd like to request new tasks for:

• Video object detection (models which use multi-frame information to improve detection quality compared to single-frame detection)
• Video multi-object tracking (using either the tracking-by-detection or the joint-detection-association paradigm)

Motivation

These are two very common tasks, but I can't seem to find implementations for either of them that are part of a larger, multi-task vision library, especially with support for pytorch lightning. The closest is probably the mmtracking repository, though it lacks many convieniences of PL. If you want to do either of these tasks right now, your best bet is probably to use a single-model library like ByteTrack, CenterTrack, etc. that often are not robust in terms of training features, distributed training, etc.

Pitch

There's several models available for each. I'd maybe suggest ByteTrack for the MOT task and Temporal RoI Align for video object detection task, though any models that strike a good balance between ease of integration and performance would be great additions. Ideally this would be compatible with pytorch lightning.

[shardulparab97]

Hey @austinmw would you like to have a look at #92 and let me know your initial thoughts(we could discuss and plan which models to use too). Additionally, would be great if the repo maintainers can also know your thoughts on this as it can be a really good feature to add to the PyTorchVideo Library! :)