Try LitePT

[jpata]

LitePT is a lightweight, high-performance 3D point cloud architecture. LitePT-S has 3.6 × fewer parameters, 2 × faster runtime and 2 × lower memory footprint than PTv3. Moreover, it remains faster and more memory-efficient even when scaled up to LitePT-L with a parameter count twice that of PTv3. Already the smallest variant, LitePT-S, matches or outperforms state-of-the-art point cloud backbones across a range of benchmarks.

https://github.com/prs-eth/LitePT

NB: A major caveat of this PR right now is that LitePT only supports Nvidia out of the box due to complex dependencies such as spconv-cu124. It also requires pinning torch to torch==2.9.0.

• Add litept as a submodule to ensure tests can run
• do a careful comparison of LitePT vs. Transformer on the CLD hit-based training.