Replace usage of argparse args with config dataclasses #3554
Description
Relying on the global argparse args object throughout the training code for config settings is unpredictable and makes modularity somewhat impossible. We can instead use the dataclasses introduced in #2313 (and add more as needed) to define all necessary config settings for specific modules (training.py, checkpointing.py, etc.) and then pass instances of those dataclasses within modules instead of accessing global args.
This is a large effort that should be broken up into several PRs. First target module to refactor is checkpointing.
Design doc available on request.