enable resume training from where stopped

[shane-huang]

make use of ray tune parameter "resume" from "ray.tune.run" -

resume (str|bool) – One of “LOCAL”, “REMOTE”, “PROMPT”, or bool. LOCAL/True restores the checkpoint from the local_checkpoint_dir. REMOTE restores the checkpoint from remote_checkpoint_dir. PROMPT provides CLI feedback. False forces a new experiment. If resume is set but checkpoint does not exist, ValueError will be thrown.

[shane-huang]

maybe we should check whether training results can use below options -

https://ray.readthedocs.io/en/latest/tune-usage.html

local_dir (str) – Local dir to save training results to. Defaults to ~/ray_results.
upload_dir (str) – Optional URI to sync training results to (e.g. s3://bucket).