Add --times flag to exp run/repro

[dberenbaum]

It would be great to have a --times flag (maybe @jorgeorpinel has a better naming suggestion) that runs an experiment X many times. In other words, dvc exp run --times 5 would run the same experiment 5 times.

This benefits a couple of use cases:

1. Checkpoints (see Document checkpoints without make_checkpoint() dvc.org#2292): A user may want to repeat a checkpoint experiment multiple times to run through many epochs rather than having that hard-coded into the script or requiring the user to break out of an infinite loop.
2. Non-deterministic stage: A user may want to compare multiple non-deterministic experiment runs. For example, a user could read in random splits of training and test data on each experiment run to perform cross-validation.

One complication is that the first scenario is sequential and the second is parallelizable, but similar concerns are true with --queue/--run-all. Maybe this could even be merged with --queue, like allowing multiple identical experiments to be added to accomplish the same idea.

Thoughts @pmrowla?

[jorgeorpinel]

Current workaround (Linux):

$ for run in {1..10}; do
  dvc exp run ... ;
done

Questions:

1. How would users run that kind of code without DVC? (Using some other logger e.g. Dvclive or without logging) Same as above?
2. Not sure it makes sense to support non-deterministic stages, as they're not reproducible anyway.

As for the name, maybe --re(peat) <number>

[dberenbaum]

2\. Not sure it makes sense to support non-deterministic stages, as they're not reproducible anyway.

Good point, but we do support those now with --always-changed, right?

[jorgeorpinel]

Is that the use case for stage add --always-changed? TBH I'm not sure 🤷

UPDATE: Doesn't seem like that's the intended use per #2479. Something about forcing "callback stages" (without deps) in the middle of a pipeline.

[pmrowla]

IMO this seems like something that should just be done by the user through shell scripting?

[dberenbaum]

I see some downstream uses for this, but I can put them in an enhancement proposal with other thoughts on future experiment/checkpoint developments. If you don't mind, we can leave this here until I get to that.

[jorgeorpinel]

Apparently it can be difficult to quit shell loops (e.g. a while true) per iterative/dvc.org#2292 (comment). That would be an argument for this.

[pmrowla]

That's specifically when it comes to using an infinite loop in a shell script. This flag would be for a fixed number of iterations, which isn't the same thing.

[jorgeorpinel]

True. But you may want to exit a loop of 100,000 that takes all night. I'm guessing something like that has the same problem via shell scripting? BTW this assumes exp run --times 100000 is easier to interrupt.