exp branch vs exp apply for persisting experiments

[dberenbaum]

There have been a few discussions around when to use exp branch vs. exp apply:

• ref: corrections to exp run/apply/branch + dvc.org#2417 (comment)
• https://discord.com/channels/485586884165107732/485596304961962003/836626346544594995

The default workflow we have promoted uses apply as the default for persisting experiments:

• https://dvc.org/doc/user-guide/experiment-management#experiments
• https://dvc.org/doc/start/experiments#persisting-experiments

Since apply doesn't actually make anything persistent on its own, this might be adding to confusion between the commands.

It might make more sense to promote branch as the default for persisting experiment (+ git merge if people want the experiment in their current branch). apply could be seen as a special case when needing to debug or to apply changes on top of a particular past checkpoint.

The branch workflow for persisting experiments is no more complex than apply, and it has advantages:

• Any number of experiments can be made persistent.
• No surprising behavior from merging experiments with workspace (any merging is explicitly done by the user, making it more transparent).
• Checkpoint history is preserved.
• The shas in exp show (and other exp commands) will match the shas in the persistent Git branch.

@pmrowla @jorgeorpinel

[dberenbaum]

One more thought:

There are some exp branch convenience features that could make it easier for users:

• Don't require a branch name - Can dvc use the exp name by default? This seems especially redundant for named exps.
• Add a --merge option or similar to try to merge the experiment into the current branch - To be honest, I'm not sure I like this option since it introduces some of the complications found in apply, but it's probably worth discussing since users may find this extra step inconvenient.

[jorgeorpinel]

I'm not sure I like this option since it introduces some of the complications found in apply

Only if we do git merge --theirs too (like in exp apply), but it could just start the merge normally (basically a shortcut for users who will git merge immediately after anyway), and leave any conflicts unresolved.

[dberenbaum]

maybe we should consider deleting the experiment ref entirely on exp branch

That's an interesting idea. In that case, branch replaces an experiment with a Git branch. Makes sense to avoid confusion/duplication.

Edit: It also gives dvc exp branch some additional convenience over git branch exp-sha.

[karajan1001]

Looks like exp ref are in some remote Git repo, and dvc exp branch equals to git fetch while dvc exp branch --merge equals to git pull.

[pmrowla]

to clarify, exp refs can be both local or remote. exp branch is a local only operation. dvc exp pull is the git fetch/git pull equivalent command.

[karajan1001]

Yes, I know exp refs can be both local or remote, I just feel that they behave like different branches in some remote Git repo.

[jorgeorpinel]

we have promoted uses apply as the default for persisting experiments: ...
Since apply doesn't actually make anything persistent

Just a note that those docs don't state that exp apply makes exps persistent. The fist one says "Make the selected experiment persistent by committing its results to Git.", the second "Let's make it persistent in our regular pipeline by committing it ..."

That said I do expect users to continue bumping into the limitation that exp apply exp-1234 requires the Git HEAD to be the parent of exp-1234. All the other reasons @dberenbaum mentioned seem compelling too. You have by vote!

Would this basically mean updating the docs? Or is there some CLI messaging to update too (@pmrowla?)

[dberenbaum]

The workspace and the experiment commits can become disconnected, and this can be confusing and problematic. This seems at least partially related to the use of dvc exp apply. Some ideas that might improve the workflow:

1. dvc exp checkout to see the experiment in the workspace but stay connected to the experiment commits. We can provide a hint and explain in docs that you are in a detached head state and should use git checkout original_branch to return to your previous state or...
2. dvc exp merge to merge experiments to/from other branches. Above, I suggested dvc exp branch --merge, but maybe it's more useful to merge directly rather than first creating a branch that is the same as the experiment. Also, if dvc exp checkout is supported, it might be nice to support merging from the current experiment to an existing git branch, so that I can examine the experiment result in the workspace followed by merging it into my existing git branch to preserve it.
3. dvc exp run --checkout/--no-checkout (related to guide: clarify what to do with workspace changes after exp run dvc.org#2803 (comment)) to provide an option for whether to keep the experiment in the workspace at the end. If keeping the experiment in the workspace at the end, stay on the experiment branch and provide messaging similar to dvc exp checkout. This again keeps the experiment and workspace connected.
4. Deemphasize dvc exp apply and dvc exp branch. dvc exp apply in particular breaks the connection between the experiment and the workspace. dvc exp branch is less problematic, but it still seems mostly unnecessary. The branch already exists in experiments, so I only really need to be able to merge it to some existing git branch.

@pmrowla @daavoo @iesahin

[jorgeorpinel]

exp merge can stash the workspace changes, merge the experiment branch into the workspace branch, and then reapply any workspace changes
This is the behavior in https://git-scm.com/docs/git-merge#Documentation/git-merge.txt---autostash

If we want this operation I think we should rename it exp stash-merge or something else to avoid confusion for people familiar with regular git stash.

although it [exp merge] could result in conflicts

Here the user would be forced to know how to use Git anyway.

To me this goes back to #5896 : we are adding more Git-operation wrappers instead of exposing the Git implementation and letting the user figure it out. Hopefully we can find a good middle-ground. Idea: give more hints about how to clear the tree or git stash after running an exp.

we can write something like "We are stashing you changes ... to get them back do ...."?

That could work too (no possibility for conflicts) but if the target user of this feature is not Git-savvy I imagine their stash may just grow a lot and they won't know what to do with it so could end up ignoring it and being annoyed that their work went away.

No easy answer...

[iesahin]

Another option is to isolate the workspace from the experiments completely. (Like --temp and --queue do now.) If dvc exp run doesn't apply the experiment results to the workspace, most of the complexity of handling the results vs Git commits is resolved.

If the user wants to apply the experiment results, they can do so by dvc exp apply manually. apply can check whether the workspace is dirty and fail if so. We can also have an --apply flag in exp run to do this automatically. After applying the experiments to a clean Git workspace, it will be the user's responsibility to do whatever they want to do with the results with the usual Git commands. They may discard it (git clean), commit it, create a new branch, stash it, etc. In the current case, we have to think about all of these options.

Just making the experiments isolated (i.e. set --temp as default) will resolve this issue I believe.

[shcheklein]

Another option is to isolate the workspace from the experiments completely.

yep, I think there was a discussion about this initially. I don't remember all the logic behind the decision ... simplicity for the happy path (at least the way folks were considering it) - e.g. run-stop-change-resume-stop-change-resume

workspace is dirty and fail if so.

also, check if we are applying it to a different commit, btw?

[iesahin]

check if we are applying it to a different commit, btw?

dvc exp apply is already doing that check in the current version. It may also take a --merge or --force flag to merge from a different commit.

[iesahin]

I don't remember all the logic behind the decision ... simplicity for the happy path (at least the way folks were considering it) - e.g. run-stop-change-resume-stop-change-resume

The reason is probably implementation simplicity and checkout speed. Checkout takes time for --temp. A "fully isolated" experiment may also require a clean repo to run.

I can see the plus side of the current behavior. Nevertheless, the simpler design would prevent more headaches. We are now in "what might the user be thinking" territory. We can't know this and the decisions to alleviate the current behavior won't make everyone happy.

[iesahin]

I believe, in general, we need to split experiments from Git. They shouldn't require a Git repository in the first place, and from the latest experience, I think "experiments are like git branches/stashes/commits" model brings unnecessary complexity. We used Git to solve some problems easily but now we are on a bus that takes us to where Git goes. We'll have to deal with these issues and more as Git itself is not a simple solution either.

[jorgeorpinel]

split experiments from Git. They shouldn't require a Git repository in the first place

This would be ideal but how do you implement experiments without reinventing the wheel? They're basically revisions you want to control, and that's what SCM does.

p.s. I want to think that since we use dulwich there could be an internal Git repo created just for this feature instead of reusing the user repo. @pmrowla can probably give a better informed opinion. And this was probably already considered at some point...

[iesahin]

We discussed this with @pmrowla in Slack as well. I believe, in the long run, the (team's) energy spent to solve the user's DVC + Git problems is better spent elsewhere but this is a personal opinion and I may be wrong.

They're basically revisions you want to control, and that's what SCM does.

Experiments in their basic definition and use case have nothing to do with SCM. They are discardable copies of the workspace with some helper commands to see the results quickly. SCM part is our implementation detail that became a requirement now.