Batali cache: git sources problems building working tree and uncommitted changes

[webframp]

Filing an issue for tracking this one since it's kind of strange. Using latest code from #92 in an attempt to fix #88

Install fails for what appears to be a dirty git working tree for a git source, but the tree is actually clean.

[ERROR]: Reason - git "--git-dir=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=/.git" "--work-tree=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=" pull "origin" "master"  2>&1:error: cannot pull with rebase:
You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them.
ERROR: Git::GitExecuteError: git "--git-dir=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=/.git" "--work-tree=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=" pull "origin" "master"  2>&1:error: cannot pull with rebase: You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them.

C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> cd "C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ="

C:\Users\sescriva\AppData\Local\.batali\cache\git\Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ= [master ≡]> git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean

Scrubbing cache and re-installing results in same error as #88 (which was pre-existing):

C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> be batali cache -s
[Batali]: Remove all contents from local cache (C:\Users\sescriva\AppData\Local/.batali/cache) (Y/N): Y
[Batali]: Scrubbing local cache... complete!
Batali cache information:
  Path: C:\Users\sescriva\AppData\Local/.batali/cache
  Size: 0.00M
C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> be batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... error!
[ERROR]: Reason - No such file or directory - Failed to locate asset within `C:\Users\sescriva\AppData\Local/.batali/cache/site/remote_site/aHR0cHM6Ly9zdXBlcm1hcmtldC5jaGVmLmlvOjQ0My9hcGkvdjEvY29va2Jvb2tzL3dpbmRvd3MvdmVyc2lvbnMvMi4xLjEvZG93bmxvYWQ=`
ERROR: Errno::ENOENT: No such file or directory - Failed to locate asset within `C:\Users\sescriva\AppData\Local/.batali/cache/site/remote_site/aHR0cHM6Ly9zdXBlcm1hcmtldC5jaGVmLmlvOjQ0My9hcGkvdjEvY29va2Jvb2tzL3dpbmRvd3MvdmVyc2lvbnMvMi4xLjEvZG93bmxvYWQ=`

[webframp]

Oddly ran into a similar thing again, in our CI environment:

$ bundle exec batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... ERROR: Git::GitExecuteError: git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0' checkout 'ed03aee3721ff69158ed40fb10dd10ecdb01aff9'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	metadata.rb
Please, commit your changes or stash them before you can switch branches.
Aborting
error!
[ERROR]: Reason - git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0' checkout 'ed03aee3721ff69158ed40fb10dd10ecdb01aff9'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	metadata.rb
Please, commit your changes or stash them before you can switch branches.
Aborting

[webframp]

I don't know why the initial checkout of a repo would result in a git index out of sync with master but I've ran into two different scenarios. One were files are reported as deleted, and another where files are modifed.

Maybe something as dumb as this would be enough in the right place in: https://github.com/spox/batali/blob/master/lib/batali/git.rb#L19

if repo.status.changed.any?
  # whatever 'git checkout -- .' looks like using git gem
end

if repo.status.deleted.any?
  # whatever 'git reset HEAD' looks like using the git gem
end

[webframp]

Happening pretty consistently for me in our CI environment, let me know what sort of good debug info I could provide if needed

[webframp]

Here's a little more strangeness, same basic issue. First the error:

ERROR: Git::GitExecuteError: git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0' checkout '7682c1df1c070f5c3ff12654bae479b7f4913150'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	README.md
	attributes/default.rb
	files/artifactory.system.properties
	metadata.rb
	recipes/artifactory.rb
	spec/unit/recipes/artifactory_spec.rb
	test/integration/artifactory/inspec/artifactory_spec.rb
	test/integration/default/inspec/default_spec.rb
Please, commit your changes or stash them before you can switch branches.
error: The following untracked working tree files would be removed by checkout:
	files/artifactory.base.config.xml
	recipes/artifactory_config.rb
	recipes/artifactory_monitoring.rb
	spec/unit/recipes/artifactory_config_spec.rb
	spec/unit/recipes/artifactory_monitoring_spec.rb
Please move or remove them before you can switch branches.
Aborting

But then if I look at the actual repo in the cache it's fine:

$ cd /home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0/
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean

Then if I simply rm -fr /home/gitlab-runner/.batali the next attempt gives this error:

ERROR: Git::GitExecuteError: git  clone '--' 'git@gitlab.selfhosted.url:chef/deploy_key_vault.git' '/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2RlcGxveV9rZXlfdmF1bHQuZ2l0'  2>&1:fatal: working tree '/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9iYXNlLmdpdA==' already exists.

A bit stumped I guess.

[webframp]

Best guess at this point from a little testing: git repos where files have been committed with windows line endings are used as git sources in the Batali file and then batali is run on a linux system. Somewhere along the line the resulting git repo in the batali cache shows changes due to line ending differences.

If there was an option to always clobber the cache, or a --no-cache behavior, it might be enough to workaround it at the cost of being a little slower.

[webframp]

I forced core.eol = lf globally in a few places where I was seeing this but it didn't help

[chrisroberts]

@webframp After some git digging, it looks like this might fix things up:

git config core.autocrlf false

[webframp]

Unfortunately I tried that and still get the issues. Line endings with git are the most annoying cross platform issue. This page has more info too: http://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/

I still either get 'Your local changes to the following files would be overwritten by checkout' error, or I get the following, which seems related but looks like a different error:

$ rm -fr ~/.batali/
$ bundle exec batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... error!
[ERROR]: Reason - git  clone '--' 'git@gitlab.selfhosted.url:chef/cookbook-build_machine.git' '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJ1aWxkX21hY2hpbmUuZ2l0'  2>&1:fatal: working tree '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9jaGVmX2NsaWVudC5naXQ=' already exists.
ERROR: Git::GitExecuteError: git  clone '--' 'git@gitlab.selfhosted.url:chef/cookbook-build_machine.git' '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJ1aWxkX21hY2hpbmUuZ2l0'  2>&1:fatal: working tree '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9jaGVmX2NsaWVudC5naXQ=' already exists.

I'm probably going to have to stop using a git source to actually get past this which is ok. If I can help debug or step through things better let me know. I'm happy to monkey patch the installed batali with debug code even if you think it will help pinpoint the issue

[webframp]

@chrisroberts I might have a scenario to recreate this, will contact you on irc to discuss

[webframp]

Annoyingly, this does seem to be a line ending issue. Add a .gitattributes file to every git repo referenced by the Batali file "fixed" the issue. I just used a simple one with the following contents:

* text eol=native