Refactor: Reduce disk size of git history

[kaedonkers]

At time of writing, cloning the repo involved downloading ~165MB of data, which is large for a git repository. In addition, by far the largest culprit is the .git/ directory, which would imply that there is a lot of relatively large files which were committed in the past but are no longer part of the project. Give the high number of data files stored in this repository, I suspect them to be the main source of this bloat.

I don't suggest we do anything about this yet, but once the repository starts approaching a state in which it looks most like the finished thing, we can look at refactoring the git history to remove this bloat.

For posterity, here are some links to potential solutions:

• https://www.perforce.com/blog/vcs/git-beyond-basics-using-shallow-clones
• https://stackoverflow.com/questions/45864180/how-can-i-reduce-the-size-of-git-folder-with-huge-history-of-commits#comment78697675_45864411

[jucordero]

I've found a few tools and strategies to clean the history:

• Do an --orphan checkout and push to a new main branch after removing the old one. Apparently this option doesn't preserve any history.
• Wonder if a shallow clone and then a new main branch would have a similar effect but with the option to preserve up to a few commits.
• BFG looks like a tool that does exactly what we are looking for. But I have no idea what the side effects in terms of commit history are.
• git filter-branch which I think is the native tool which operates similarly to BFG.

I'm not particularly worried about losing the commit history. Specially now at such an early stage.
The takeaway message, I imagine, is that any new large data commit has to be done through a PR with a disposable branch and possibly squashing commits to simplify the git history.

[jucordero]

Ok, I was able to reduce the size of a fork of this repo to about 100MB using BFG to remove old revisions of large files in the git history.
This leaves every last commit of a file untouched.
If in addition we remove some unnecessary data from the repo to close #21 (there is a population .csv which is ~20MB and we don't really need it) then we should be good.

[kaedonkers]

Great options for de-bloating the commit history - I wasn't aware of BFG or git filter-branch! As you have tested, best to try this on a fork of the repo and make absolutely sure there are no undesirable side-effects.

Regarding JOSS, what are there requirements regarding a git history? Might be worth checking before we push any lighter history to the main branch