Skip to content

Conversation

@fwip
Copy link
Contributor

@fwip fwip commented Oct 3, 2021

This is a draft PR to track the implementation of automatic cleanup of work directories.

TODO list:

  • Add cleanup scripts (not called).
  • Publish hardlinks by default (makes cleanup easier)
  • Smart non-nextflow solution to prevent duplicate work (e.g: Look for presence of completed file, check LIMS, etc).
  • Automatically clean workdir after successful run and stat upload

See: https://www.nextflow.io/docs/latest/process.html#publishdir

Of the variety of publishing modes, `link` is the only mode that:

1) Allows immediate removal of /work/ without losing output
2) Can be used in any process (unlike `move`)
3) Does not add significant disk I/O (unlike `copy`)
@fwip fwip requested a review from rsandstromUW October 3, 2021 18:35
This safety measure is no longer applicable after we changed to hardlink
publish mode.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants