Dockerize #1

fwip · 2018-03-16T20:42:28Z

Most of the work on this is just wrangling the hidden script dependencies and hard-coded paths, parameterization, etc. You can see that the Dockerfile itself is not that complex.

This depends on the hotspot2 dockerfile: Altius/hotspot2#25.

This will run all steps sequentially.

Example usage, assuming your data is in a folder ./test_data/:

docker run \
    --mount type=bind,source="$(pwd)"/test_data,target=/data \
    masterlist \
    ./run_from_scratch.sh \
    /data/fileOf3BAMfiles.txt \
    /data/docker_out \
    /data/chr21.chrom.bed \
    /data/GRCh38_36merMappableOnly_minusBlacklist_chrs1-22XY.starch \
    20

Which will create output like:

test_data/docker_out/allPeaks.starch
test_data/docker_out/centersites.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.allcalls.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.cleavage.total
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.cutcounts.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.density.bw
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.density.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.fragments.sorted.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.hotspots.fdr0.0010.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.peaks.narrowpeaks.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.peaks.starch
test_data/docker_out/chr21.LN10051.GRCh38_no_alts.uniques.sorted.SPOT.txt
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.allcalls.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.cleavage.total
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.cutcounts.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.density.bw
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.density.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.fragments.sorted.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.hotspots.fdr0.0010.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.peaks.narrowpeaks.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.peaks.starch
test_data/docker_out/chr21.LN10085.GRCh38_no_alts.uniques.sorted.SPOT.txt
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.allcalls.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.cleavage.total
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.cutcounts.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.density.bw
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.density.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.fragments.sorted.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.hotspots.fdr0.0010.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.peaks.narrowpeaks.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.peaks.starch
test_data/docker_out/chr21.LN20451.GRCh38_no_alts.uniques.sorted.SPOT.txt
test_data/docker_out/subdir/chunk0001.bed
test_data/docker_out/subdir/chunk0002.bed
test_data/docker_out/subdir/chunk0003.bed
test_data/docker_out/subdir/chunk0004.bed
test_data/docker_out/subdir/chunk0005.bed
test_data/docker_out/subdir/chunk0006.bed
test_data/docker_out/subdir/DHSs_all/chunk0001.bed
test_data/docker_out/subdir/DHSs_all/chunk0002.bed
test_data/docker_out/subdir/DHSs_all/chunk0003.bed
test_data/docker_out/subdir/DHSs_all/chunk0004.bed
test_data/docker_out/subdir/DHSs_all/chunk0005.bed
test_data/docker_out/subdir/DHSs_all/chunk0006.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0001.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0002.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0003.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0004.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0005.bed
test_data/docker_out/subdir/DHSs_nonovl_any/chunk0006.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0001.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0002.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0003.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0004.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0005.bed
test_data/docker_out/subdir/DHSs_nonovl_core/chunk0006.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0001.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0002.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0003.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0004.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0005.bed
test_data/docker_out/subdir/DHSs_nonovl_stats/chunk0006.bed
test_data/docker_out/subdir/masterlist_DHSs_masterlist_all.bb
test_data/docker_out/subdir/masterlist_DHSs_masterlist_all.bed
test_data/docker_out/subdir/masterlist_DHSs_masterlist_all_chunkIDs.txt
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_any.bb
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_any.bed
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_any_chunkIDs.txt
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_core.bb
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_core.bed
test_data/docker_out/subdir/masterlist_DHSs_masterlist_nonovl_core_chunkIDs.txt
test_data/docker_out/subdir/peaks_all/chunk0001.bed
test_data/docker_out/subdir/peaks_all/chunk0002.bed
test_data/docker_out/subdir/peaks_all/chunk0003.bed
test_data/docker_out/subdir/peaks_all/chunk0004.bed
test_data/docker_out/subdir/peaks_all/chunk0005.bed
test_data/docker_out/subdir/peaks_all/chunk0006.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0001.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0002.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0003.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0004.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0005.bed
test_data/docker_out/subdir/peaks_nonovl_any/chunk0006.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0001.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0002.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0003.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0004.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0005.bed
test_data/docker_out/subdir/peaks_nonovl_core/chunk0006.bed
test_data/docker_out/subdir/Rout/output_build_chunk_1.Rout
test_data/docker_out/subdir/Rout/output_build_chunk_2.Rout
test_data/docker_out/subdir/Rout/output_build_chunk_3.Rout
test_data/docker_out/subdir/Rout/output_build_chunk_4.Rout
test_data/docker_out/subdir/Rout/output_build_chunk_5.Rout
test_data/docker_out/subdir/Rout/output_build_chunk_6.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_1.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_2.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_3.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_4.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_5.Rout
test_data/docker_out/subdir/Rout/output_overlap_chunk_6.Rout

fwip added 14 commits March 14, 2018 15:52

Add general.R to repository

ffdceb6

Add preliminary dockerfile

7cb8d92

code_build.R: read file from commandlint

9f721f2

Fix up existing scripts

3882c1e

Add script to run masterlist generation

b0e6e4e

Add script to run all of hotspot2 & masterlist gen

8ab3503

scratch script fixups

11baeed

Add perl to dependencies

5ab7173

Fix matrix arg parsing

2a83bfb

Add namingDHS src dir

2142fc0

Update masterlist code

81f042f

Fix awk & dir errors in run_from_scratch

9b59832

Fix bugs in run_sequential, ignore matrix code

50fb582

Merge remote-tracking branch 'origin/master' into dockerize

faa33a8

fwip requested a review from meuleman March 16, 2018 20:42

fwip added 2 commits March 16, 2018 14:24

Remove perl scripts and dependency

9c2a912

Remove general.R dependency

f48e362

meuleman approved these changes Mar 16, 2018

View reviewed changes

meuleman merged commit 28504b5 into master Mar 16, 2018

fwip deleted the dockerize branch March 19, 2018 23:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dockerize #1

Dockerize #1

Uh oh!

fwip commented Mar 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Dockerize #1

Dockerize #1

Uh oh!

Conversation

fwip commented Mar 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants