cannot fit error model

[XinyuWu666]

Hi, I run DimSum to process my data with around 40M reads. The data once split can be processed but the whole data can not be processed efficiently. Specifically, I let the program run for 24h at least and it stays at stage 5 and has no sign to go forward:

******* DiMSum STAGE 5 (STEAM): ANALYSE VARIANT COUNTS *******

Filtering out low count nucleotide variants...
Done
Aggregating counts for biological output replicates...
Done
Fit error model...

Do you think there may be something I can do to improve the software behaviour or it is just because of the reads are too much?

[XinyuWu666]

Sorry just to add the experimental design file. I found if I don't put 2 experiment rep here, it runs very fast.

NegA 1 0 1 test1.fastq
NegA 1 0 2 test2.fastq
NegA 1 0 3 test3.fastq
NegB 2 0 1 test4.fastq
NegB 2 0 2 test5.fastq
NegB 2 0 3 test6.fastq
PosA 1 1 1 1 test7.fastq
PosA 1 1 1 2 test8.fastq
PosA 1 1 1 3 test9.fastqq
PosB 2 1 1 1 test10.fastq
PosB 2 1 1 2 test11.fastq
PosB 2 1 1 3 test12.fastq

[andrefaure]

Hi @XinyuWu666 , it seems the bottleneck is the error model fitting process - without a second experiment replicate the errors are simply poisson and no error model is fit.

40M reads is not a particularly large number - the problem is more likely to be a very large number of (distinct) variants.

I suggest trying:

1. Running on a bigger memory machine and/or increasing the number of cores used --numCores or
2. Setting --fitnessErrorModel to FALSE (i.e. skip error model fitting) or
3. Apply more stringent filtering of variant sequences (stages 2,3,4) or minimum variant sequence counts (stage5) https://github.com/lehner-lab/DiMSum/blob/master/docs/ARGUMENTS.md

Also, if you share the html report output I'd be happy to take a look.

[XinyuWu666]

Hi, Andre. Thanks for replying.

Maybe it is because of the variants? There are around 120K unique mutations in the library. Currently, the filter is:

1. cutadaptMinLength=10
2. cutadaptErrorRate=0.1
3. fitnessMinInputCountAll=1
4. barcodeErrorRate=0.1

I think skipping error model fitting solves my problem but i will try to use more strict filters, as i think DimSum's error modelling is very helpful in giving the confidence towards data.