Sample ID and Version string for output files

[tinavisnovska]

Sample ID and Version string are reported in many output files generated by TSOPPI, it would be great to generate these in some sort of utils module in tsoppy so that these strings are generated consistently across all tsoppy commands

[danielvo]

It's very practical to use sample IDs in output files that relate to a single sample (or perhaps a DNA/RNA/normal sample group of a given patient that has been processed together), as these files can be aggregated later in a single directory without any issues and the meta-information in the filename is very useful. The one issue I can think of would be situations where we rename samples when running TSOPPI. If tsoppy aims to provide this feature (i.e., allowing different input and output sample IDs) and if the various tsoppy tools are not run together within the same NextFlow workflow, then it might be difficult to keep track of the renaming and do things consistently. Else I don't think that there should be much wiggle room when it comes to sample ID handling - I'd think that each tool should treat them as 'constants' and they should always be referred to in the logs of any process. We could definitely try to standardize this and ensure that the logs for all tools are easy to machine read/parse.

The version string in TSOPPI is a concatenation of version of the individual components that I found most critical to keep track of. I assume the NextFlow framework will keep track of versions of all used software and produce a log with them for each run (?). If different parts of tsoppy are updated at different rates, do we want to keep track of individual components' version in addition to the overal tsoppy version? If we want to inform about versions also in the final report (as we do now), we could coordinate with the work-package that aims to update the reports, but that's probably an issue for the report-creating software, whatever it will be (it can extract whatever is deemed relevant from the NextFlow logs). Do we need a separate tsoppy module for dealing with any version string then?

[tinavisnovska]

I see, I was not careful enough with wording saying 'generate sample ID in tsoppy'... sample ID is passed in TSOPPI as input parameter and I assume that is what we expect to be happening in tsoppy too. However, I would like to avoid the situation that every high level command takes sample ID as input parameter and generates output files for that specific functionality, it kind of makes more sense to me to have that sample ID stored somewhere where from all the high level functions will retrieve it when needed, if that makes sense...

[danielvo]

Let's see. I think the idea is that we call tsoppy with sub-commands for individual tools, similarly to "samtools view" and "samtools pielup", right? We can discuss on what level to supply the sample IDs for the Python scripts and how they are stored/retrieved in the code. I assume that when we run tsoppy as a NextFlow pipeline, the sample ID setting/passing would be handled by the NextFlow wrappers, which is very practical if the pipeline takes care of executing let's say 12 separate feature tools for the same sample.
A thing to consider is that some tools will process only one DNA tumor sample, other tools only one RNA tumor sample, some tools will utilize both of these, yet another tool might want to process two different DNA tumor samples (e.g., the tool that compares small variants in two DNA samples in order to help judge whether they are from the same patient or whether one of them serves as a contamination source for the other), etc. The Python code should be able to handle all of that well without NextFlow, but when using NextFlow wrappers, the IDs would be distributed automatically to the right places when executing multiple tools. (Or?)
Does that help in any way, Tina? Or am I still answering the wrong questions?

[tinavisnovska]

Yes, I also imagine that the nextflow part will handle passing sample id to the rules (or whatever the chunks of code are called) that should be executed for a given sample id. However, if sample id is meant to be used to stamp all the output files generated with tsoppy (as it is in TSOPPI and to me it seems that we want it that way for tsoppy too) , then the sample id has to be passed to the python scripts. Am I right here? If so, then maybe the answer to my question about sample id is, that every command in cli.py generating any output files should take sample id as one of the input parameters and use it in the generated output tables and figures. Does this make sense?

Regarding the version string, good that you are pointing out that this is not simply a constant string across all the intended tsoppy cli functions (I did not realise this before), some of the functions may use pcgr to do some jobs, some of them not, other may be using other tools that makes sense to report... that means that the functions should be able to modify the string as they do their job and then print all the used tools in the output files. Nextflow does not necessarily have overview of all the used tools if they are used from inside the python code - or do we want strictly to avoid situations like that?

I find your answers super useful Daniel.