interval index dag

[xyg123]

this PR adds the step config for running the interval index needed to generate interval features for L2G.

It processes the interval files from their raw format, then identifies interval regions that overlaps our variant index.

[project-defiant]

This PR will require additional changes to the gentropy.yaml and pis.yaml

• Ideally we want to move the interval raw sources to the $release_bucket/input/interval with PIS. Have a look at pis.yaml
• After having raw intervals in the input/intervals we want to refer to them from the gentropy.yaml in simillar way how you defined it in the genetics_etl.yaml.

Note

genetics_etl.yaml is going to be superseded by gentropy.yaml once we define the development workflow for the orchestration and allow to run a subpart of the unified pipeline dag.

[project-defiant]

@xyg123 can you post into the PR the link to the execution of the intervalStep from google cloud dtaproc job and other steps that depend on it?

[xyg123]

Successful run of interval step to generate interval index, took ~10 hours.

[project-defiant]

@xyg123 we need to make some strategic decisions here, because we can not have a step that takes ~10h to calculate during each release.