Skip to content

COO-173 DAG based pipeline engine#4

Open
Yashvi-Sharma wants to merge 5 commits intomainfrom
COO-173_dag
Open

COO-173 DAG based pipeline engine#4
Yashvi-Sharma wants to merge 5 commits intomainfrom
COO-173_dag

Conversation

@Yashvi-Sharma
Copy link
Collaborator

Major upgrades to the pipeline engine:

  • Pipeline config can have multiple pipelines, each with their own tasks
  • Lazy execution is supported at the pipeline level, i.e. all tasks in that pipeline are executed lazily or eagerly, other pipelines can have a different setting.
  • Double DAG order: a DAG is constructed per pipeline, and one DAG is constructed for inter-pipeline dependencies
  • Tasks/Pipelines in the same generation of DAG order are executed in parallel

Added a base class for preprocessing tasks to reduce redundant code, updated outputs to conform to dict structure.

Minor updates and fixes to utils, data models, and tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant