fix: resolving circular dependencies in MoA steps

[DSuveges]

Context

Upon assembling the unified pipeline, it was apparent that the rewrite introduced a circular dependency. Mechanism of action required target, target required safety, safety requires pharmacogenomics, pharmacogenomics requires mechanism of action.

We decided to use the intermediate target/ensembl dataset as it already contains swissprot and trembl protein identifiers that are used to link ChEMBL target with our target index.

!! This is a temporary solution, the real solution is to separate target etl step from unnecessary annotations.

[DSuveges]

The resulting dataset looks good:

In [3]: spark.read.parquet('/Users/dsuveges/project_data/releases/26.03/output/drug_mechanism_of_action/').printSchema()
root
 |-- actionType: string (nullable = true)
 |-- mechanismOfAction: string (nullable = true)
 |-- chemblIds: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- targetName: string (nullable = true)
 |-- targetType: string (nullable = true)
 |-- targets: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- references: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- source: string (nullable = true)
 |    |    |-- ids: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- urls: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)


In [4]: spark.read.parquet('/Users/dsuveges/project_data/releases/26.03/output/drug_mechanism_of_action_old/').printSchema()
root
 |-- actionType: string (nullable = true)
 |-- mechanismOfAction: string (nullable = true)
 |-- chemblIds: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- targetName: string (nullable = true)
 |-- targetType: string (nullable = true)
 |-- targets: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- references: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- source: string (nullable = true)
 |    |    |-- ids: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- urls: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)


In [5]: spark.read.parquet('/Users/dsuveges/project_data/releases/26.03/output/drug_mechanism_of_action_old/').count()
Out[5]: 6505

In [6]: spark.read.parquet('/Users/dsuveges/project_data/releases/26.03/output/drug_mechanism_of_action/').count()
Out[6]: 6505

[ireneisdoomed]

Nice one!