cfde-submit validation error from sample substance.tsv #330
-
|
cfde-submit -ing (--dry-run for validation) a datapackage today with sample substance.tsv, I get this error: |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 7 replies
-
|
@abradyIGS do you know what is causing this? |
Beta Was this translation helpful? Give feedback.
-
|
Typo in the wiki; apologies: the synonyms column was missing. Fixed now, double-checking all other table pages now. |
Beta Was this translation helpful? Give feedback.
-
|
Update: all wiki pages now triple checked: this was the only missing field. Have updated & clarified a bunch of the table description blurbs on various pages in addition to field-name checks. Please yell if anything at all still seems amiss. |
Beta Was this translation helpful? Give feedback.
-
|
Still getting this error:
|
Beta Was this translation helpful? Give feedback.
-
|
Please attach offending substance.tsv? First five rows should suffice if it's big |
Beta Was this translation helpful? Give feedback.
-
|
It appears the offending substance.tsv is being generated by prepare_C2M2_submission.py. Two data rows only.
|
Beta Was this translation helpful? Give feedback.
-
|
Assuming there are two tab characters between the values in the @karlcz do you happen to recognize what's causing the error reported two comments up? @jeremyjyang it might help streamline things if you could double-check that the C2M2_datapackage.json you're submitting with is an exact match for the one here. (TBC this doesn't seem to me like an issue with your local schema copy, but as long as i don't understand what's going on, i figure standard checks can't hurt) |
Beta Was this translation helpful? Give feedback.
-
|
@karlcz could you double check that the submission system is set to expect an array for the synonyms column for substance? It looks like the error from yesterday is a type error in the submission code to me. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for slow response, but yes to question from @abradyIGS :
|
Beta Was this translation helpful? Give feedback.
-
|
I think that error is coming from frictionless-py, not from our CFDE-specific code. If the error message is showing the actual value (not transforming it in some way), then the offending array is not being supplied in JSON format. The meta-syntax for the synonyms array of strings must follow JSON:
i.e. each string is surrounded by the double-quote character Please provide a URL to the exact input files if you want me to continue to investigate this... |
Beta Was this translation helpful? Give feedback.
-
|
This is a problem with my preprocessing code and is also related to this issue -- will push a fix ASAP, need a day or so to reprocess master files |
Beta Was this translation helpful? Give feedback.
-
|
Problem has been identified and fixed; I'm doing final checks on my reprocessing pipeline now and I expect to upload new PubChem reference tables to OSF today that will solve this and a couple of other reported issues. (note to self: stop using phrases like "a day or so") |
Beta Was this translation helpful? Give feedback.
-
|
Please report status and indicate when the fix should be available, and whether via reference tables, and/or CV files, and/or revised submission script, etc. I see there are new CV files dated April 1. |
Beta Was this translation helpful? Give feedback.
-
|
Status is imminent; today is public release day for the new features. Script, final CV files and explicit responses to all pending issues will be delivered by (or shortly after) COB Eastern. Apologies for delays: our schedule was accelerated this quarter by 50%. Thank you for your patience and tire-kicking during pre-deployment testing. |
Beta Was this translation helpful? Give feedback.
-
|
This issue has now tested as "fixed" -- marking "answered," but please reopen and report if you find otherwise. Ontology files and the submission prep script have all been updated and published here: please overwrite local copies and try again. Note: the master C2M2 JSON schema has also updated (to match previous development schema): it is here and an up-to-date copy needs to be included with submissions during prep with the script. |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks, much better, almost there it seems. One thing I notice the sample collection.tsv is missing column has_time_series_data. I should probably be using the schema only and not the sample TSVs I suppose.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for catch - have pushed a fix to the sample TSVs on OSF. |
Beta Was this translation helpful? Give feedback.
This issue has now tested as "fixed" -- marking "answered," but please reopen and report if you find otherwise.
Ontology files and the submission prep script have all been updated and published here: please overwrite local copies and try again.
Note: the master C2M2 JSON schema has also updated (to match previous development schema): it is here and an up-to-date copy needs to be included with submissions during prep with the script.