Metadata correction

[candleindark]

This PR provides a solution to correct the corruptions in metadata documented in dandi/dandi-schema#276.

The solution is implemented in two parts.

1. A general command, correct_metadata, to correct dandiset metadata.
2. A specific helper function which the general command calls to do the actual correction.

Extensive tests are provided for the helper function and its supporting function. However, because I am not familiar with this repo and Django in general, I am not able to provide tests for the command which interacts with the database. Advice and additional tests are very much appreciated.

The command can be run on a targeted dandiset at a particular version and run on all versions of all dandisets. Running on all dandiset versions will only correct the corrupted dandisets. If running only on targeted dandiset version is preferable, please let me know, and I will provide the list of corrupted dandiset versions. (Additionally, would changing the interface of the command to a file consisting of corrupted dandiset versions be better?)

[asmacdo]

a couple of questions to start off with-- I'll have another look tomorrow.

[waxlamp]

Thank you, @candleindark! One thing to change listed below. And I've asked @jjnesbitt to provide a more comprehensive review on this R as well.

[candleindark]

@jjnesbitt I have put in changes requested by @waxlamp regarding the command line interface. With the latest push, two unrelated tests failed. Can they be related to the latest changes in dandi-cli?

[jjnesbitt]

I've pushed a commit which aims to simplify the logic in this management command. Namely:

• Remove the configuration of the correcting function.
  • We have no real use case for this yet, and no way to change this function is provided.
• Removal of verbose error handling logic
  • Along the same lines as the above point, since this is a single function, this is largely unnecessary. We have tests, and if the correcting function fails for some reason, the error will simply bubble up.
• Only support correction of the Affiliation schema key
  • Similarly again, we only have the use case of Affiliation at the moment, so all of the configuration surrounding other schema keys is unnecessary at the moment.
• Remove find_objs tests against generalized schema key
• Write manifest files synchronously
  • Since we are not in a request context, it's better to write the manifest files to S3 synchronously, so that we can be sure it succeeds.
• Use transactions to couple logic of metadata save and manifest file generation

IMO this is ready, maybe @waxlamp should take another look.

[waxlamp]

As per the meeting of 2025-03-10 (involving @yarikoptic, @candleindark, @jjnesbitt, @kabilar, and @waxlamp), we decided to reduce the scope of this change to just the draft versions; @jjnesbitt will work on adjusting the PR to that narrowed scope.

@candleindark, @yarikoptic: are we planning to release a new schema version that has restrictions on extra fields? If we apply the metadata correction to draft versions, then I suppose whether or not such a schema is released, there will not be any further validation errors.

[candleindark]

@candleindark, @yarikoptic: are we planning to release a new schema version that has restrictions on extra fields?

Yes, some minor changes to the schema will be introduced by this PR

If we apply the metadata correction to draft versions, then I suppose whether or not such a schema is released, there will not be any further validation errors.

I can't say that after the corrections to the draft version that there will be no more validation errors in those versions. I can only say that if you make the corrections to the draft versions, there will not be validation errors in those version due to that particular change brought by that PR

dandi/dandi-schema#266 (comment) provides an analysis of validation errors in the metadata instances caused by the changes in the PR but not an analysis of validation errors in the metadata instances in general.

[waxlamp]

If we apply the metadata correction to draft versions, then I suppose whether or not such a schema is released, there will not be any further validation errors.

I can't say that after the corrections to the draft version that there will be no more validation errors in those versions. I can only say that if you make the corrections to the draft versions, there will not be validation errors in those version due to that particular change brought by that PR

Sorry, this was what I meant.

[jjnesbitt]

Placing this into draft mode until #2224 is merged, at which point this PR can be updated to make use of it.

[yarikoptic]

do you think you would have time to work out solution for #2224 (it is an issue) or should someone else try to approach it to facilitate a potentially (not necessarily "likely" ;) ) faster resolution?

[jjnesbitt]

do you think you would have time to work out solution for #2224 (it is an issue) or should someone else try to approach it to facilitate a potentially (not necessarily "likely" ;) ) faster resolution?

I'm already working on #2224, it shouldn't take much time to complete.

[jjnesbitt]

@waxlamp @mvandenburgh This is ready to go now.

[mvandenburgh]

Just some suggested optimizations to memory usage. It's probably not needed due to the number of Versions not being super high, but the heroku run dynos have pretty limited memory so it seems logical to do.

Otherwise, LGTM

Stale

[yarikoptic]

So this was merged and now we got the command. Who is allowed to run it, as could I ?

[jjnesbitt]

So this was merged and now we got the command. Who is allowed to run it, as could I ?

It is only in staging at the moment, not in production. I am planning to run in staging, and then once this is deployed in production (blocked by vue3), run there as well.

[dandibot]

🚀 PR was released in v0.5.0 🚀

[jjnesbitt]

This has been successfully applied in staging and production.