Skip to content

Check Flag on Storage Before Doing Progressive Post-Processing #10

@matthewwithanm

Description

@matthewwithanm

Currently, we're catching ValueErrors raised by the post_process function and assuming they indicate a storage that doesn't support progressive post-processing, however, this isn't the case. The storage may throw a ValueError for another reason or, worse, it could fail to raise an exception and simply return the wrong URL (for example, if the storage contains an old version of the file).

One possible fix is to add a flag to storage objects to indicate that they generate URLs without requiring a copy of the file on the storage (local_urls?). (Though this might be a little strange/a deviation from expected behavior.) If the flag is not present (or its value is False), eccollect should fail early.

Related: #6

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions