GeoTiff Products exposed as files in jobs

[cmarshak]

I have a research plugin that I am using and my job after succeeding looks like this:

{'name': 'chiapas',
 'job_type': 'OPERA_DIST_S1',
 'job_parameters': {'high_confidence_alert_threshold': 4.5,
  'mgrs_tile_id': '15QUU',
  'post_date_buffer_days': 1,
  'stride_for_norm_param_estimation': 7,
  'n_workers_for_norm_param_estimation': 8,
  'delta_lookback_days_mw': 'none',
  'use_date_encoding': False,
  'model_source': 'transformer_optimized',
  'memory_strategy': 'high',
  'batch_size_for_norm_param_estimation': 32,
  'post_date': '2024-01-25',
  'track_number': 34,
  'low_confidence_alert_threshold': 2.5,
  'n_workers_for_despeckling': 8,
  'device': 'best',
  'max_pre_imgs_per_burst_mw': 'none',
  'model_compilation': False},
 'thumbnail_images': [],
 'job_id': 'fa126dac-873b-41f1-bd53-aea8b96893b2',
 'user_id': 'cmarshak',
 'priority': 9999,
 'status_code': 'SUCCEEDED',
 'request_time': '2025-09-04T21:06:49+00:00',
 'files': [],
 'expiration_time': '2025-09-19T00:00:00+00:00',
 'browse_images': ['https://hyp3-tibet-jpl-test-contentbucket-hratibh1y9pa.s3.us-west-2.amazonaws.com/fa126dac-873b-41f1-bd53-aea8b96893b2/OPERA_L3_DIST-ALERT-S1_T15QUU_20240125T002334Z_20250904T212246Z_S1_30_v0.1.png'],
 'credit_cost': 1,
 'logs': [],
 'processing_times': [723.955]}

The product looks like a bunch of tif files in a directory see this: https://opera-adt.github.io/dist-s1/stable/pages/product_documentation/product_documentation/

My plugin is working as expected uploading files to s3, but they just don't show up as files. Here is one I just created.

Thanks for your help: https://hyp3-tibet-jpl-test-contentbucket-hratibh1y9pa.s3.us-west-2.amazonaws.com/fa126dac-873b-41f1-bd53-aea8b96893b2/OPERA_L3_DIST-ALERT-S1_T15QUU_20240125T002334Z_20250904T212246Z_S1_30_v0.1_GEN-METRIC.tif

[cmarshak]

I think hyp3 expects netcdf, geojson or zip files: https://github.com/search?q=repo%3AASFHyP3%2Fhyp3+.nc&type=code

[asjohnston-asf]

Yes, currently only those file extensions are exposed in the files entry of the API response, implementation is at https://github.com/ASFHyP3/hyp3/blob/develop/apps/get-files/src/get_files.py#L39C5-L39C20

We've been interested in changing that behavior, but it impacts a few of our plugins that upload a zipped copy of their output files in addition to their individual output files. Historically we've made the bundled .zip file visible to end users, but not the individual files. We want to make both discoverable, but haven't through through how to best implement it.

[cmarshak]

What would you recommend I do (save from writing simple scripts to list contents of the bucket) to download all the tif files quickly? Should I zip the tif files and download that?

[jhkennedy]

@cmarshak, typically, we upload a zip of all the "product" files as well as the files individually. That way, the HyP3 job response has a link to download everything, and it's easy, but you can also edit that URL to get to the individual files directly.

E.g., if you're job uploads files like:

- PRODCUT_NAME.zip
- PRODUCT_NAME_thing1.tif
- PRODUCT_NAME_thing2.tif
- ...

then you can just do a .replace('.zip', '_thing1.tif') on the URL to download "thing1".

with the SDK that might look like:

for job in jobs:
    if job.succeeded():
        sdk.util.download_file(job.files['url'].replace('.zip', '_thing1.tif'))

[jhkennedy]

@cmarshak looking at your product structure:

<OPERA_ID>/
├── <OPERA_ID>_GEN-DIST-STATUS.tif
├── <OPERA_ID>_GEN-METRIC.tif
├── <OPERA_ID>_GEN-DIST-STATUS-ACQ.tif
├── <OPERA_ID>_GEN-METRIC-MAX.tif
├── <OPERA_ID>_GEN-DIST-CONF.tif
├── <OPERA_ID>_GEN-DIST-DATE.tif
├── <OPERA_ID>_GEN-DIST-COUNT.tif
├── <OPERA_ID>_GEN-DIST-PERC.tif
├── <OPERA_ID>_GEN-DIST-DUR.tif
└── <OPERA_ID>_GEN-DIST-LAST-DATE.tif

If you zip and upload the <OPERA_ID> folder as <OPERA_ID>.zip, you'd be able to easily follow our undocumented norm.

This does double the storage cost, but that tradeoff has typically been worth it to enable easy downloads of everything.

All that said, we should do a better job of reporting all the files a job uploads.