Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Build

on:
push:
branches:
- develop
tags:
- 'v*'
pull_request:
branches:
- main
- develop

jobs:
call-version-info-workflow:
uses: ASFHyP3/actions/.github/workflows/reusable-version-info.yml@v0.20.0
permissions:
contents: read
with:
python_version: '3.10'

call-docker-ghcr-workflow:
needs: call-version-info-workflow
uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.20.0
permissions:
contents: read
packages: write
with:
version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
user: ${{ github.actor }}
secrets:
USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@
jobs:
call-changelog-check-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.20.0

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI 8 months ago

To fix the issue, we will add a permissions block at the root of the workflow file. This block will specify the minimal permissions required for the workflow to function. Based on the context, the workflow appears to check for changelog updates, which likely requires read access to the repository contents. Therefore, we will set contents: read as the permission.


Suggested changeset 1
.github/workflows/changelog.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/changelog.yml b/.github/workflows/changelog.yml
--- a/.github/workflows/changelog.yml
+++ b/.github/workflows/changelog.yml
@@ -2,2 +2,5 @@
 
+permissions:
+  contents: read
+
 on:
EOF
@@ -2,2 +2,5 @@

permissions:
contents: read

on:
Copilot is powered by AI and may make mistakes. Always verify output.
2 changes: 1 addition & 1 deletion .github/workflows/create-jira-issue.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

jobs:
call-create-jira-issue-workflow:
uses: ASFHyP3/actions/.github/workflows/reusable-create-jira-issue.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-create-jira-issue.yml@v0.20.0
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
JIRA_PROJECT: ${{ secrets.JIRA_PROJECT }}
JIRA_FIELDS: ${{ secrets.JIRA_FIELDS }}

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
2 changes: 1 addition & 1 deletion .github/workflows/labeled-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@
jobs:
call-labeled-pr-check-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.20.0

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI 8 months ago

To fix the issue, we need to add a permissions block to the workflow. This block should specify the least privileges required for the workflow to function correctly. Since the workflow interacts with pull requests (e.g., checking labels), it likely requires contents: read and pull-requests: read. If additional permissions are needed, they should be added explicitly based on the functionality of the reusable workflow.

The permissions block should be added at the root level of the workflow, ensuring it applies to all jobs unless overridden by job-specific permissions.


Suggested changeset 1
.github/workflows/labeled-pr.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/labeled-pr.yml b/.github/workflows/labeled-pr.yml
--- a/.github/workflows/labeled-pr.yml
+++ b/.github/workflows/labeled-pr.yml
@@ -2,2 +2,6 @@
 
+permissions:
+  contents: read
+  pull-requests: read
+
 on:
EOF
@@ -2,2 +2,6 @@

permissions:
contents: read
pull-requests: read

on:
Copilot is powered by AI and may make mistakes. Always verify output.
2 changes: 1 addition & 1 deletion .github/workflows/release-checklist-comment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
jobs:
call-release-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/reusable-release-checklist-comment.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-release-checklist-comment.yml@v0.20.0
permissions:
pull-requests: write
secrets:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
jobs:
call-release-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.20.0
with:
release_prefix: HyP3 OPERA-RTC
release_branch: main
develop_branch: develop
sync_pr_label: actions-bot
secrets:
USER_TOKEN: ${{ secrets.TOOLS_BOT_PAK }}

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
6 changes: 3 additions & 3 deletions .github/workflows/static-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@

jobs:
call-secrets-analysis-workflow:
uses: ASFHyP3/actions/.github/workflows/reusable-secrets-analysis.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-secrets-analysis.yml@v0.20.0

call-ruff-workflow:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
uses: ASFHyP3/actions/.github/workflows/reusable-ruff.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-ruff.yml@v0.20.0

call-mypy-workflow:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
uses: ASFHyP3/actions/.github/workflows/reusable-mypy.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-mypy.yml@v0.20.0

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI 8 months ago

To fix the issue, we need to add a permissions block to the root of the workflow file. This block will define the minimum required permissions for the workflow. Since the workflow calls reusable workflows, we should assume the most basic permissions (contents: read) unless the reusable workflows explicitly require additional permissions. If the reusable workflows need more permissions, they should ideally define them internally.

The fix involves:

  1. Adding a permissions block at the root level of the workflow file.
  2. Setting contents: read as the default permission, which is sufficient for most basic CI workflows.

Suggested changeset 1
.github/workflows/static-analysis.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/static-analysis.yml b/.github/workflows/static-analysis.yml
--- a/.github/workflows/static-analysis.yml
+++ b/.github/workflows/static-analysis.yml
@@ -4,2 +4,5 @@
 
+permissions:
+  contents: read
+
 jobs:
EOF
@@ -4,2 +4,5 @@

permissions:
contents: read

jobs:
Copilot is powered by AI and may make mistakes. Always verify output.
2 changes: 1 addition & 1 deletion .github/workflows/tag-version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
jobs:
call-bump-version-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.18.1
uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.20.0
with:
user: tools-bot
email: UAF-asf-apd@alaska.edu
secrets:
USER_TOKEN: ${{ secrets.TOOLS_BOT_PAK }}

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
38 changes: 0 additions & 38 deletions .github/workflows/test-and-build.yml

This file was deleted.

21 changes: 21 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: Test

on:
push:
branches:
- main
- develop
pull_request:
branches:
- main
- develop

jobs:
call-pytest-workflow:
uses: ASFHyP3/actions/.github/workflows/reusable-pytest.yml@v0.20.0
permissions:
contents: read
with:
local_package_name: hyp3_opera_rtc
python_versions: >-
["3.10", "3.11", "3.12", "3.13"]
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.1.3]

### Changed
- Download opera burst db during container build instead of at runtime.
- Publish docker container to public GHCR repo instead of private Amazon ECR repo.

## [0.1.2]

### Changed
Expand Down
4 changes: 3 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM 845172464411.dkr.ecr.us-west-2.amazonaws.com/opera_pge/rtc_s1:2.1.1
FROM ghcr.io/nasa/opera-sds-pge/opera_pge/rtc_s1:2.1.1

# For opencontainers label definitions, see:
# https://github.com/opencontainers/image-spec/blob/master/annotations.md
Expand All @@ -15,6 +15,8 @@ USER root
RUN chown rtc_user:rtc_user /home/rtc_user/scratch
USER rtc_user

RUN curl https://asf-dem-west.s3.amazonaws.com/AUX/opera-burst-bbox-only.sqlite3 -o /home/rtc_user/opera-burst-bbox-only.sqlite3

COPY --chown=rtc_user:rtc_user . /home/rtc_user/hyp3-opera-rtc/
RUN conda env create -f /home/rtc_user/hyp3-opera-rtc/environment.yml && \
conda clean -afy && \
Expand Down
17 changes: 5 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ A HyP3 plugin for OPERA-RTC processing

**ALL CREDIT FOR THIS PLUGIN'S RTC PRODUCTS GOES TO THE [JPL OPERA TEAM](https://www.jpl.nasa.gov/go/opera). THIS PLUGIN MERELY ALLOWS US TO RUN THEIR WORKFLOW IN A HYP3 ENVIRONMENT.**

**VALIDATION OF THIS PLUGIN IS STILL ONGOING. CONFIDENCE THAT THIS PLUGIN PRODUCES EQUIVALENT RTC DATA PRODUCTS AS THE OPERA MISSION IS NOT (YET) HIGH ENOUGH FOR PRODUCTION USE.**

## Earthdata Login Credentials

To use this plugin, the user must provide their Earthdata Login credentials in order to download input data.
Expand All @@ -14,20 +16,11 @@ Your credentials can be passed to the workflows via environmental variables that

## Usage

> [!WARNING]
> This plugin is designed to run within the HyP3 processing system, and directly relies on the JPL OPERA OPERA-RTC-S1 Product Generation Executable (PGE) docker container (see architecture section below). Currently this container is not publicly available, but the OPERA team is working to make it available in the near future.

This plugin is designed to run within the HyP3 processing system,
and directly relies on the JPL OPERA OPERA-RTC-S1 Product Generation Executable (PGE) docker container (see architecture section below).
For this reason, the plugin is only runnable via the docker container.

Because our docker image is based on `845172464411.dkr.ecr.us-west-2.amazonaws.com/opera_pge/rtc_s1` (see the [Dockerfile](./Dockerfile)),
you'll need to authenticate to Amazon ECR in the HyP3 AWS account before building our docker image
(you must be an ASF developer with access to the HyP3 AWS account):

```
aws --profile hyp3 ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 845172464411.dkr.ecr.us-west-2.amazonaws.com
```

Then build our container:
Build the container:

```bash
git clone https://github.com/ASFHyP3/hyp3-OPERA-RTC.git
Expand Down
2 changes: 1 addition & 1 deletion requirements-static.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ruff==0.11.8
ruff==0.11.11
mypy==1.15.0
lxml-stubs
types-shapely
Expand Down
18 changes: 8 additions & 10 deletions src/hyp3_opera_rtc/prep_rtc.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import argparse
import os
import shutil
import warnings
from pathlib import Path
from zipfile import ZipFile
Expand All @@ -18,16 +19,13 @@
CMR_URL = 'https://cmr.earthdata.nasa.gov/search/granules.umm_json'


def download_burst_db(save_dir: Path) -> Path:
db_path = save_dir / 'opera-burst-bbox-only.sqlite3'
def prep_burst_db(save_dir: Path) -> Path:
db_filename = 'opera-burst-bbox-only.sqlite3'
db_path = save_dir / db_filename

if db_path.exists():
return db_path
shutil.copy(Path.home() / db_filename, db_path)

# Currently using a version created using opera-adt/burst_db v0.4.0, but hope to switch to ASF-provide source.
url = 'https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/opera/opera-burst-bbox-only.sqlite3'
db_path = hyp3lib.fetch.download_file(url, str(save_dir))
return Path(db_path)
return db_path


def get_s1_granule_bbox(granule_path: Path, buffer: float = 0.025) -> Polygon:
Expand Down Expand Up @@ -135,8 +133,8 @@ def prep_rtc(
orbit_path = orbit.get_orbit(safe_path.with_suffix('').name, save_dir=input_dir)
print(f'Downloaded orbit file: {orbit_path}')

db_path = download_burst_db(input_dir)
print(f'Downloaded burst database: {db_path}')
db_path = prep_burst_db(input_dir)
print(f'Burst database: {db_path}')

dem_path = input_dir / 'dem.tif'
granule_bbox = get_s1_granule_bbox(safe_path)
Expand Down