Dataset cache purge cronjob

[ponyisi]

Address #1167 , except the cache lifetime is reduced to one day.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.04%. Comparing base (0bfd9f9) to head (30ccd81).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1211      +/-   ##
===========================================
+ Coverage    85.93%   86.04%   +0.11%     
===========================================
  Files           94       95       +1     
  Lines         3235     3261      +26     
  Branches       371      373       +2     
===========================================
+ Hits          2780     2806      +26     
  Misses         380      380              
  Partials        75       75              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR implements a dataset cache purge cronjob that automatically marks datasets as stale after they've exceeded a configurable age threshold (default 24 hours). The implementation adds a new internal API endpoint for dataset lifecycle management and a Kubernetes CronJob to periodically trigger cache cleanup.

Key changes:

• Added new internal API endpoint /servicex/internal/dataset-lifecycle for dataset cache purging
• Introduced Kubernetes CronJob to automatically purge old cached datasets every hour
• Refactored dataset deletion logic to enable reuse between manual and automated operations

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
servicex_app/routes.py	Registers the new DatasetLifecycleOps resource endpoint
servicex_app/resources/internal/dataset_lifecycle_ops.py	New endpoint implementation that purges datasets older than specified age
servicex_app/resources/datasets/get_all.py	Extracts dataset fetching logic into reusable function
servicex_app/resources/datasets/delete_dataset.py	Extracts deletion logic into reusable function
helm/servicex/values.yaml	Adds configuration parameters for dataset lifecycle cronjob
helm/servicex/templates/dataset-lifecycle/cronjob.yaml	Defines Kubernetes CronJob for periodic dataset cache cleanup
docs/deployment/reference.md	Documents new configuration parameters

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The command should be formatted as an array with shell invocation, similar to the data-lifecycle cronjob. Change to use '/bin/sh' with '-c' flag for better error handling and consistency: command: ['/bin/sh', '-c', 'curl --request POST \"http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME\"']

Suggested change

	command:
	command:
	- /bin/sh
	- -c

[Copilot]

The curl command lacks error handling flags. Add '-f' (fail on HTTP errors) and '--silent' or '--show-error' for better error reporting: curl -f --silent --show-error --request POST ... This ensures the job fails appropriately when the API returns an error.

Suggested change

	- curl --request POST "http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME"
	- curl -f --silent --show-error --request POST "http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME"

[Copilot]

Pull Request Overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@ponyisi I've opened a new pull request, #1214, to work on those changes. Once the pull request is ready, I'll request review from you.