Skip to content

[wip] add raw data path to apps#740

Open
cosmicBboy wants to merge 2 commits intomainfrom
nielsb/add-raw-data-path-to-apps
Open

[wip] add raw data path to apps#740
cosmicBboy wants to merge 2 commits intomainfrom
nielsb/add-raw-data-path-to-apps

Conversation

@cosmicBboy
Copy link
Contributor

@cosmicBboy cosmicBboy commented Mar 2, 2026

This pull request introduces support for configuring and accessing the raw data path in Flyte apps, enabling applications to read the raw data path from the app context. The changes allow the raw data path to be set via command-line options, environment variables, or programmatically, and ensure it is correctly propagated in local and remote serving scenarios. An example FastAPI app is also added to demonstrate this new feature.

Raw Data Path Support

  • Added a new raw_data_path attribute to the AppContext dataclass and exposed it via the ctx() function, allowing apps to access the raw data path. Also implemented the set_raw_data_path() function to set this value and propagate it via environment variables for thread safety. (src/flyte/app/_context.py)
  • Updated the serving context (_Serve) and its factory function (with_servecontext) to accept and propagate the raw_data_path parameter, defaulting to /tmp/flyte/raw_data for local mode. The serving logic ensures the raw data path is set in the correct thread context for request handlers. (src/flyte/_serve.py)* Extended the CLI (serve.py) to accept a --raw-data-path option and set the raw data path in the app context when provided. (src/flyte/_bin/serve.py)
  • Ensured that the container command for remote serving includes a template variable for the raw data path, allowing the backend to substitute the correct value at runtime. (src/flyte/app/_app_environment.py)

Documentation and Example

  • Added an example FastAPI app (app_with_raw_data_path.py) that demonstrates accessing the raw data path from the app context and returning it in an API response. (examples/apps/app_with_raw_data_path.py)

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
@cosmicBboy cosmicBboy changed the title add raw data path to apps [wip] add raw data path to apps Mar 3, 2026
if raw_data_path:
from flyte.app._context import set_raw_data_path

set_raw_data_path(raw_data_path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of updating the env var in set_raw_data_path, would it be better to just pass raw_data_path to _serve()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants