run python script supports debugging, code bundling, output dir#789
run python script supports debugging, code bundling, output dir#789cosmicBboy wants to merge 6 commits intomainfrom
Conversation
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
| @dataclass | ||
| class PythonScriptOutput: | ||
| exit_code: int | ||
| stdout: str |
There was a problem hiding this comment.
I dont think we should capture the stdout, this is already in the logs. This can be a lot!
There was a problem hiding this comment.
note here that I'm only getting the last 1000 characters: https://github.com/flyteorg/flyte-sdk/pull/789/changes#diff-bc180013a121c671af8420075855a9a12efdcd6a01b24e90c07a6322a09001ceR75-R76
This is still useful for local debugging... user (or agent) gets stderr and stdout in the CLI, we have to think LLM-first!
| class PythonScriptOutput: | ||
| exit_code: int | ||
| stdout: str | ||
| stderr: str |
There was a problem hiding this comment.
note here that I'm only getting the last 1000 characters: https://github.com/flyteorg/flyte-sdk/pull/789/changes#diff-bc180013a121c671af8420075855a9a12efdcd6a01b24e90c07a6322a09001ceR75-R76
This is still useful for local debugging... user (or agent) gets stderr and stdout in the CLI, we have to think LLM-first!
| __all__ = [ | ||
| "CopyFiles", | ||
| "build_code_bundle", | ||
| "build_code_bundle_from_relative_paths", |
There was a problem hiding this comment.
is this an existing function, is it used for apps?
| initialize_config = common.initialize_config | ||
|
|
||
|
|
||
| @syncify |
There was a problem hiding this comment.
needed because there are two call sites. one in the run python script path (called synchronously), the other is in _run.py:
flyte-sdk/src/flyte/cli/_run.py
Line 334 in a9acdb4
flyte-sdk/src/flyte/cli/_run.py
Line 557 in a9acdb4
Where it's called async
|
I think we should also add support for additional files regex (as a follow up us ok), so i like the include files thing |
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
yep, gonna do that in a separate PR |
2 participants
This pull request introduces support for running plain Python scripts as Flyte tasks without pickling, by bundling the script and using a new
ScriptTaskResolver. It also adds a new copy style (python_script) for code bundles, updates the task and environment APIs to support custom resolvers, and improves the developer experience for remote script execution and debugging. Additional minor changes update dependencies and resource specifications in example files.Python script task support and code bundling:
ScriptTaskResolver(src/flyte/_internal/resolvers/script.py) to enable bundling and execution of plain Python scripts as Flyte tasks, avoiding pickling and enabling proper remote debugging.run_python_script(src/flyte/_run_python_script.py) to use the new resolver, bundle scripts with the newpython_scriptcopy style, and support VS Code debugging. [1] [2] [3] [4]build_code_bundle_from_relative_pathsand added"python_script"to theCopyFilesliteral for selective script bundling. [1] [2]_Runnerand remote/hybrid run logic to support the new copy style and script bundling parameters. [1] [2] [3] [4] [5]Task and environment API enhancements:
task_resolverargument toTaskEnvironment.taskandAsyncFunctionTaskTemplate, and ensured it is used during serialization and container argument construction. [1] [2] [3] [4] [5]Developer experience and examples:
hello.pyscript example demonstrating running a plain Python script with Flyte.