Update eager: make sure client secret can be specified as env var

[cosmicBboy]

Why are the changes needed?

Currently, the @eager workflows are too strict with its handling of the client secret values. Both secret group and key need to be specified for the secret request to be passed down to the underlying task. There is also an issue with the async event loop not being available for the downstream data persistence step in the task execution.

What changes were proposed in this pull request?

This PR loosens this requirement to be compatible with Flyte backends that do not require secret groups. This PR also:

• Allows the user to bind the client secret to an environment variable in cases where the FlyteRemote object can authenticate with just the client secret
• Makes sure that an async loop is available for downstream data persistence step in the task execution

How was this patch tested?

The following workflow was tested on flyte sandbox

from flytekit import task, ImageSpec
from flytekit.configuration import Config
from flytekit.experimental import eager
from flytekit.remote import FlyteRemote

flytekit = "git+https://github.com/flyteorg/flytekit@54c4471a56311ff8201ba76d9bda726bc5b4689b"

image = ImageSpec(
    name="polars",
    packages=["flytekit", flytekit],
    registry="localhost:30000",
    apt_packages=["git"],
)


@task
def add_one(x: int) -> int:
    return x + 1


@task
def double(x: int) -> int:
    return x * 2


@eager(
    container_image=image,
    remote=FlyteRemote(
        config=Config.for_sandbox(),
        default_project="flytesnacks",
        default_domain="development",
    ),
)
async def simple_eager_workflow(x: int) -> int:
    out = await add_one(x=x)
    if out < 0:
        return -1
    return await double(x=out)

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

[codecov]

Codecov Report

Attention: Patch coverage is 30.76923% with 9 lines in your changes missing coverage. Please review.

Project coverage is 71.82%. Comparing base (65bf9e7) to head (f0b5d67).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/experimental/eager_function.py	30.76%	9 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2720       +/-   ##
===========================================
+ Coverage   45.47%   71.82%   +26.34%     
===========================================
  Files         193      193               
  Lines       19516    19560       +44     
  Branches     2818     3845     +1027     
===========================================
+ Hits         8875    14049     +5174     
+ Misses      10208     4832     -5376     
- Partials      433      679      +246     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cosmicBboy]

this issue comes up not just with eager. Frameworks like Langchain that have async functionality will cause the event loop to end before the flyte task can run the async data persistence operation.

[eapolinario]

Could we use a separate event loop just for flytekit?

[cosmicBboy]

sounds right, I think someone more well-versed in async can help with that

[Future-Outlier]

thank you for this!
can we have

1. register the task to remote cluster (sandbox or dogfood gcp) to prove it works?
2. add workflow in unit tests.

[cosmicBboy]

thank you for this! can we have

1. register the task to remote cluster (sandbox or dogfood gcp) to prove it works?
2. add workflow in unit tests.

Tests for this exist here and here. The changes made to the PR (adding the new client_secret_env_var argument) and loosening the strictness of secrets only really applies to Union serverless.

The code snippet in the PR description should work on flyte sandbox

[Future-Outlier]

LGTM, thank you

[thomasjpfan]

This assumes that the other FlyteRemote object can be created without a Config and uses an environment variable. Is this okay for OSS?

Also, should this carry over the default_domain and default_project from the original remote obj? Or should the other remote object figure out the domain and project from the execution environment?

[cosmicBboy]

ooo yeah it should probably carry over those args.

admittedly this code path is basically: "tell me you only run on serverless without saying you run on serverless" 🙃

[cosmicBboy]

Basically client_secret_env_var only really applies to serverless (this will fail with Flyte), unless at some point down the road Flyte provides an single API key UX.

[wild-endeavor]

some questions, but this event loop thing... can you explain why it's needed in entrypoint.py but there doesn't seem to be another one for local execution? Won't both need it?

[wild-endeavor]

what if there's nothing in the client_secret_env_var environment variable?

[cosmicBboy]

client_secret_env_var doesn't figure into this try except block, or am I missing something?

[wild-endeavor]

no yeah i know, i mean, what happens if all three (client_secret_env_var, client_secret_group and client_secret_key) are None?

[wild-endeavor]

okay with this for now, but I think we should see if we can't move this to some higher level in the future. the secret here we're talking about is the secret for hitting admin itself right?

[cosmicBboy]

this should be implemented in Secret but we haven't moved on that for a while: #1726

[wild-endeavor]

confused what's happening here. if we're removing the asserts, then that means the group/key might be None. If they're none, won't the secrets_manager.get fail? So then what is the client_secret? Or is there some default in the secrets manager that somehow gets returned?
I don't get why we're setting the environment variable here to the value of client secret.

[cosmicBboy]

this is to make it easy to authenticate with UnionRemote on serverless (via UNION_SERVERLESS_API_KEY env var). Secrets on serverless only needs a secret key (no group).

[wild-endeavor]

just wondering, in the following scenario:

• client_secret_key is None
• client_secret_group is None
  (and this scenario is now valid because the asserts are now removed)
  after line 626 client_secret = secrets_manager.get(client_secret_group, client_secret_key)
  what is the value of client_secret?

[wild-endeavor]

going through the secretsmanager, it looks like it raises an IndexError if you call get(group=None, key=None), so I don't think it ever gets past line 626. Could we move the assertion that at least one of those has be be non-None into the get function?

[wild-endeavor]

Had a few more comments and went over this with @eapolinario. Could we

• Add an assert to secretsmanager.get that at least one of group/key should be non-None
• Could we move the async loop stuff to another PR? I think it'd be helpful to discuss it separately. Also do you remember how you triggered the issue? I wanna repro locally to understand more what langchain is doing. Ideally we handle in such a way that it's completely independent of any other library.

[cosmicBboy]

Had a few more comments and went over this with @eapolinario. Could we

• Add an assert to secretsmanager.get that at least one of group/key should be non-None

Updated PR with client secret key/group assertion.

• Could we move the async loop stuff to another PR? I think it'd be helpful to discuss it separately. Also do you remember how you triggered the issue? I wanna repro locally to understand more what langchain is doing. Ideally we handle in such a way that it's completely independent of any other library.

Deleted this. For some reason I'm unable to repro the async issue I was seeing earlier on eager or with langchain async document loaders https://github.com/unionai-oss/union-rag/blob/main/union_rag/langchain.py#L138-L141