Skip to content

fix(sdk): skip evaluation when no controls apply#124

Open
lan17 wants to merge 3 commits intomainfrom
codex/fix-issue-123-repro-test
Open

fix(sdk): skip evaluation when no controls apply#124
lan17 wants to merge 3 commits intomainfrom
codex/fix-issue-123-repro-test

Conversation

@lan17
Copy link
Contributor

@lan17 lan17 commented Mar 13, 2026

Summary

  • Make the SDK evaluation path a true no-op when no enabled control applies to the current step/stage.
  • Prefilter both local (execution="sdk") and server (execution="server") controls using the same applicability rules before any evaluation work starts.
  • Preserve the existing fail-open behavior for malformed server controls by still deferring those to the server.
  • Add regression coverage for disabled and non-matching controls across both local and server execution.

Scope

  • User-facing/API changes:
    • If no control applies to a step invocation, the SDK does nothing: no local control engine execution and no POST /api/v1/evaluation request.
  • Internal changes:
    • Reuse the SDK control engine's applicability logic to determine whether local or server evaluation should run at all.
    • Extend local-evaluation tests to cover disabled, stage-mismatched, step-type-mismatched, step-name-mismatched, and regex-mismatched controls for both execution modes.
  • Out of scope:
    • Server-side evaluation behavior when applicable controls do exist.
    • Broader control loading or caching changes.

Risk and Rollout

  • Risk level: low
  • Rollback plan:
    • Revert this PR to restore the previous behavior where the SDK entered evaluation as long as local or server controls existed for the agent.

Testing

  • Added or updated automated tests
  • Ran make check (or explained why not)
    • Ran make test, plus targeted ruff and mypy on the touched SDK files.
  • Manually verified behavior

Checklist

@lan17 lan17 changed the title fix: skip server evaluation when no controls apply fix(sdk): skip server evaluation when no controls apply Mar 13, 2026
@lan17 lan17 changed the title fix(sdk): skip server evaluation when no controls apply fix(sdk): skip evaluation when no controls apply Mar 13, 2026
@codecov
Copy link

codecov bot commented Mar 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

"""
server_controls: list[_ControlAdapter] = []

for control in controls:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for control in controls:
        control_data = control.get("control")
        if not isinstance(control_data, dict):
            _logger.warning(f"Malformed control payload: {control}")
            return True

        if control_data.get("execution") != "server":
            continue

        try:
            control_def = ControlDefinition.model_validate(control_data)
            server_controls.append(
                _ControlAdapter(
                    id=control["id"],
                    name=control["name"],
                    control=control_def,
                )
            )
        except Exception:
            _logger.warning(f"Failed to parse server control: {control}")
            return True

    if not server_controls:
        return False

    return bool(_get_applicable_controls(server_controls, request, context="server"))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: Adding server check, to ensure we are looking at server controls. If the intention is to look at both sdk as well as server controls, then we should rename this function appropriately.

if control_data.get("execution") != "server":
            continue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or atleast call out the details in the DocString

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. The helper only ever receives the server subset after the execution partitioning in check_evaluation_with_local(), so I did not want to add a second execution == "server" check inside it and make the contract even less explicit.\n\nI pushed a small follow-up in 91f4596 that makes that assumption clearer instead: renamed the helper to _has_applicable_prefiltered_server_controls(...), renamed the input to server_control_payloads, and expanded the docstring to say that the caller has already partitioned by execution before calling it.\n\nSo behavior is unchanged, but the boundary should be much less surprising now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip evaluation entirely when no controls apply to the current step

2 participants