From 4f3ec4bc7b20f461ac4258f4a3166c59c3bf5fb2 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Thu, 12 Mar 2026 15:47:00 -0700 Subject: [PATCH 1/4] docs: update condition tree controls --- components/models.mdx | 18 ++-- concepts/controls.mdx | 138 +++++++++++++++++++++-------- concepts/overview.mdx | 16 ++-- core/quickstart.mdx | 12 ++- core/reference.mdx | 3 +- examples/agent-control-demo.mdx | 13 +-- examples/aws-strands.mdx | 10 ++- examples/deepeval.mdx | 27 +++--- how-to/decorate-llm-tool-calls.mdx | 30 ++++--- integrations/crewai.mdx | 10 ++- integrations/langchain.mdx | 19 ++-- 11 files changed, 195 insertions(+), 101 deletions(-) diff --git a/components/models.mdx b/components/models.mdx index 68f5301..eddd2b7 100644 --- a/components/models.mdx +++ b/components/models.mdx @@ -52,7 +52,14 @@ step = Step( ### Control Models ```python -from agent_control_models import ControlDefinition, ControlScope, ControlAction +from agent_control_models import ( + ConditionNode, + ControlAction, + ControlDefinition, + ControlScope, + ControlSelector, + EvaluatorSpec, +) control = ControlDefinition( description="Block toxic user messages", @@ -62,8 +69,10 @@ control = ControlDefinition( step_types=["llm"], stages=["pre"], ), - selector={"path": "input"}, - evaluator={"name": "regex", "config": {"pattern": "toxic"}}, + condition=ConditionNode( + selector=ControlSelector(path="input"), + evaluator=EvaluatorSpec(name="regex", config={"pattern": "toxic"}), + ), action=ControlAction(decision="deny"), ) ``` @@ -135,8 +144,7 @@ Complete control specification. - `enabled` (bool): Whether control is active - `execution` (str): Execution mode (`server` or `sdk`) - `scope` (ControlScope): When to apply the control -- `selector` (ControlSelector): What data to evaluate -- `evaluator` (EvaluatorSpec): How to evaluate +- `condition` (ConditionNode): Recursive condition tree; leaf nodes contain `selector` + `evaluator` - `action` (ControlAction): What to do on match - `tags` (List[str]): Tags for categorization diff --git a/concepts/controls.mdx b/concepts/controls.mdx index 908902d..6417dcb 100644 --- a/concepts/controls.mdx +++ b/concepts/controls.mdx @@ -7,7 +7,7 @@ description: Understand how controls work and how to define them. A Control is a protection rule that evaluates agent interactions (inputs/outputs) and takes action based on configured criteria. It defines when to check, what to check, how to evaluate it, and what to do with the results. -Control formula: **Control = Scope (When) + Selector (What) + Evaluator (How) + Action (Decision)** +Control formula: **Control = Scope (When) + Condition (What + How) + Action (Decision)** ## 1. Scope (When to Check) @@ -75,9 +75,66 @@ Fields: --- -## 2. Selector (What to Check) +## 2. Condition (What and How to Check) -The **Selector** specifies which portion of the step's data to extract and pass to the evaluator for analysis. It uses a path specification to navigate the step object. +The **Condition** is a recursive boolean tree. Leaf conditions pair a `selector` with an `evaluator`, and composite conditions can combine child conditions with `and`, `or`, and `not`. + +### Example 1: Leaf condition that checks tool output for PII + +```json +{ + "condition": { + "selector": { + "path": "output" + }, + "evaluator": { + "name": "regex", + "config": { + "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" + } + } + } +} +``` + +### Example 2: Composite condition with `and` and `not` + +```json +{ + "condition": { + "and": [ + { + "selector": { + "path": "context.risk_level" + }, + "evaluator": { + "name": "list", + "config": { + "values": ["high", "critical"] + } + } + }, + { + "not": { + "selector": { + "path": "context.user_role" + }, + "evaluator": { + "name": "list", + "config": { + "values": ["admin", "security"] + } + } + } + } + ] + } +} +``` + +### 2.1 Selector (What to Check Inside a Leaf) + +Inside a leaf condition, the **Selector** specifies which portion of the step's data to extract and pass to the evaluator for analysis. It uses a path specification to navigate the step object. Field: @@ -141,9 +198,9 @@ Common Paths: --- -## 3. Evaluator (How to Check) +### 2.2 Evaluator (How to Check Inside a Leaf) -The **Evaluator** receives the data extracted by the selector and evaluates it against configured rules, returning whether the data matches specified criteria. +Inside a leaf condition, the **Evaluator** receives the data extracted by the selector and evaluates it against configured rules, returning whether the data matches specified criteria. Components: @@ -217,7 +274,7 @@ Agent Control supports custom evaluators for domain-specific requirements. See [ --- -## 4. Action (What to Do) +## 3. Action (What to Do) The **Action** defines what happens when the evaluator matches/detects an issue. @@ -313,13 +370,15 @@ Putting it all together - a control that blocks Social Security Numbers in tool "step_types": ["tool"], "stages": ["post"] }, - "selector": { - "path": "output" - }, - "evaluator": { - "name": "regex", - "config": { - "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" + "condition": { + "selector": { + "path": "output" + }, + "evaluator": { + "name": "regex", + "config": { + "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" + } } }, "action": { @@ -354,10 +413,12 @@ async with AgentControlClient() as client: "enabled": True, "execution": "server", "scope": {"step_names": ["generate_response"], "stages": ["post"]}, - "selector": {"path": "output"}, - "evaluator": { - "name": "regex", - "config": {"pattern": r"\\b\\d{3}-\\d{2}-\\d{4}\\b"} + "condition": { + "selector": {"path": "output"}, + "evaluator": { + "name": "regex", + "config": {"pattern": r"\\b\\d{3}-\\d{2}-\\d{4}\\b"} + }, }, "action": {"decision": "deny"} } @@ -384,10 +445,12 @@ curl -X PUT "http://localhost:8000/api/v1/controls/$CONTROL_ID/data" \ "enabled": true, "execution": "server", "scope": {"step_names": ["generate_response"], "stages": ["post"]}, - "selector": {"path": "output"}, - "evaluator": { - "name": "regex", - "config": {"pattern": "\\\\b\\\\d{3}-\\\\d{2}-\\\\d{4}\\\\b"} + "condition": { + "selector": {"path": "output"}, + "evaluator": { + "name": "regex", + "config": {"pattern": "\\\\b\\\\d{3}-\\\\d{2}-\\\\d{4}\\\\b"} + } }, "action": {"decision": "deny"} } @@ -413,13 +476,15 @@ await controls.create_control( "enabled": True, "execution": "server", "scope": {"step_names": ["process_user_message"], "step_types": ["llm"], "stages": ["pre"]}, - "selector": {"path": "input"}, - "evaluator": { - "name": "galileo.luna2", - "config": { - "metric": "input_toxicity", - "operator": "gt", - "target_value": 0.5 + "condition": { + "selector": {"path": "input"}, + "evaluator": { + "name": "galileo.luna2", + "config": { + "metric": "input_toxicity", + "operator": "gt", + "target_value": 0.5 + } } }, "action": {"decision": "deny"} @@ -445,13 +510,15 @@ curl -X PUT "http://localhost:8000/api/v1/controls/$CONTROL_ID/data" \ "enabled": true, "execution": "server", "scope": {"step_names": ["process_user_message"], "step_types": ["llm"], "stages": ["pre"]}, - "selector": {"path": "input"}, - "evaluator": { - "name": "galileo.luna2", - "config": { - "metric": "input_toxicity", - "operator": "gt", - "target_value": 0.5 + "condition": { + "selector": {"path": "input"}, + "evaluator": { + "name": "galileo.luna2", + "config": { + "metric": "input_toxicity", + "operator": "gt", + "target_value": 0.5 + } } }, "action": {"decision": "deny"} @@ -460,4 +527,3 @@ curl -X PUT "http://localhost:8000/api/v1/controls/$CONTROL_ID/data" \ ``` > **Note**: For the Luna-2 evaluator, set the `GALILEO_API_KEY` environment variable. See the [Evaluators](/concepts/evaluators/overview) for all available evaluators. - diff --git a/concepts/overview.mdx b/concepts/overview.mdx index 0286262..16ccc31 100644 --- a/concepts/overview.mdx +++ b/concepts/overview.mdx @@ -1,6 +1,6 @@ --- title: Overview -description: Core concepts behind Agent Control — policies, controls, selectors, evaluators, and actions. +description: Core concepts behind Agent Control — policies, controls, conditions, selectors, evaluators, and actions. --- Understanding these core concepts will help you get the most out of Agent Control. Start with the high-level [architecture](/concepts/architecture) to see how components fit together, then dive into [evaluators](/concepts/evaluators/overview) to understand how checks are implemented. @@ -10,7 +10,7 @@ Understanding these core concepts will help you get the most out of Agent Contro A **[Control](/concepts/controls)** is a single rule that defines what to check and what to do when a condition is met. ```text -Control = Scope + Selector + Evaluator + Action +Control = Scope + Condition + Action ``` Example: "If the output contains an SSN pattern, block the response." @@ -20,15 +20,19 @@ Example: "If the output contains an SSN pattern, block the response." "name": "block-ssn-in-output", "execution": "server", "scope": { "step_types": ["llm"], "stages": ["post"] }, - "selector": { "path": "output" }, - "evaluator": { - "name": "regex", - "config": { "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" } + "condition": { + "selector": { "path": "output" }, + "evaluator": { + "name": "regex", + "config": { "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" } + } }, "action": { "decision": "deny" } } ``` +Leaf conditions pair a `selector` with an `evaluator`. Composite conditions can use `and`, `or`, and `not` to combine multiple leaf checks. + ## Scope **Scope** defines when a control runs by filtering which steps are evaluated. diff --git a/core/quickstart.mdx b/core/quickstart.mdx index 998d7ee..af763ff 100644 --- a/core/quickstart.mdx +++ b/core/quickstart.mdx @@ -160,10 +160,12 @@ async def setup(): "enabled": True, "execution": "server", "scope": {"stages": ["post"]}, - "selector": {"path": "output"}, - "evaluator": { - "name": "regex", - "config": {"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + "condition": { + "selector": {"path": "output"}, + "evaluator": { + "name": "regex", + "config": {"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + }, }, "action": {"decision": "deny"}, }, @@ -182,6 +184,8 @@ async def setup(): asyncio.run(setup()) ``` +Controls store leaf `selector` and `evaluator` definitions under `condition`, which also enables composite `and`, `or`, and `not` trees. + Now, run your agent code. **🎉 Done!** Your agent now blocks SSN patterns automatically. diff --git a/core/reference.mdx b/core/reference.mdx index 96519ec..8fea6f1 100644 --- a/core/reference.mdx +++ b/core/reference.mdx @@ -37,7 +37,7 @@ uv run mypy . # Type-check A **Control** is a single rule that defines what to check and what to do when a condition is met. ```text -Control = Scope + Selector + Evaluator + Action +Control = Scope + Condition + Action ``` **Control associations** are direct links between controls and agents. @@ -286,4 +286,3 @@ make alembic-upgrade - [Models](/components/models) - [Evaluators](/components/evaluators) - [UI Dashboard](/components/ui) - diff --git a/examples/agent-control-demo.mdx b/examples/agent-control-demo.mdx index 2bbb6e1..54c2c1a 100644 --- a/examples/agent-control-demo.mdx +++ b/examples/agent-control-demo.mdx @@ -103,8 +103,7 @@ Key insight: controls are enforced server-side, so you can update rules in real Controls are defined on the server with: - Scope: When to check (step types, stages: pre or post) -- Selector: What to check (input, output, specific fields) -- Evaluator: How to check (regex patterns, list matching, AI-based) +- Condition: What and how to check - Action: What to do (allow, deny, steer, warn, log) Example from `setup_controls.py`: @@ -116,10 +115,12 @@ control_data = ControlDefinition( enabled=True, execution="server", scope=ControlScope(step_types=["tool"], stages=["post"]), - selector=ControlSelector(path="output"), - evaluator=EvaluatorConfig( - name="regex", - config={"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + condition=ConditionNode( + selector=ControlSelector(path="output"), + evaluator=EvaluatorSpec( + name="regex", + config={"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + ), ), action=ControlAction(decision="deny"), ) diff --git a/examples/aws-strands.mdx b/examples/aws-strands.mdx index 9c2c876..b847e27 100644 --- a/examples/aws-strands.mdx +++ b/examples/aws-strands.mdx @@ -75,7 +75,10 @@ Apply to all model interactions: ```python { "scope": {"step_types": ["llm"], "stages": ["pre"]}, - "evaluator": {"name": "regex", "config": {"pattern": r"\d{3}-\d{2}-\d{4}"}}, + "condition": { + "selector": {"path": "input"}, + "evaluator": {"name": "regex", "config": {"pattern": r"\d{3}-\d{2}-\d{4}"}}, + }, "action": {"decision": "deny"} } ``` @@ -91,7 +94,10 @@ Target specific tools: "step_names": ["lookup_order"], "stages": ["pre"] }, - "evaluator": {"name": "regex", "config": {"pattern": r"ORD-\d+"}}, + "condition": { + "selector": {"path": "input"}, + "evaluator": {"name": "regex", "config": {"pattern": r"ORD-\d+"}}, + }, "action": {"decision": "deny"} } ``` diff --git a/examples/deepeval.mdx b/examples/deepeval.mdx index 37b11a7..8bbcfb9 100644 --- a/examples/deepeval.mdx +++ b/examples/deepeval.mdx @@ -159,21 +159,20 @@ control_definition = { "enabled": True, "execution": "server", "scope": {"stages": ["post"]}, # Apply to all steps at post stage - "selector": {}, # Pass full data (input + output) - "evaluator": { - "name": "deepeval-geval", # From metadata.name - "config": { - "name": "Coherence", - "criteria": "Evaluate whether the response is coherent", - "evaluation_params": ["input", "actual_output"], - "threshold": 0.6, - "model": "gpt-4o", + "condition": { + "selector": {"path": "*"}, # Pass full data (input + output) + "evaluator": { + "name": "deepeval-geval", # From metadata.name + "config": { + "name": "Coherence", + "criteria": "Evaluate whether the response is coherent", + "evaluation_params": ["input", "actual_output"], + "threshold": 0.6, + "model": "gpt-4o", + }, }, }, - "action": { - "decision": "deny", - "message": "Response failed coherence check", - }, + "action": {"decision": "deny"}, }, } ``` @@ -182,7 +181,7 @@ control_definition = { - `execution: "server"` - Required field - `scope: {"stages": ["post"]}` - Apply to all function calls at post stage -- `selector: {}` - Pass full data so evaluator gets both input and output +- `condition.selector.path: "*"` - Pass full data so evaluator gets both input and output - `evaluation_params: ["input", "actual_output"]` - Both fields required for relevance checks ## Getting Started from Fresh Clone diff --git a/how-to/decorate-llm-tool-calls.mdx b/how-to/decorate-llm-tool-calls.mdx index 76b5e43..133723a 100644 --- a/how-to/decorate-llm-tool-calls.mdx +++ b/how-to/decorate-llm-tool-calls.mdx @@ -76,10 +76,12 @@ async def main(): "enabled": True, "execution": "server", "scope": {"step_types": ["llm"], "stages": ["post"]}, - "selector": {"path": "output"}, - "evaluator": { - "name": "regex", - "config": {"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + "condition": { + "selector": {"path": "output"}, + "evaluator": { + "name": "regex", + "config": {"pattern": r"\b\d{3}-\d{2}-\d{4}\b"}, + }, }, "action": {"decision": "deny"}, }) @@ -88,15 +90,17 @@ async def main(): "enabled": True, "execution": "server", "scope": {"step_types": ["llm"], "stages": ["pre"]}, - "selector": {"path": "input"}, - "evaluator": { - "name": "list", - "config": { - "values": ["DROP", "DELETE", "TRUNCATE"], - "logic": "any", - "match_on": "match", - "match_mode": "contains", - "case_sensitive": False, + "condition": { + "selector": {"path": "input"}, + "evaluator": { + "name": "list", + "config": { + "values": ["DROP", "DELETE", "TRUNCATE"], + "logic": "any", + "match_on": "match", + "match_mode": "contains", + "case_sensitive": False, + }, }, }, "action": {"decision": "deny"}, diff --git a/integrations/crewai.mdx b/integrations/crewai.mdx index 1a1b7e7..c99928e 100644 --- a/integrations/crewai.mdx +++ b/integrations/crewai.mdx @@ -104,10 +104,12 @@ control_data = { "step_names": ["handle_ticket"], "stages": ["pre"], }, - "selector": {"path": "input.ticket"}, - "evaluator": { - "name": "regex", - "config": {"pattern": r"(?i)(admin|password|other user)"}, + "condition": { + "selector": {"path": "input.ticket"}, + "evaluator": { + "name": "regex", + "config": {"pattern": r"(?i)(admin|password|other user)"}, + }, }, "action": {"decision": "deny"}, } diff --git a/integrations/langchain.mdx b/integrations/langchain.mdx index 870a31e..aed166d 100644 --- a/integrations/langchain.mdx +++ b/integrations/langchain.mdx @@ -100,14 +100,16 @@ control_data = { "step_names": ["sql_db_query"], "stages": ["pre"], }, - "selector": {"path": "input.query"}, - "evaluator": { - "name": "sql", - "config": { - "blocked_operations": ["DROP", "DELETE", "TRUNCATE"], - "allow_multi_statements": False, - "require_limit": True, - "max_limit": 100, + "condition": { + "selector": {"path": "input.query"}, + "evaluator": { + "name": "sql", + "config": { + "blocked_operations": ["DROP", "DELETE", "TRUNCATE"], + "allow_multi_statements": False, + "require_limit": True, + "max_limit": 100, + }, }, }, "action": {"decision": "deny"}, @@ -128,4 +130,3 @@ See the full implementation and setup scripts in the example: ## Resources - [`LangChain SQL Example Docs`](/examples/langchain-sql) - From e09fda82785b5c590c8afc4980d9a6eab3265f52 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Thu, 12 Mar 2026 15:47:46 -0700 Subject: [PATCH 2/4] build: fix docs validation script --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 4da7756..1907a7a 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "private": true, "scripts": { "dev": "mintlify dev", - "build": "mintlify build", + "build": "mintlify validate", "broken-links": "mintlify broken-links" }, "dependencies": { From 6be287aa991a265d8bd7ee398fc99cf75429c0d7 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Thu, 12 Mar 2026 16:28:29 -0700 Subject: [PATCH 3/4] docs: clarify condition submodels --- components/models.mdx | 22 ++++++++++++++++++++++ concepts/controls.mdx | 4 +++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/components/models.mdx b/components/models.mdx index eddd2b7..d0f5a0d 100644 --- a/components/models.mdx +++ b/components/models.mdx @@ -148,6 +148,28 @@ Complete control specification. - `action` (ControlAction): What to do on match - `tags` (List[str]): Tags for categorization +### ConditionNode + +Recursive condition tree used by `ControlDefinition.condition`. + +- Leaf nodes contain both `selector` and `evaluator` +- Composite nodes contain exactly one of `and`, `or`, or `not` +- Maximum nesting depth is 6 + +### ControlSelector + +Selector used inside a leaf condition. + +- `path` (Optional[str]): Dot-path for the value to evaluate; defaults to `"*"` +- Common paths: `input`, `output`, `input.query`, `context.user_id`, `name`, `*` + +### EvaluatorSpec + +Evaluator used inside a leaf condition. + +- `name` (str): Evaluator name, such as `regex`, `list`, `sql`, or `galileo.luna2` +- `config` (Dict[str, Any]): Evaluator-specific configuration payload + ### EvaluationRequest Request for evaluating controls. diff --git a/concepts/controls.mdx b/concepts/controls.mdx index 6417dcb..32522b6 100644 --- a/concepts/controls.mdx +++ b/concepts/controls.mdx @@ -417,7 +417,7 @@ async with AgentControlClient() as client: "selector": {"path": "output"}, "evaluator": { "name": "regex", - "config": {"pattern": r"\\b\\d{3}-\\d{2}-\\d{4}\\b"} + "config": {"pattern": r"\b\d{3}-\d{2}-\d{4}\b"} }, }, "action": {"decision": "deny"} @@ -457,6 +457,8 @@ curl -X PUT "http://localhost:8000/api/v1/controls/$CONTROL_ID/data" \ }' ``` +Regex pattern note: the pattern itself is `\b\d{3}-\d{2}-\d{4}\b`. Python raw strings render that as `r"\b\d{3}-\d{2}-\d{4}\b"`, while JSON payloads must escape backslashes as `"\\b\\d{3}-\\d{2}-\\d{4}\\b"`. + ### Example: Block Toxic Input (Luna-2 AI) To use this evaluator, install the package and restart the server. From 06be2f77350dd390a6eaed29e8ba258d40f88a4f Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Thu, 12 Mar 2026 16:35:38 -0700 Subject: [PATCH 4/4] docs: fix regex escaping in curl example --- concepts/controls.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/concepts/controls.mdx b/concepts/controls.mdx index 32522b6..b46e3b7 100644 --- a/concepts/controls.mdx +++ b/concepts/controls.mdx @@ -449,7 +449,7 @@ curl -X PUT "http://localhost:8000/api/v1/controls/$CONTROL_ID/data" \ "selector": {"path": "output"}, "evaluator": { "name": "regex", - "config": {"pattern": "\\\\b\\\\d{3}-\\\\d{2}-\\\\d{4}\\\\b"} + "config": {"pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b"} } }, "action": {"decision": "deny"}