Skip to content

FEAT add TargetRequirements#1582

Open
hannahwestra25 wants to merge 3 commits intomicrosoft:mainfrom
hannahwestra25:hawestra/add_target_requirements
Open

FEAT add TargetRequirements#1582
hannahwestra25 wants to merge 3 commits intomicrosoft:mainfrom
hannahwestra25:hawestra/add_target_requirements

Conversation

@hannahwestra25
Copy link
Copy Markdown
Contributor

Description

Adds TargetRequirements, the consumer-side counterpart to TargetConfiguration. Useful for consumers (attacks, converters, scorers) to validate their requirements against a target's configuration at construction time. TargetRequirements is the mechanism for that — a frozen, declarative description of what capabilities a consumer needs, validated in a single call.

for example, a CrescendoAttack would use requirements :

 class CrescendoAttack:
    REQUIREMENTS = TargetRequirements(
        required_capabilities=frozenset({
            CapabilityName.MULTI_TURN,
            CapabilityName.SYSTEM_PROMPT,
        })
    )


    def __init__(self, *, objective_target: PromptTarget, ...) -> None:
        # Validates all requirements in one call — fails fast with all violations
        self.REQUIREMENTS.validate(configuration=objective_target.configuration)

If the target natively supports the capabilities, validation passes. If not, the TargetConfiguration's CapabilityHandlingPolicy is consulted — if the policy is ADAPT, normalization will handle it; if RAISE (or no policy exists), validate() collects all violations and raises a single ValueError eg:
ValueError: Target does not satisfy 2 required capability(ies):

  • Target does not support 'supports_multi_turn' and the handling policy is RAISE.
  • Target does not support 'supports_system_prompt' and no handling policy exists for it.

Tests and Documentation

Copy link
Copy Markdown
Contributor

@ValbuenaVC ValbuenaVC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two nits, looks good!

assert CapabilityName.SYSTEM_PROMPT in reqs.required_capabilities


def test_frozen_dataclass_is_immutable():
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: isn't this guaranteed by Python itself? Or do you want to just make sure that the attribute is frozen rather than that freezing works



@dataclass(frozen=True)
class TargetRequirements:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'm not sure I understand the point of this dataclass. What does this do that a validate method attached to TargetConfiguration or TargetCapabilities doesn't? Attaching TargetCapabilities to a consumer object is weird but this class seems like it could be reduced to a private attribute for consumers (_REQUIRED_CAPABILITIES: frozenset) while the validation method is moved elsewhere. Non-blocking comment since this works but worth documenting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants