-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Summary
Extend agent-scenario testing with a scenario that exercises the iterate workflow using real requests, events, and attempts in Hookdeck. Currently receive-webhooks uses a documentation-only Stage 04: the agent documents how to list request → event → attempt and retry; no live traffic is required.
Goal
A scenario (or variant) where:
- Traffic exists: After the agent sets up the app and
hookdeck listen, at least one request is sent to the Source URL (e.g. viacurlin the prompt or via test harness) so Hookdeck has real requests, events, and attempts. - Agent runs CLI: The agent runs
hookdeck gateway request list,event list,attempt list, andevent retry(or equivalent) against that live data. - Assessment: The assessor can optionally check that the run log contains actual command output (list/retry) rather than only written documentation.
Considerations
- Credentials / project: Requires a real Hookdeck project and connection in the test environment (e.g. secrets for CI).
- Failure for retry: To test retry meaningfully, there may need to be a failed attempt (e.g. temporarily wrong destination URL or down endpoint).
- Flakiness: Live API and traffic add variability; may be better as a separate "integration" or "live" scenario run on-demand or weekly rather than on every PR.
References
- receive-webhooks Stage 04: documentation-only iterate stage in
scenarios.yaml,tools/agent-scenario-tester/src/assess.ts,TESTING.md. - Phase F content:
skills/event-gateway/references/04-iterate.md,cli-workflows.md,monitoring-debugging.md.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels