-
Notifications
You must be signed in to change notification settings - Fork 64
Open
Description
To better categorize classes of eval, I'd love to have a method that provides the same structure offered by describe/it in test frameworks.
I'm personally partial to those two method names — they're familiar and declarative, but I understand if you have an another API in mind.
At the moment, evals have a flat structure within one eval file.
I'd love to be able to have orchistrator.ts be associated with orchistrator.eval.ts, within which, I can organize across categories of eval:
describe("Subagent routing", () => {
it("correctly routes to the knowledge search subagent", () => { ... });
it("correctly routes to the plan creation subagent", () => { ... });
});
describe("Policy enforcement", () => {
it("formats output as valid markdown", () => { ... });
it("refuses to disclose prompt", () => { ... });
});Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels