Skip to content

Add describe/it or other ways to better organize evals within a file #364

@matchai

Description

@matchai

To better categorize classes of eval, I'd love to have a method that provides the same structure offered by describe/it in test frameworks.

I'm personally partial to those two method names — they're familiar and declarative, but I understand if you have an another API in mind.

At the moment, evals have a flat structure within one eval file.

I'd love to be able to have orchistrator.ts be associated with orchistrator.eval.ts, within which, I can organize across categories of eval:

describe("Subagent routing", () => {
  it("correctly routes to the knowledge search subagent", () => { ... });
  it("correctly routes to the plan creation subagent", () => { ... });
});

describe("Policy enforcement", () => {
  it("formats output as valid markdown", () => { ... });
  it("refuses to disclose prompt", () => { ... });
});

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions