A/B Testing

[JasonMoho]

Champion/Challenger A/B Testing Pool

Adds deployment-scoped champion/challenger A/B testing so admins can manage an experiment pool, compare a baseline agent against configured variants, and collect user preference data.

What It Does

When A/B testing is enabled and the pool is valid, eligible turns can trigger two parallel agent runs:

• one arm is always the configured champion
• the other arm is a randomly sampled challenger from the variant pool
• arm positions (Response A vs Response B) are randomized per comparison
• users vote A, B, or Tie
• per-variant win/loss/tie metrics are tracked in Postgres

The current implementation is deployment-wide and server-authoritative:

• experiment configuration is stored under services.chat_app.ab_testing
• runtime state is persisted in Postgres-backed static config, not in the original YAML after deploy
• sampling decisions are made on the server
• unresolved comparisons are tracked per conversation, with max_pending_per_conversation enforced by the backend and UI

Current Configuration Model

Configure A/B testing under:

services:
  chat_app:
    ab_testing:
      enabled: true
      ab_agents_dir: /root/archi/ab_agents
      sample_rate: 0.25
      disclosure_mode: post_vote_reveal
      default_trace_mode: minimal
      max_pending_per_conversation: 1
      target_roles: []
      target_permissions: []
      pool:
        champion: baseline
        variants:
          - label: baseline
            agent_spec: baseline-ab.md
          - label: poet
            agent_spec: poet-ab.md
            provider: openrouter
            model: anthropic/claude-3.5-sonnet
            recursion_limit: 30
          - label: concise
            agent_spec: concise-ab.md
            num_documents_to_retrieve: 3

Important differences from the original version:

• A/B config lives only at services.chat_app.ab_testing
• A/B variants use label plus agent_spec, not name
• agent_spec must be a filename under ab_agents_dir
• A/B agent specs are isolated from normal agents_dir
• incomplete A/B config no longer blocks startup; it starts inactive and surfaces warnings in the admin UI/runtime

New and Updated API Endpoints

Participant and Runtime Endpoints

Method	Endpoint	Description
GET	/api/ab/pool	Returns participant-safe pool info, or full admin payload for admins
GET	/api/ab/decision	Server-side decision for whether the next turn should use A/B
POST	/api/ab/compare	Streams a champion-vs-challenger comparison as NDJSON
POST	/api/ab/preference	Submits the user vote (a, b, tie)
GET	/api/ab/pending	Returns unresolved comparisons for a conversation
GET	/api/ab/metrics	Returns aggregate per-variant metrics

Admin Configuration Endpoints

Method	Endpoint	Description
POST	/api/ab/pool/set	Legacy full-pool save path
POST	/api/ab/pool/settings/set	Saves experiment settings only
POST	/api/ab/pool/variants/set	Saves variants only
POST	/api/ab/pool/disable	Disables A/B testing while retaining saved variant config

Current UI

Admin Management

The original pool editor in the chat settings modal has been replaced by a dedicated admin page:

• data page top-right action button: A/B Testing
• admin route: /admin/ab-testing
• page is admin-only

The admin page includes:

• experiment status badge
• sample rate
• disclosure mode
• default agent-activity visibility
• max pending per conversation
• champion selector
• full variant list with:
  • label
  • A/B-only agent spec
  • provider override
  • model override
  • recursion limit override
  • document retrieval override
• split save actions:
  • Save Configuration
  • Save Variants
• Disable
• add/remove variant actions
• inline creation of new A/B-only agent specs from the agent-spec selector

Participant Experience

For sampled turns:

• both responses stream side-by-side
• the comparison uses neutral Response A / Response B labels
• variant identity is shown according to disclosure_mode
• agent activity visibility is controlled by default_trace_mode
• voting UI appears after the comparison is ready

Pending Comparison Behavior

The implementation now supports queued unresolved comparisons:

• max_pending_per_conversation controls how many unresolved comparisons a conversation may hold
• the UI restores all pending comparisons on reload
• input is blocked only when the configured pending limit is reached

Agent Spec Isolation

A/B testing no longer reuses the normal user agent pool.

• normal user-visible agent specs live in agents_dir
• A/B-only specs live in ab_agents_dir
• A/B-created specs are hidden from normal user agent CRUD
• when ab_agents_dir is configured in source YAML, deployment staging copies those files into the deployment and rewrites runtime config to /root/archi/ab_agents

Admin Access and Auth

The current product path is:

• auth required
• admin page and admin A/B endpoints are RBAC-gated
• temporary basic-auth admin testing overrides were removed

That means:

• basic auth remains identity-only
• real A/B admin access should come from the permanent auth/RBAC path

Database

Current database support includes:

• ab_comparisons
  • stores each comparison
  • variant metadata
  • prompt and response references
  • recorded preference
• ab_variant_metrics
  • aggregate wins
  • losses
  • ties
  • totals

Backend Changes in the Current Version

• src/interfaces/chat_app/app.py
  • dedicated A/B admin page route
  • admin and participant A/B APIs
  • server-side A/B decision logic
  • split settings/variants save paths
  • queued pending-comparison handling
  • runtime config refresh from persisted static config
• src/utils/ab_testing.py
  • ABVariant
  • ABPool
  • validation and pool loading
  • isolated ab_agents_dir resolution
  • warning-based inactive startup behavior
• src/utils/conversation_service.py
  • A/B comparison creation
  • preference submission
  • pending comparison list/count support
  • metrics updates
• src/utils/sql.py
  • SQL for A/B comparison and metrics flows
• src/cli/templates/base-config.yaml
  • current services.chat_app.ab_testing rendering
  • ab_agents_dir
  • split current pool fields

Notable Behavioral Differences from the Original PR Draft

• Admin management is no longer inside the chat settings modal; it is on a dedicated /admin/ab-testing page.
• The system no longer uses shared agents_dir for A/B variants.
• The admin page saves deployment-wide config through Postgres-backed static config, not in-memory only.
• Sampling is server-side, not frontend-only.
• max_pending_per_conversation now supports multiple unresolved comparisons instead of a single hardcoded pending state.
• Incomplete A/B config warns and stays inactive instead of failing deployment startup.
• The temporary basic-auth admin bridge used during development/testing has been removed.

Test Coverage Status

Automated coverage has been updated to reflect the current implementation rather than the original PR draft. The current test surface includes:

• updated Playwright fixtures for current admin payloads and isolated A/B agent pools
• updated A/B workflow coverage for the dedicated admin page and current selectors
• coverage for queued pending comparisons
• backend unit coverage for admin payloads, disable behavior, and pending-comparison enforcement
• regression coverage proving the temporary basic-auth admin bridge was removed without changing permanent auth behavior

Notable Follow-Up Fixes

After the original feature landed, a number of follow-up fixes and product-shape changes were made:

• moved A/B management from the chat settings modal to a dedicated admin page
• split admin saves into separate settings and variants save flows
• fixed runtime refresh so admin saves rehydrate from persisted Postgres config instead of stale cached state
• made archi restart -c ... reseed deployment config so static-config changes are actually applied on restart
• updated disable behavior so the inactive state persists and re-renders correctly
• changed A/B variant config from name-based matching to explicit label plus agent_spec
• isolated A/B-only agent specs into ab_agents_dir instead of sharing normal agents_dir
• added staging/rendering support for ab_agents_dir in deployment generation
• enabled inline creation of new A/B-only agent specs from the admin variant selector
• moved server-side sampling decisions out of frontend-only Math.random() logic
• expanded pending-comparison handling from a single unresolved comparison to queue-aware enforcement using max_pending_per_conversation
• fixed A/B compare/runtime bugs around A/B agent directory resolution and variant initialization
• cleaned up A/B arm headers and trace presentation to better match the current chat theme
• fixed agent-dropdown delete confirmation behavior so the menu stays open while confirming
• removed the temporary basic-auth admin-testing bridge once dedicated admin-page work was complete

[haozturk]

Thanks Jason. First impressions: Couple of issues I noticed:

• I don't see where to see select the liked response. The feedback buttons disappeared
• Timer doesn't work
• It's not clear which response belongs to which config/model.
• Agent activity is shown by default. I like it better when it's not shown by default but the user can see them if they like
• Can we configure the frequency? I don't think we want to do it for every message.

[Copilot]

Pull request overview

Adds a server-driven champion/challenger A/B testing “pool” to the chat app, including admin configuration UI, a new comparison streaming endpoint, and Postgres-backed per-variant metrics.

Changes:

• Introduces ABPool/ABVariant utilities and server endpoints for pool config, streaming comparisons, preference submission, and metrics.
• Refactors frontend A/B behavior to stream a single interleaved NDJSON comparison (two arms) and adds an admin-only pool editor + quick variant creation.
• Extends DB schema and SQL to store variant metadata on comparisons and maintain aggregate variant metrics; adds Playwright E2E coverage.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/ui/workflows/21-ab-testing.spec.ts	New E2E workflows covering admin gating, pool editor, comparison streaming, and voting.
tests/ui/fixtures.ts	Adds A/B-related mock data and route helpers for Playwright.
src/utils/sql.py	Extends A/B comparison queries and adds variant-metrics upsert/select queries.
src/utils/conversation_service.py	Stores/reads variant metadata on comparisons; adds variant-metrics update/query helpers.
src/utils/ab_testing.py	New pool/variant config loader, validation, and challenger sampling logic.
src/interfaces/chat_app/templates/index.html	Replaces old A/B toggle UI with admin-only pool editor section.
src/interfaces/chat_app/static/chat.js	Adds pool endpoints, NDJSON reader, pool editor UI, A/B stream handling, and “clone as variant” UX.
src/interfaces/chat_app/static/chat.css	Styles pool editor, quick-variant panel, and new A/B arm layout.
src/interfaces/chat_app/event_formatter.py	New shared formatter to unify streaming event shaping across normal and A/B streaming.
src/interfaces/chat_app/app.py	Adds pool/compare/metrics endpoints; implements threaded A/B streaming; integrates formatter; refactors helpers.
src/cli/templates/init.sql	Adds variant metadata columns to ab_comparisons and creates ab_variant_metrics.
src/cli/templates/base-config.yaml	Adds template support for ab_testing configuration.
src/bin/service_chat.py	Makes service startup more robust to missing config keys; resolves template/static paths.
src/archi/providers/local_provider.py	Changes Ollama model kwargs to keep models alive for longer.
src/archi/pipelines/agents/base_react.py	Emits implicit thinking events when providers produce empty chunks.
src/archi/pipelines/agents/agent_spec.py	Adds ab_only frontmatter flag support to agent specs.
docs/docs/configuration.md	Documents A/B testing pool configuration and variant fields.
docs/docs/api_reference.md	Documents pool/compare/metrics endpoints (and notes legacy).
configs/submit76/config.yaml	Adds a deployment config example enabling the A/B pool.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pmlugato]

@haozturk @hassan11196 @JasonMoho The PR has been updated and is ready to be reviewed now. One thing that needs your testing (Hasan, Hassan) is the integration with the RBAC setup, since I don't have the according secrets, config entries, whatever else, to launch this myself. I set up a proxy admin role to test, which worked, but not the real thing (you can also send me what I need to run what you guys have going).

PR description at the top of the page has been updated, including notable and minor fixes/changes since last update.

Let me know what y'all think and if it works, and what we need to do to get it in.

[haozturk]

Overall looks very good, thank you @pmlugato ! There's a few changes I requested inline.

[pmlugato]

hey @haozturk , I've addressed your comments, and tested with full RBAC set up thanks to help from you and @hassan11196 . Here are the details for your convenience:

• Added the following A/B permissions:

  • ab:participate
  • ab:view
  • ab:metrics
  • ab:manage

• Added a per-user A/B participation slider in Chat Settings:

  • deployment sample_rate remains the default
  • each participating user can override their own rate from 0-1
  • the setting is stored per user in Postgres

• Kept runtime A/B spec resolution Postgres-only:

  • staged ab_agents_dir files are import inputs only
  • runtime reads from the database only

• ab:participate

  • can be sampled into A/B comparisons
  • can see and use the personal participation-rate slider

• ab:view

  • can open the A/B Testing page in read-only mode

• ab:metrics

  • can open the A/B Testing page
  • can view aggregate comparison metrics

• ab:manage

  • can change experiment settings
  • can manage variants
  • can create/delete A/B agent specs

target_roles and target_permissions are additional, optional filters on top of ab:participate.

• If neither is set, all users with ab:participate are eligible.
• If only target_roles is set, the user must match at least one listed role.
• If only target_permissions is set, the user must match at least one listed permission.
• If both are set, the user must satisfy both filters.

[haozturk]

Thanks a lot @pmlugato looks almost ready. See my inline comments. The restart issue looks major, others are minor.

[haozturk]

Here's one problem I noticed: If I restart the app, the UI created/set variants and the experiment settings are reset, since the YAML config overwrites them. This will become a problem when we create new variants on UI and restart the app. We'll have to need to re-create these variants on every restart.

I can think of the following solution: For A/B testing config, use the YAML values during bootstrap if the A/B testing in the DB is empty. Otherwise, use the values in the DB. Keep an explicit force override mode for cases where you intentionally want YAML to overwrite A/B too. This should be false by default.

In order words, YAML remains authoritative for most static config that can't be edited in the UI. DB becomes authoritative only for UI-managed A/B settings (services.chat_app.ab_testing).

This is introducing an extra complication, but it's necessary if we want to manage these settings over the UI. Let me know what you think.

[haozturk]

The word interruptions implies a negative disruption. Use something like to lower for a more standard single-response flow.

[haozturk]

Minor: Don't know where it's exactly managed, but when the quicker response finishes, its timer doesn't stop. It somehow made me think that "is there more to come?" on that response. I think we should make it stop when the response ends.

[haozturk]

As far as I understand, the following is the mapping between the config and what's shown in the UI:

• minimal: Hidden
• normal: Collapsed
• verbose: Expanded

What's in the config doesn't make sense to me. I would use in the config what's shown in the UI.

[haozturk]

Can we document the mapping between what's shown in the UI and the config? Also these names aren't very intuitive. Maybe consider renaming.

[pmlugato]

@haozturk thanks again for the comments, I agree with them. Any UI changes to A/B testing configuration now persist across restarts. I also followed your suggestion and added a force_yaml_override option should it be needed. Timer is fixed, and naming is clearer and consistent between config/UI/docs. Thanks for the good testing, let me know if you find anything else or if we're good to go.