Skip to content

add system prompt and other cli args#1

Open
hjc-puro wants to merge 1 commit intoEQ-bench:mainfrom
NousResearch:cli-args
Open

add system prompt and other cli args#1
hjc-puro wants to merge 1 commit intoEQ-bench:mainfrom
NousResearch:cli-args

Conversation

@hjc-puro
Copy link
Copy Markdown

@hjc-puro hjc-puro commented Aug 12, 2025

  • We add cli args for test model system prompt, base url, api key, max tokens, and rpm rate limit
  • We delay <think></think> stripping so that the full generation with reasoning is logged but the judge still sees only the answer post-

example command:

python eqbench3.py \
  --test-model "model-name" \
  --judge-model "claude-3-7-sonnet-latest" \
  --test-base-url "http://localhost:30000/v1" \
  --test-api-key "x" \
  --test-max-tokens 32768 \
  --iterations 3 \
  --runs-file "${RESULTS_DIR}/run.json" \
  --elo-results-file "${RESULTS_DIR}/elo.json" \
  --threads 4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant