Fverac/swebench self test agent #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

fverac wants to merge 14 commits into main from fverac/swebench_self_test_agent

Contributor

fverac commented Oct 17, 2025

No description provided.

MarcCote force-pushed the fverac/swebench_self_test_agent branch from 2b609a7 to 63ce0f1 Compare

October 22, 2025 00:39

MarcCote reviewed

View reviewed changes

debug_gym/agents/base_agent.py Outdated Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

debug_gym/agents/swe_agent.py Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

debug_gym/gym/envs/env.py Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

debug_gym/gym/envs/swe_bench.py Outdated Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

debug_gym/gym/envs/swe_bench.py Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

debug_gym/gym/tools/eval.py Outdated Show resolved Hide resolved

MarcCote reviewed

View reviewed changes

scripts/config_swebench.yaml Outdated Show resolved Hide resolved

Collaborator

MarcCote commented Oct 22, 2025 •

edited

Loading

Things to do:

fix failing tests
add missing tests for the debug_mode
decide what we do with pdb's entrypoint when debug_mode is False

MarcCote marked this pull request as ready for review

October 22, 2025 03:41

Contributor Author

fverac commented Oct 23, 2025

I feel like debug_mode will be easily confused with the other --debug flag. Should we call the mode something else? And/or add docstring

pre_apply_test_patch
oracle_test_available
debug_oracle_tests

matheper reviewed

View reviewed changes

debug_gym/gym/tools/submit.py Outdated Show resolved Hide resolved

MarcCote requested review from MarcCote and matheper

October 24, 2025 19:56

fverac and others added 11 commits

October 25, 2025 03:20


          initial changes, messy

8b66e7c

Signed-off-by: fverac <fabiovera@microsoft.com>


          minor fix with entrypoint, swebench config

11e63ea

Signed-off-by: fverac <fabiovera@microsoft.com>


          working

4db176e

Signed-off-by: fverac <fabiovera@microsoft.com>


          Adding debug_mode and simplifying changes.

ac5a3d1


          fix test failure

516f046

Signed-off-by: fverac <fabiovera@microsoft.com>


          add tests for debug mode

29d213c

Signed-off-by: fverac <fabiovera@microsoft.com>


          minor cleanup

c475cfe

Signed-off-by: fverac <fabiovera@microsoft.com>


          cleanup

20e8027

Signed-off-by: fverac <fabiovera@microsoft.com>


          Use terminated insead of done. Remove debug_mode. Add tool init argum…

e5a8f33

…ents


          Fixing tests

4171cde


          Fixing bug and add tests to check entrypoint

MarcCote force-pushed the fverac/swebench_self_test_agent branch from 3efb55b to 9989196 Compare

October 25, 2025 11:29

MarcCote and others added 2 commits

October 27, 2025 08:37


          Fix issue with setting up entrypoint

95135fd


          update pdb description

7d2d91e

Signed-off-by: fverac <fabiovera@microsoft.com>

Collaborator

MarcCote commented Oct 27, 2025

TODO:

Add integration tests that runs the solution agent on SWE-Bench, SWE-Smith, and R2E-Gym
Add a unit that that checks that env.debug_entrypoint has -m pdb in it.


          Update config for swe-agent

a27b870

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet