Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add a wrapper for solve #2

Closed
wants to merge 43 commits into from
Closed

feat: Add a wrapper for solve #2

wants to merge 43 commits into from

Conversation

dustinbyrne
Copy link
Contributor

@dustinbyrne dustinbyrne commented Jun 1, 2024 •
edited
Loading

This change runs solve (from getappmap/appmap-js#1832) over each instance from within a SWE-bench test bed.

TODO

  • This wrapper does not yet incorporate AppMap data from tarballs.
  • Patch files are not yet staged for eval

@dustinbyrne dustinbyrne force-pushed the feat/solve branch 2 times, most recently from 329b290 to 872b5d8 Compare June 1, 2024 01:49
@kgilpin
Copy link
Contributor

kgilpin commented Jun 3, 2024 •
edited
Loading

Suggested lint command:

--lint-command="./venv/bin/python -m flake8 --extend-ignore=BLK100,W293,E501,E302,D"

Be sure to pip install flake8 into the virtualenv before running.

D is for Documentation.

@dustinbyrne dustinbyrne closed this Jun 3, 2024
@dustinbyrne dustinbyrne reopened this Jun 3, 2024
@dustinbyrne dustinbyrne mentioned this pull request Jun 3, 2024
Merged
dustinbyrne and others added 3 commits June 3, 2024 15:35
@dustinbyrne @kgilpin
feat: Add a wrapper to the solve command
04fad95 
WIP, and subject to change, but in a working state.
@dustinbyrne @kgilpin
ci: Add an action to run solve over SWE bench
f65005b
@kgilpin
fix: Write issue descriptions into individual directories
5da05dc 
Propagate variables that are needed by the solver.
Remove global variable.
dustinbyrne and others added 19 commits June 3, 2024 16:27
@dustinbyrne
fixup! Checkout base commit, add --keep argument
5b90c73
@kgilpin
drop: Run only requests tests
dae867a
@kgilpin
feat: Capture solver logs
f05235a
@kgilpin
feat: Support linting
484c84c
@kgilpin
fixup
e883e41
@dustinbyrne
Run with a single process pool, use swebench env management
2f2eecf
@dustinbyrne
fixup! Run with a single process pool, use swebench env management
c7c42b8
@dustinbyrne
Merge branch 'feat/limited-run-requests-only' into feat/solve
12e1386
@dustinbyrne
filter sympy
5045fcb
@dustinbyrne
include conda path
ca6a13c
@dustinbyrne
validate conda path exists
74c37cd
@dustinbyrne
update appmap-js
f25b78e
@dustinbyrne
drop: ssh debug
534fc08
@dustinbyrne
Revert "drop: ssh debug"
c7e03e5 
This reverts commit 534fc08.
@dustinbyrne
Add reporting script
9c0bfdf 
This will generate a csv with default args
@dustinbyrne
generate and output csv to artifacts
b1ba0d0
@dustinbyrne
skip problematic task envs
804c978
@kgilpin @dustinbyrne
feat: Bring solver Python code over from appmap-js
d95df5c
@kgilpin @dustinbyrne
drop: Run 'requests' (smaller)
18edaaa
@dustinbyrne dustinbyrne added the evaluate runs evaluation label Jun 4, 2024
@kgilpin kgilpin marked this pull request as ready for review June 4, 2024 22:35
@dustinbyrne dustinbyrne mentioned this pull request Jun 5, 2024
Merged
@dustinbyrne dustinbyrne closed this Jun 5, 2024
@kgilpin kgilpin deleted the feat/solve branch June 10, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
evaluate runs evaluation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dustinbyrne @kgilpin @dividedmind