GitHub - jiayuanz3/SWEContextBench · GitHub
/" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
jiayuanz3
SWEContextBench
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star<br>12
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>20 Commits<br>20 Commits
assets
assets
cases
cases
predictions
predictions
swebench_memory
swebench_memory
.gitignore
.gitignore
README.md
README.md
environment.yml
environment.yml
evaluation.sh
evaluation.sh
View all files
Repository files navigation
SWE Context Bench: A Benchmark for Context Learning in Coding
SWE-ContextBench is a benchmark created to evaluate how well programming agents, such as AI coding systems, can reuse past experience when solving new tasks . It is built on top of existing datasets including SWE-Bench Lite, SWE-Bench Multilingual, and SWE-Bench Verified. The dataset contains 1,100 base tasks along with 376 related tasks that are derived from real dependency and reference relationships among GitHub issues and pull requests. These tasks are organized in a way that groups together problems with shared context, enabling the study of how effectively an agent can transfer knowledge across similar tasks . The dataset spans 51 real-world GitHub repositories and covers 9 different programming languages.
SWE-ContextBench is introduced as part of the research paper titled SWE Context Bench: A Benchmark for Context Learning in Coding.
SWE-ContextBench dataset can be downloaded from Hugging Face.
Run Evaluation
We provide pre-built Docker images for the related tasks.
Step1: Put your predictions in the predictions folder, with naming {instance_id}_preds.json
{instance_id}_preds.json follow the same data format convention as the SWE-Bench series datasets:
"instance_id": {<br>"model_name_or_path": {model_name_or_path},<br>"instance_id": {instance_id},<br>"model_patch": {model_patch}
Step2: Run
# Run only on "lite" subset<br>./evaluation.sh {run_id} lite
# Run on full dataset<br>./evaluation.sh {run_id} full
, {run_id} can be any text, e.g., my_run_id
🚨 News
02-05-26. Code Uploaded and Dataset Released 👩💻
About<br>No description, website, or topics provided.<br>Resources<br>Readme<br>Activity<br>Stars<br>12 stars<br>Watchers<br>0 watching<br>Forks<br>1 fork<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.