SWE-Touch | Benchmarking Coding Agents When Users Touch the Code

User-agent interaction beyond messages

Benchmarking Coding Agents When Users Touch the Code

SWE-Touch tests whether a coding agent can complete a software repair after a user has directly changed the code in their shared workspace.

Yuqiao Tan · Jinxiang Meng · Fangyu Lei · Minzheng Wang · Shizhu He · Jun Zhao · Kang Liu

Code & documentation<br>Dataset

9 coding models

3 repair benchmarks

−7.7 mean points on Verified

Research question

When a user contributes code during an ongoing task, can the agent interpret that edit against the issue, repository, and tests and still reach a correct solution?

Evaluation design

From autonomous repair to a shared workspace

Each task is evaluated under matched conditions. The user edit is the only change to the live task state.

Vanilla<br>The agent repairs the original checkout

The issue, repository, tools, verifier, and interaction budget follow the standard autonomous setting.

vs.

Counter-Edit<br>A user changes relevant code during the repair

A small, plausible but task-conflicting edit and a contextual user message enter the same live workspace.

Click the figure for the full-resolution pipeline.

1Mine critical regions Aggregate code locations touched by successful repair trajectories.

2Generate and validate Build a sparse edit that fails alone and still blocks the reference repair.

3Deliver during repair Apply the edit when the evaluated agent reaches the relevant code.

same issuesame checkoutsame toolssame budgetsame verifier

Main evidence

User edits expose a gap hidden by autonomous repair

Resolve rates are percentages. Verified results report mean ± standard deviation.

−7.7 average change on SWE-bench Verified All nine evaluated models decline under Counter-Edit.

−1.3 to −16.5 large variation across models Strong autonomous scores do not imply consistent recovery.

63.3% failures retain the conflict Most failed trajectories end with the user-introduced behavior still active.

SWE-bench Verified

Retention is the fraction of tasks solved under Vanilla that remain solved under Counter-Edit.

Resolve-rate decrease

Counter-Edit minus Vanilla, percentage points

GPT 5.5−1.3

Claude 4.8−1.8

GLM 5.1−4.3

Qwen 3.7−4.8

Kimi K2.6−6.0

MiniMax M2.5−9.5

DeepSeek V4−11.0

MiniMax M2.7−13.8

Qwen Coder−16.5

ModelVanillaCounter-EditΔRetention

Claude Opus 4.885.2 ± 1.883.3 ± 0.6−1.896.0<br>GPT 5.580.5 ± 1.079.2 ± 0.6−1.395.0<br>GLM 5.172.7 ± 2.068.3 ± 0.8−4.383.3<br>MiniMax M2.776.5 ± 1.562.7 ± 2.4−13.878.1<br>MiniMax M2.575.7 ± 3.366.2 ± 1.0−9.578.3<br>Qwen 3.7 Max75.2 ± 1.070.3 ± 0.8−4.890.3<br>Qwen3-Coder-480B57.2 ± 3.540.7 ± 1.0−16.560.8<br>Kimi K2.670.3 ± 2.064.3 ± 3.4−6.087.2<br>DeepSeek V4 Pro74.8 ± 0.863.8 ± 1.8−11.081.5

Finding. The ordering changes substantially: Qwen3-Coder-480B and MiniMax M2.7 lose more than 13 points, while GPT 5.5 and Claude Opus 4.8 retain over 95% of their Vanilla-solved tasks.

Harder repair tasks

Selected SWE-Bench Pro and DeepSWE tasks use a longer budget and trajectory-relative delivery.

ModelSWE-Bench ProDeepSWEVanillaCounterΔVanillaCounterΔ

Claude Opus 4.868.068.00.056.046.0−10.0<br>GPT 5.538.038.00.064.056.0−8.0<br>GLM 5.143.132.8−10.319.416.8−2.5<br>MiniMax M2.730.624.6−6.02.22.20.0<br>MiniMax M2.532.624.6−8.00.00.00.0<br>Qwen 3.7 Max36.026.0−10.04.12.1−2.0<br>Qwen3-Coder-480B20.014.0−6.00.00.00.0<br>Kimi K2.650.048.0−2.018.012.0−6.0<br>DeepSeek V4 Pro34.032.0−2.04.12.0−2.1<br>Mean39.134.2−4.918.615.2−3.4

Finding. The aggregate direction persists on both harder benchmarks, although the effect varies by model and task family.

Message, edit, and frequency

ConditionGPT 5.5GLM 5.1M2.7Qwen 3.7

Vanilla81.570.576.574.0<br>Message, K=379.573.076.577.0<br>Code edit, K=380.566.567.071.5<br>Both, K=178.572.064.571.5<br>Both, K=379.569.064.571.0<br>Both, K=578.069.060.069.0

Exploratory resolve rates on SWE-bench Verified. K is the maximum number of interventions.

Finding. Messages alone do not reproduce the effect consistently. Direct changes to executable code are the stronger intervention in this comparison.

Edits stay local

BenchmarkReference lines / filesCounter lines / files

Verified13.3 / 1.207.0 / 1.04<br>Pro361.0 / 5.4413.0 / 1.40<br>DeepSWE730.2 / 7.2410.8 / 1.52

Mean changed lines and files per patch.

Finding. Counter-Edits remain small even when the underlying reference repair spans hundreds of lines and several files.

Trajectory analysis

Failure is not only passive acceptance

Agents sometimes remove or replace the inserted edit and still fail. Recognizing a problematic contribution does not guarantee that the model can restore the repository to a correct state.

63.3% of Vanilla-solved / Counter-Edit-failed runs end with the conflicting behavior retained.

Failure modes and edit-response behavior on SWE-bench Verified.

Open release<br>Reproduce and extend SWE-Touch

The release separates reusable benchmark records from the Harbor-based evaluation implementation.

Code<br>Runner, schemas, user simulator,...