GitHub - msv-lab/just-tri-it: Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation (OOPSLA'26) · GitHub
/" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
msv-lab
just-tri-it
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>415 Commits<br>415 Commits
datasets
datasets
doc
doc
proofs
proofs
scripts
scripts
src/just_tri_it
src/just_tri_it
tests
tests
.gitignore
.gitignore
.python-version
.python-version
README.md
README.md
pyproject.toml
pyproject.toml
test_requirements.txt
test_requirements.txt
uv.lock
uv.lock
View all files
Repository files navigation
Reducing Hallucinations in LLM-Generated Code<br>via Semantic Triangulation
Yihan Dai, Sijie Liang, Haotian Xu, Peichu Xie, Sergey Mechtaev
arXiv:2511.12288
LLM-generated code often contains hallucinated bugs, and since expected behavior is rarely formally specified, they are hard to detect automatically. Identifying which, if any, of the sampled programs are correct is akin to a police detective questioning suspects. Because LLMs make correlated errors, most suspects have colluded on the same fake alibi — so plurality (majority) voting does not identify the truth; it merely amplifies their shared deception.
Previous methods bring in extra witnesses: LLM-generated tests, or specifications auto-formalized from the problem description (e.g., Hoare-style postconditions). But these witnesses are produced by the same LLM and share the suspects' flawed logic — a biased witness who swears an oath with crossed fingers and corroborates the false alibi:
just-tri-it plays the inspector, exposing lies by questioning the problem from an unexpected angle to obtain a reliable witness. This is called semantic triangulation :
A semantic triangulation (τ, φ) consists of a dissociative problem transformation τ and a relation over pairs of programs (a hyperproperty) φ that induces a bijection between semantic equivalence classes of programs, mapping correct solutions of a problem d to correct solutions of the transformed problem τ(d).
Given the problem d under investigation, the inspector opens an unexpected line of inquiry d′ = τ(d), samples solutions to both problems independently, and cross-examines them with φ:
Each requirement plays its own role in the interrogation:
Dissociative : solving τ(d) requires a fundamentally different algorithm, so the question comes from an angle wholly unrelated to the suspects' rehearsed alibi — unlike mere paraphrasing, which they withstand. Implemented transformations include partial inversion (swap input and output), answer enumeration (output all valid answers), and problem decomposition.
Bijection-inducing : distinct errors in solutions to d map to distinct errors in solutions to d′, so even subtle inconsistencies in the suspects' stories are detected.
Correctness-coupling : the bijection maps correct solutions to correct solutions, so a truthful account is corroborated by a truthful witness, while two independent lies rarely match by coincidence.
A correct program's story holds up under any line of questioning; hallucinated programs, like liars caught off guard, betray themselves through contradiction. Under our mathematical model of LLM hallucinations, we prove that agreement with such a triangulated witness yields strictly higher confidence of correctness than plurality voting — the selected program reflects accurate generalization rather than spurious statistical correlations. Detailed illustrations on CodeElo and LiveCodeBench problems, the full theory, and the evaluation are in the paper.
Setup
Set your 302.ai API key via the environment variable AI302_API_KEY. Dependencies are managed by uv.
Lint, type-check and test:
uv run ruff check .<br>uvx mypy src/<br>uv run pytest
Usage
Benchmarking
Compare tool configurations on code generation datasets and compute basic statistics:
uv run benchmark --dataset DATASET [--task TASK_ID] --selector TOOL_CONFIG --model MODEL
For example:
uv run benchmark --dataset datasets/test.json --selector Plurality --model gpt-4o
For LiveCodeBench v6, first decompress the dataset (unzip datasets/lcb_part6.json.zip), then:
uv run benchmark --dataset datasets/lcb_part6.json --selector CodeT_IO --model gpt-4o --task atcoder_abc387_b
Available configurations: Plurality, MaxTest_Assert, MaxTest_IO, CodeT_Assert, CodeT_IO, Syntactic, OffByOne, Postcondition, FWD_INV, FWD_SINV. Example...