GitHub - Steel-predictor-project/steel-llm-eval: Open benchmark: how well can LLMs predict knife-steel properties (edge retention, toughness) from chemical composition, scored against CATRA/Charpy lab measurements. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
Steel-predictor-project
steel-llm-eval
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>3 Commits<br>3 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>data
data
docs
docs
harness
harness
results
results
.gitignore
.gitignore
LICENSE
LICENSE
NOTICE
NOTICE
README.md
README.md
requirements.txt
requirements.txt
run_benchmark.sh
run_benchmark.sh
View all files
Repository files navigation
steel-llm-eval
How well can large language models predict knife-steel properties from chemical composition alone?
An open, reproducible benchmark that gives an LLM only a steel's composition (e.g. C=1.45%, Cr=20%, V=4%, Mo=1%, powder-metallurgy: yes) and asks it to rate two properties on a 1–10 scale, then scores those ratings against objective laboratory measurements :
Edge retention ← CATRA standardized machine-cutting test (total card stock cut, mm) — 48 steels
Toughness ← Charpy impact energy (ft-lbs) — 12 steels
Scoring is scale-free (rank correlation + pairwise ranking accuracy), so a model is judged purely on whether it orders steels correctly, not on how it calibrates the 1–10 scale.
📊 Site with charts & analysis: https://steel-predictor-project.github.io/steel-llm-eval/ · Docs: methodology · results & analysis
Leaderboard
Ranked by mean Spearman rank correlation (ρ) vs. the measurements. Higher is better; 1.0 = perfect ordering, 0.0 = random.
Model<br>Edge ρ (n)<br>Edge pairwise<br>Tough ρ (n)<br>Tough pairwise<br>Mean ρ
steel-predictor (reference ML) †<br>0.992 (48)<br>0.98<br>0.946 (12)<br>0.938<br>0.969
anthropic/claude-sonnet-5<br>0.894 (48)<br>0.918<br>0.844 (12)<br>0.881<br>0.869
google/gemini-3.6-flash<br>0.918 (48)<br>0.913<br>0.698 (12)<br>0.797<br>0.808
openai/gpt-4o<br>0.868 (48)<br>0.907<br>0.600 (12)<br>0.746<br>0.734
meta-llama/llama-3.3-70b-instruct<br>0.864 (47)<br>0.964<br>0.514 (12)<br>0.780<br>0.689
deepseek/deepseek-chat-v3.1<br>0.869 (48)<br>0.910<br>0.380 (12)<br>0.661<br>0.625
openai/gpt-4o-mini<br>0.850 (48)<br>0.984<br>0.385 (12)<br>0.689<br>0.617
Edge retention n=48 (CATRA), toughness n=12 (Charpy). Zero-shot, temperature 0, one sample per steel.
† Important fairness caveat: the reference ML model (Steel-predictor) was trained on these same CATRA/Charpy measurements , so its scores here are largely in-sample and are shown as an upper-reference bar, not as a fair head-to-head with the zero-shot LLMs. The model's honest out-of-sample performance is its LOOCV MAE (0.391), reported in that repo. The LLMs, by contrast, have never seen this labeled set.
What the numbers say
LLMs are genuinely good at ranking edge retention (ρ ≈ 0.85–0.92). Wear resistance is strongly and legibly encoded in composition (carbide-forming elements — C, V, Cr, W, Mo), and frontier models clearly "know" that chemistry.
Toughness is where they struggle (ρ 0.38–0.84). It depends on subtler factors (carbide size/distribution, powder-metallurgy processing, matrix state) that aren't obvious from a composition string, and the spread across models is large.
Frontier > small. Claude Sonnet and Gemini lead; the smaller/cheaper models drop off sharply on toughness while staying competitive on edge retention.
Reproduce
git clone https://github.com/Steel-predictor-project/steel-llm-eval.git<br>cd steel-llm-eval
export OPENROUTER_API_KEY=sk-or-... # one key → OpenAI, Anthropic, Google, Meta, DeepSeek, ...<br>./run_benchmark.sh # runs every model and rebuilds the leaderboard
Run a single model, or a quick offline sanity check with no API key:
python harness/run_eval.py --model anthropic/claude-sonnet-5<br>python harness/run_eval.py --provider mock # deterministic heuristic, no key needed<br>python harness/score.py
Raw per-steel responses are written to results/raw_.csv; scores to results/scores.csv and results/leaderboard.md.
How it works
Prompt (harness/prompts.py) — a fixed system + user prompt gives the model the composition, PM flag, and test hardness (when known) and asks for JSON: {"edge_retention": n, "toughness": n}. Identical for every model.
Run (harness/run_eval.py) — queries a model for all 51 steels via OpenRouter and parses the JSON.
Score (harness/score.py) — vs. the measurements:
Spearman ρ and Kendall τ rank correlation (headline; scale-free).
Pairwise accuracy — over all steel pairs, how often the model orders...