Every Eval Ever | EvalEval Coalition
Infrastructure @EvalEval
One schema for<br>Every Eval Ever.
A unified, open data format and public dataset for AI evaluation results.<br>We are collecting all evaluation results in a standardized schema to enable rigorous research and broader impact.
Explore Schema
Launch Announcement
Developed by the EvalEval Coalition
with feedback from researchers at
Inspect
Why this Schema?
Addressing the fragmentation in AI evaluation to enable trust and comparability.
Unifying the Ecosystem
Evaluation results are currently siloed by framework. This schema creates a common interchange format, allowing results from HELM, EleutherAI, Inspect, and custom scripts to co-exist and be compared directly without complex mapping.
Trust through Provenance
A score without configuration is just noise. We go beyond the metric to capture the full experimental context—prompt templates, inference parameters, and system states—making every result traceable, transparent, and reproducible.
Actionable Meta-Science
Liberating evaluation results from static PDFs and closed leaderboards. We transform scattered metrics into a structured, queryable global dataset, powering the next generation of meta-analysis and automated leaderboard construction.
The Schema
A granular, line-by-line breakdown of the standardized format.
Aggregate Schema<br>Instance Schema
59ee0934-f60d-4d4b-b986-844fc51e89a3.json
"schema_version": "0.2.2",<br>"evaluation_id": "lm-eval/moonshotai.../17642...",<br>"retrieved_timestamp": "1764204739.50717",
"source_metadata": {<br>"source_name": "lm-eval",<br>"source_type": "evaluation_run",<br>"source_organization_name": "EleutherAI",<br>"source_organization_url": "https://www.eleuther.ai",<br>"evaluator_relationship": "third_party"<br>},
"model_info": {<br>"name": "Kimi K2 Instruct",<br>"id": "moonshotai/kimi-k2-instruct",<br>"developer": "moonshotai",<br>"inference_platform": "Together AI",<br>"inference_engine": {<br>"name": "vLLM",<br>"version": "0.6.3"<br>},
"eval_library": {<br>"name": "lm-eval",<br>"version": "0.4.11"<br>},
"evaluation_results": [<br>"evaluation_result_id": "mmlu_pro_cot_correct",<br>"evaluation_name": "MMLU-Pro - COT correct",<br>"evaluation_timestamp": "1764204739.50717",<br>"source_data": {<br>"source_type": "hf_dataset",<br>"dataset_name": "MMLU-Pro",<br>"hf_repo": "TIGER-Lab/MMLU-Pro",<br>"hf_split": "test",<br>"samples_number": 12032<br>},<br>"metric_config": {<br>"evaluation_description": "Fraction of correct answers after chain of thought",<br>"lower_is_better": false,<br>"score_type": "continuous",<br>"min_score": 0.0,<br>"max_score": 1.0<br>},<br>"score_details": {<br>"score": 0.819,<br>"details": {<br>"tab": "Accuracy"<br>},<br>"generation_config": {<br>"generation_args": {<br>"temperature": 0.0,<br>"top_p": 1.0,<br>"top_k": -1,<br>"max_tokens": 2048,<br>"reasoning": true<br>},<br>"additional_details": {<br>"description": "Chain of thought prompting used."<br>],
"detailed_evaluation_results": {<br>"file_path": "detailed_results/e70acf51....jsonl",<br>"format": "jsonl",<br>"checksum": "a1b2c3d4..."
Metadata & ID
Every file starts with versioning and a unique identifier.
schema_version: Now v0.2.2.
evaluation_id: Unique ID.
retrieved_timestamp: Unix Time.
Source Provenance
Links to original data and distinguishes source types.
Model Specification
Complete model specification using Hugging Face format.
name: Human-readable model name.
id: HuggingFace model ID.
inference_platform: API provider.
inference_engine: Local runner.
Evaluation Library
Mandatory field specifying the framework used, like HELM, lm-eval, or custom code.
Flexible Metrics
Supports continuous scores and level-based ratings.
evaluation_result_id: Stable key linking to instance records.
metric_config: Configuration for the metric.
score_details: The actual score value.
Generation Config
Reproducibility parameters for the generation.
Detailed Results
This links to the separate Instance Level file.
detailed_evaluation_results: Object with file path.
checksum: Verification hash.
"schema_version": "instance_level_eval_0.2.2",<br>"evaluation_id": "lm-eval/moonshotai...",<br>"model_id": "moonshotai/kimi-k2-instruct",<br>"evaluation_name": "mmlu_pro",<br>"sample_id": "test_1042",<br>"interaction_type": "single_turn",
"input": {<br>"raw": "Question: Which is a scalar?...",<br>"reference": ["C"],<br>"choices": ["Velocity", "Force", "Speed"]<br>},
"output": {<br>"raw": ["Speed is a scalar. The answer is C."],<br>"reasoning_trace": ["Velocity has direction..."]<br>},
"answer_attribution": [<br>"turn_idx": 0,<br>"source": "output.raw",<br>"extracted_value": "C",<br>"extraction_method": "regex",<br>"is_terminal": true<br>],
"evaluation": {<br>"score": 1.0,<br>"is_correct": true<br>},<br>"token_usage": {<br>"input_tokens": 365,<br>"output_tokens": 42,<br>"total_tokens": 407
Instance Meta
Identifier for this specific sample and interaction type.
interaction_type: single_turn, multi_turn, or agentic.
Input Data
The prompt and ground truth info.
Output
The models raw response and optional reasoning trace.
Evaluation & Usage
Instance level score, correctness and token stats.
Design...