-->
-->
-->
-->
-->
-->
BEAVER: An Enterprise Benchmark for Text-to-SQL
🦫 BEAVER: An Enterprise Benchmark for Text-to-SQL
Peter Baile Chen1,
Devin Yang1,
Weiyue Li2,
Fabian Wenz1,3,
Yi Zhang4,
Nesime Tatbul1,5,
Michael Cafarella1,
Çağatay Demiralp1,6,
Michael Stonebraker1
1MIT, 2Harvard University,<br>3TU Munich,<br>4Greenshoe, Inc.,<br>5Intel,<br>6AWS AI Labs
arXiv
Dataset
Code
Overview
Leaderboard
Leaderboard
Submission Instructions
How to Submit:
Please send an email to peterbc@mit.edu, along with your method name, a brief description of the method, and, optionally, a link to your paper or codebase. We will follow up with detailed instructions.
Method
Model
Search
Rank<br>Submission Date<br>Method<br>Model<br>Execution Accuracy
Subtask Metrics Across Settings
Citation
If you find our data, code, or the paper helpful, please cite the<br>paper:
@article{chen2024beaver,<br>title={BEAVER: an enterprise benchmark for text-to-sql},<br>author={Chen, Peter Baile and Yang, Devin and Li, Weiyue and Wenz, Fabian and Zhang, Yi and Tatbul, Nesime and Cafarella, Michael and Demiralp, {\c{C}}a{\u{g}}atay and Stonebraker, Michael},<br>journal={arXiv preprint arXiv:2409.02038},<br>year={2024}
Copy
BEAVER is a large-scale enterprise text-to-SQL dataset containing 9128 queries spanning 812 tables across 19 diverse domains. Of these, 7978 queries are publicly released, while the remaining portion is held out as a private test set. Queries and databases were collected from private organizations.
To facilitate fine-grained evaluation and analysis, we provide
annotations for five subtasks: multi-table retrieval, join key detection, column mapping, domain knowledge extraction, and query decomposition
three categories of queries: complex queries without domain knowledge, domain-specific queries with minimal complexity, and domain-specific complex queries
Example data
Representative BEAVER tasks with question, SQL, and subtask annotations.
Task id
db<br>source
Question
SQL
Multi-Table retrieval
Tables used in SQL
Join Keys
Connections among used tables
Column Mapping
Mapping from question phrases to table columns(s)
Domain knowledge
Domain-specific predictates used in SQL
Subquery Decomposition
Decomposition of SQL into simpler sub-queries
Usage
We have created a unified MySQL version for our dataset. A free<br>MySQL installation can be found<br>here. After<br>the installation, import the MySQL dump files from the google<br>drive to your local MySQL databases using
mysql -u root -p
To execute a SQL statement, you can either log in to the MySQL<br>interface or you can do it via<br>mysql-connector-python.
If you want to use the Oracle version of<br>DW queries, you can download the free oracle<br>database and import the CSVs.
-->
Changelog
-->
Citation
If you find our data, code, or the paper helpful, please cite the<br>paper:
article{chen2024beaver,<br>title={BEAVER: an enterprise benchmark for text-to-sql},<br>author={Chen, Peter Baile and Yang, Devin and Li, Weiyue and Wenz, Fabian and Zhang, Yi and Tatbul, Nesime and Cafarella, Michael and Demiralp, {\c{C}}a{\u{g}}atay and Stonebraker, Michael},<br>journal={arXiv preprint arXiv:2409.02038},<br>year={2024}
Copy