OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews - ACL AnthologyOpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews<br>Maximilian Idahl,<br>Zahra Ahmadi

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account.<br>Once you create that issue, the correction will be reviewed by a staff member.<br>⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.<br>Important : The Anthology treat PDFs as authoritative. Please use this form only to correct data<br>that is out of line with the PDF. See our corrections<br>guidelines if you need to change the PDF.<br>Title<br>Adjust the title. Retain tags such as

Authors<br>Adjust author names and order to match the<br>PDF.<br>Add AuthorAbstract<br>Correct abstract if needed. Retain XML formatting tags such as . You may use ... for bold , ... for italic, ... for underline, ... for small-caps, ... for typewriter text, ... for URLs, for hyperlinks, and for paragraph breaks.

Verification against PDF<br>Ensure that the new title/authors match the snapshot below. (If there<br>is no snapshot or it is too small, consult the PDF.)<br>Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including<br>middle initials, hyphens, and accents.<br>Create GitHub issue for staff review

Abstract<br>We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer’s recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Anthology ID:2025.naacl-demo.44Volume:Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations)Month:AprilYear:2025Address:Albuquerque, New MexicoEditors:Nouha Dziri,<br>Sean (Xiang) Ren,<br>Shizhe DiaoVenues:NAACL<br>WSSIG:Publisher:Association for Computational LinguisticsNote:Pages:550–562Language:URL:https://aclanthology.org/2025.naacl-demo.44/DOI:10.18653/v1/2025.naacl-demo.44Bibkey:idahl-ahmadi-2025-openreviewerCite (ACL):Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pages 550–562, Albuquerque, New Mexico. Association for Computational Linguistics.Cite (Informal):OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews (Idahl & Ahmadi, NAACL 2025)Copy Citation:BibTeX<br>Markdown<br>MODS XML<br>Endnote<br>More<br>options…PDF:https://aclanthology.org/2025.naacl-demo.44.pdf<br>PDF<br>Cite<br>Search

Fix data

Export citation

BibTeX<br>MODS XML<br>Endnote<br>Preformatted<br>@inproceedings{idahl-ahmadi-2025-openreviewer,<br>title = "{O}pen{R}eviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews",<br>author = "Idahl, Maximilian and<br>Ahmadi, Zahra",<br>editor = "Dziri, Nouha and<br>Ren, Sean (Xiang) and<br>Diao, Shizhe",<br>booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations)",<br>month = apr,<br>year = "2025",<br>address = "Albuquerque, New Mexico",<br>publisher = "Association for Computational Linguistics",<br>url = "https://aclanthology.org/2025.naacl-demo.44/",<br>doi = "10.18653/v1/2025.naacl-demo.44",<br>pages = "550--562",<br>ISBN = "979-8-89176-191-9",<br>abstract = "We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review...