GitHub - cernodatainc/cernodata: Cernodata is an open-source ETL framework for layout-aware PDF extraction, automated quality iteration, structural layout debugging, and dataset generation for RAG and LLM fine-tuning. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
cernodatainc
cernodata
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>8 Commits<br>8 Commits
Architecture_Specification.md
Architecture_Specification.md
README.md
README.md
data_storage_decision_tree.md
data_storage_decision_tree.md
parsing_decision_tree.md
parsing_decision_tree.md
testing_decision_tree.md
testing_decision_tree.md
View all files
Repository files navigation
Cernodata
Cernodata is an open-source ETL framework for layout-aware PDF extraction, automated quality iteration, structural layout debugging, and dataset generation for RAG and LLM fine-tuning.
Parsing PDFs for RAG is a guessing game
You point a parser at a PDF, cross your fingers, and hope the chunks come out usable. They don't. And when they finally do, the next file has two columns instead of one, or a table that turns out to be a photograph of a table, or it's in a language you didn't plan for. The code you wrote yesterday breaks today. So you write another parser. And then another one. It never ends, because there is always another document.
It doesn't have to work this way.
Cernodata is a guided pipeline that turns messy, irregular PDFs into structured data you can verify. Verifiable is the important word there. Most pipelines hand you data you merely hope is right.
One framework instead of one parser per source
You don't hand-code a parser for each source anymore. You answer a few questions, and Cernodata picks the engine for you.
When a document doesn't parse cleanly, it doesn't hand the problem back to you. It switches presets, or wiggles the parameters like DPI, contrast, and table detection, and tries again until the output is right. That week of fragile glue code you used to write for every new source now happens on its own.
Never guessing what broke
Quality isn't a black box here. Every document climbs a ladder of checks that you can see, tune, and turn on as you need them. Garbage characters at the bottom, then table-grid alignment, and at the top, real retrieval scores from a vector search. Whether the chunks can actually be found is, after all, the only question that matters.
When a document fails, you know which check failed, on which page, and why. And visual overlays draw the parse right on top of the page, so you catch a mis-merged column the way you catch a typo. You just look at it, instead of digging through a stack trace three days later.
It runs on your machine without taking it hostage
Heavy OCR shouldn't freeze your laptop, or eat memory until something crashes while your local model is trying to run. So Cernodata caps concurrency, enforces hard memory ceilings, and does the heavy ML work in subprocesses that clean up after themselves. Ingestion ought to be like a print job: something running in the background that you can ignore. Not a fire drill.
Built for the ugly ones
Demos are made of clean documents. Real work isn't.
If a 100-page file fails on page 42, Cernodata slices out page 42 and retries it with a heavier engine, instead of running all 100 again. ML workers that leak memory restart themselves. The hard parts are engineered for the large, messy, documents that quietly break everyone else's pipeline.<br>You'll still have ugly documents. You just won't have to write a parser for each one.
This is the part where you decide to care
The README is the promise. The library is coming.
Want it the day it lands? And a say in what it becomes?
Leave your email.
→ Join the waitlist · takes 10 seconds, no spam, ever
Architecture & Specification
The primary application architecture specification for this repository is located in:
Architecture_Specification.md
It details the 9 core architecture subsystems:
Preset Selection Decision Tree
Curated & User-Defined Presets
Automated Confidence-Guided Iteration Loop & Quality Checks
Page-Level PDF Subdivision
Structural Extraction, Bounding Box Visual Overlays & Template Hints
Downstream Evaluation Decision Tree & SLM Prompt Generation
Downstream Packing & Target Use-Case Decision Tree
Resource Protection & Work Dispatching Subsystem
Interactive Visual Web App & Audit-Friendly Standalone Script...