Notes on document processing with LLMs | Andrew Wheeler

About

C.V.

Datascience Portfolio

Dissertation

Comment Policy

Consulting

Courses

Advanced Criminology (Undergrad) Crim 3302

Communities and Crime (Undergrad) Crim 4323

Crim 7301 – UT Dallas – Seminar in Criminology Research and Analysis

Crime Science (Graduate) Crim 7381

GIS in Criminology/Criminal Justice (Graduate)

Crime Analysis (Special Topics) – Undergrad

Job Advice Resources

Code Snippets TOC

Andrew Wheeler<br>Crime Analysis and Crime Mapping

Notes on document processing with LLMs

One of the recent papers that came across my X feed was a researcher digitizing a corpus of old Sears catalogs to estimate inflation more accurately:

The technical skills to do this work are closely related to many of the projects I am working on at Gainwell – large-scale document processing using LLMs. So if you are a PhD student and your thesis involves work like this, I would likely want to hire you for a six-figure job as a data scientist at Gainwell.

I wanted to have a quick blog post on some of the tools to process documents. So this researcher processed 180 Sears catalogs that are typically well over 1000 pages, so they likely dropped well over $10k on this project.

Like I said in the X post, for this type of volume processing, you shouldn’t be using the main models. You should first consider the cheaper models, like the flash-lite models from Gemini (not even the flash for this), or the mini/nano models from OpenAI. Using these cheaper models, which I suspect would be of similar accuracy, would reduce the cost to more like $2k in this project.

For those even more budget-conscious (I am processing this volume often daily at Gainwell for different projects), here are a few of my notes on open source models. (Also I am concerned OpenAI will entirely drop the mini/nano models in the future, hence I want to make sure I have plenty of options.)

OCR and Markdown

So this project used the images to help classify the information, but if you are dealing with pure text, a common approach is to first OCR the document, and then apply structured extraction from the text. This works well to save costs even if using served models, as we are talking about an image of a page being 10k tokens, but the extracted text often being well under 1k tokens.

In LLMs for Mortals, I show using the docling library to do this:

If the PDF is already OCRed (it has the underlying text) there are Python libraries to directly read the text. This example uses pypdf, but a better default library is pypdfium2 (it is much faster).

The difference between converting to Markdown vs. reading the text is that Markdown conversion will be a bit nicer in converting tables, graphs, and page chrome (e.g. footers/headers).

I have had good success with docling (which you can have it convert to Markdown with just the page text), but it does load in a PyTorch model (which adds a bit of latency). You can turn it off to only use the text in the PDF (and not the image), which does save time but still has a fair bit of latency.

If the PDF has the text embedded and you do not want a big model, I have been happy with the results of the liteparse library. This does not use an LLM at all to do the conversion to Markdown, so is quite fast.

Due to the increased cost, if you do need a served model, (sometimes you just have latency requirements and you don’t want to deal with your own server), Mistral OCR is worth checking out. This is a good cost-effective alternative if you are using AWS Textract and need things like table extraction.

There are many different models and providers coming out in the space (OvisOCR2 is one of the recent models for OCR, for served models the group that created liteparse (LlamaIndex) also has served model options).

Structured Extraction

OK, so now you have your pages in text that you can use. At this stage, you will want to extract out information. My go-to local model for named entity recognition (NER) is GLiNER, but the GLiNER2 library has examples that do the full structured extraction.

For a primer on the distinction between NER and structured extraction, pretend you had a free-text narrative “PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA. Pt tolerated assessment well, no acute distress noted.”.

For NER, if you extracted vitals you would get back something like:

from gliner import GLiNER

# load GLiNER model<br>model = GLiNER.from_pretrained("urchade/gliner_small-v2")

# Physician Assistant narrative<br>narrative = """PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA.<br>Pt tolerated assessment well, no acute distress noted."""

# define entities to extract<br>labels = ["heart_rate","oxygen"]

# extract entities<br>entities = model.predict_entities(narrative, labels)<br>print(entities)<br>And this prints out:

[{'start': 25, 'end': 34, 'text': 'HR 78 bpm', 'label': 'heart_rate', 'score': 0.9038965106010437}, {'start': 36, 'end': 43, 'text': 'SpO 98%', 'label': 'oxygen',...