socOCRbench - Noah Dasanaike

Noah Dasanaike

PhD Candidate<br>Department of Government<br>Harvard University

Curriculum Vitae

&larr; Back to Home | All Posts

socOCRbench

An OCR benchmark for social science documents

Paper

February 12, 2026

As discussed in a previous post, existing OCR benchmarks<br>are not especially useful for discriminating between models on the kinds of documents that social scientists<br>actually work with. Most benchmarks, like OmniDocBench v1.5, over-index on modern printed text, clean scans, and<br>well-resourced languages. Handwritten census records, historical logbooks, degraded administrative<br>forms, and other ``messy" real-world data are not well represented.

socOCRbench is a small (private) benchmark designed with this gap in mind. It evaluates OCR models on<br>samples across handwriting recognition, table extraction, and printed text recognition. The overall score<br>is the mean of three metrics: NES (Normalized Edit Similarity), chrF (character n-gram F-score) for text,<br>and TEDS (Tree Edit Distance Similarity) for tables. Each ranges from 0 to 1, where 1.0 is perfect.

You can read more about socOCRbench and the motivation behind it in the corresponding working<br>paper.

VLM<br>Classical OCR<br>Open Source<br>Proprietary

&le;3B<br>3-10B<br>10-30B<br>30B+<br>Unknown

v3

v2

v1

Model<br>socOCRbench<br>NES Region<br>W. Europe<br>E. Europe<br>E. Asia<br>S. Asia<br>MENA<br>NES Format<br>HW Text<br>Print Text<br>HW Table<br>chrF<br>W. Europe<br>E. Europe<br>E. Asia<br>S. Asia<br>MENA<br>HW Text<br>Print Text<br>TEDS<br>$/M In<br>$/M Out

Gemini 3.1 Pro (low) VLM Proprietary<br>0.6357<br>0.6577<br>0.68910.66050.56950.73150.6377<br>0.6450<br>0.64860.70220.5843<br>0.6054<br>0.68430.53020.54750.74280.5221<br>0.57520.6835<br>0.6502<br>2.00<br>12.00

Gemini 3 Pro (low) VLM Proprietary<br>0.6249<br>0.6888<br>0.76270.72100.57040.73550.6545<br>0.6350<br>0.67980.80110.4241<br>0.6479<br>0.81080.56120.57200.76620.5291<br>0.60130.7964<br>0.5650<br>2.00<br>12.00

Gemini 3.1 Flash Lite (minimal) VLM Proprietary<br>0.6214<br>0.6502<br>0.74280.64900.52660.73410.5987<br>0.6424<br>0.63910.74870.5395<br>0.5822<br>0.75050.48980.50950.72160.4393<br>0.53640.7312<br>0.6356<br>0.25<br>1.50

Gemini 3.5 Flash (low) VLM Proprietary<br>0.6096<br>0.6566<br>0.71630.68610.54270.69900.6388<br>0.6129<br>0.64930.74680.4424<br>0.6122<br>0.75640.53890.51490.74030.5104<br>0.57540.7358<br>0.5819<br>1.50<br>9.00

Gemini 3.5 Flash (minimal) VLM Proprietary<br>0.6022<br>0.6441<br>0.70810.62580.55940.70830.6190<br>0.6032<br>0.62680.75070.4321<br>0.6001<br>0.74900.48720.53160.72440.5084<br>0.56270.7282<br>0.5828<br>1.50<br>9.00

Gemini 3 Flash (low) VLM Proprietary<br>0.5995<br>0.6453<br>0.70630.61430.54460.71560.6456<br>0.6076<br>0.63920.73000.4537<br>0.6068<br>0.77070.48200.52110.73400.5261<br>0.57310.7378<br>0.5652<br>0.50<br>3.00

Claude Sonnet 4.6 VLM Proprietary<br>0.5980<br>0.5628<br>0.69380.60560.33890.67040.5052<br>0.5764<br>0.53580.70200.4914<br>0.5039<br>0.70700.44030.37600.64700.3492<br>0.46070.6674<br>0.7205<br>3.00<br>15.00

Gemini 3 Flash (minimal) VLM Proprietary<br>0.5920<br>0.6390<br>0.72870.60800.52270.72090.6145<br>0.6145<br>0.63340.73870.4712<br>0.5903<br>0.77660.46200.48890.73770.4863<br>0.56130.7258<br>0.5590<br>0.50<br>3.00

Qwen3.7 Plus VLM Proprietary<br>0.5830<br>0.6140<br>0.71180.59770.56560.70120.4936<br>0.5919<br>0.59060.74530.4399<br>0.5402<br>0.72200.42600.55650.66340.3330<br>0.48930.7052<br>0.6059<br>0.32<br>1.28

Gemini 3.1 Flash Lite (low) VLM Proprietary<br>0.5819<br>0.6359<br>0.73070.63730.51400.72030.5772<br>0.6137<br>0.62380.74750.4698<br>0.5767<br>0.75870.48650.51180.70540.4210<br>0.52950.7357<br>0.5443<br>0.25<br>1.50

Qwen3.5 122B VLM Open Source<br>0.5753<br>0.6078<br>0.71330.60140.55000.68600.4884<br>0.5929<br>0.58940.73420.4551<br>0.5398<br>0.71740.42320.54830.64620.3640<br>0.50130.6864<br>0.5858<br>0.40<br>3.20

Seed 2.0 Pro VLM Proprietary<br>0.5631<br>0.6010<br>0.65310.62720.55540.69930.4697<br>0.5643<br>0.55360.73540.4039<br>0.5513<br>0.68130.47070.57120.69110.3422<br>0.48940.7019<br>0.5554<br>0.47<br>2.37

Qwen3.6 Plus VLM<br>0.5623<br>0.5964<br>0.67840.55660.53680.70000.5102<br>0.5727<br>0.58210.70030.4355<br>0.5335<br>0.68270.40150.53010.67810.3753<br>0.49600.6608<br>0.5689<br>0.33<br>1.95

Qwen3.5 397B VLM Open Source<br>0.5616<br>0.6353<br>0.73990.60680.57810.74120.5105<br>0.6152<br>0.61400.76710.4644<br>0.5716<br>0.74680.44960.57050.70750.3838<br>0.52820.7219<br>0.4879<br>0.60<br>3.60

Qwen3.5 Plus VLM Proprietary<br>0.5576<br>0.6279<br>0.72990.58120.56930.72980.5293<br>0.6069<br>0.61010.75330.4574<br>0.5663<br>0.73880.44180.56990.70170.3789<br>0.52040.7186<br>0.4891<br>0.80<br>2.00

Qwen3.5 397B (thinking) VLM Open Source<br>0.5504<br>0.5935<br>0.69350.56230.57070.66980.4710<br>0.5809<br>0.56300.73210.4477<br>0.5436<br>0.72060.41350.56520.67000.3487<br>0.48920.7112<br>0.5204<br>0.60<br>3.60

Qwen3 VL 235B VLM Open Source<br>0.5478<br>0.5967<br>0.71150.59430.55400.69050.4330<br>0.6021<br>0.56270.73960.5040<br>0.5236<br>0.68930.43020.54470.66780.2859<br>0.46370.6908<br>0.5204<br>0.20<br>0.88

Gemini 2.5 Flash VLM Proprietary<br>0.5446<br>0.5833<br>0.65810.54890.49870.69780.5131<br>0.5720<br>0.53520.72190.4587<br>0.5471<br>0.70120.40540.50000.71280.4163<br>0.47850.7198<br>0.5091<br>0.30<br>2.50

Qwen3.5 27B VLM Open Source<br>0.5417<br>0.5926<br>0.69840.55740.56150.66860.4770<br>0.5786<br>0.58310.70620.4465<br>0.5155<br>0.69530.39460.54600.59980.3418<br>0.48410.6548<br>0.5242<br>0.30<br>2.40

Claude Opus 4.6 VLM Proprietary<br>0.5415<br>0.5568<br>0.68270.60210.34960.65690.4927<br>0.5537<br>0.52960.70380.4279<br>0.5057<br>0.71090.45270.38530.64990.3295<br>0.45680.6781<br>0.5637<br>5.00<br>25.00

Qwen3.5 Plus...