Pull 12.4M US business registrations from state Socrata open-data portals (free, commercial-OK). Python stdlib, no deps. · GitHub

/" data-turbo-transient="true" />

Skip to content

-->

Search Gists

Search Gists

Sign in

Sign up

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

Instantly share code, notes, and snippets.

BhackerJ/README.md

Created<br>July 7, 2026 04:12

Show Gist options

Download ZIP

Star

(0)

You must be signed in to star a gist

Fork

(0)

You must be signed in to fork a gist

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/BhackerJ/22a35fead8764782189c7a20c5ded0d7.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-6213e22b-6284-4032-a1bc-b40306b47ac6" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-sized-down" />

Save BhackerJ/22a35fead8764782189c7a20c5ded0d7 to your computer and use it in GitHub Desktop.

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/BhackerJ/22a35fead8764782189c7a20c5ded0d7.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-8ec1b721-6e5a-48ba-89f4-b5ba82756745" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-original" />

Save BhackerJ/22a35fead8764782189c7a20c5ded0d7 to your computer and use it in GitHub Desktop.

Download ZIP

Pull 12.4M US business registrations from state Socrata open-data portals (free, commercial-OK). Python stdlib, no deps.

Raw

README.md

US business-entity registries from state open-data portals

Pulls complete business registries (every LLC, corporation, nonprofit) from the five US states that publish them as free, commercially usable open data on Socrata portals:

State<br>Dataset<br>Records<br>License

New York<br>n9v6-gdp6<br>4.22M<br>NY Open Data, commercial OK

Colorado<br>4ykn-tg5h<br>3.06M<br>Public Domain

Pennsylvania<br>xvd7-5r2c<br>2.31M entities<br>Public Domain

Oregon<br>tckn-sxa6<br>1.56M<br>Public record

Connecticut<br>n7gp-d28j<br>1.28M<br>Public Domain

~12.4 million entities total — names, entity types, formation dates, addresses, registered agents. Normalized to one unified JSONL schema, raw row preserved under _raw.

Python 3 stdlib only, no dependencies.

Usage

python3 fetch-us-business-entities.py --sample # 1000 rows/state, quick validate<br>python3 fetch-us-business-entities.py # full pull (~7-8h anonymous)<br>python3 fetch-us-business-entities.py --state co # one state<br>APP_TOKEN=xxx python3 fetch-us-business-entities.py # free Socrata app token = higher rate limits

Output: data/us-business-entities/_entities.jsonl. Interrupted pulls resume automatically (line-count offset, append mode).

Rate-limit notes (measured June 2026, anonymous)

Plain $limit/$offset paging: ~500 rows/sec — the best you'll do without a token. Deep offsets are fine (offset 1M returns in ~3s); the bottleneck is per-page transfer, not depth.

Keyset paging via $select=:*,*: dead end — 50k-row page takes 200s+ then times out.

CSV export (/api/views/{id}/rows.csv): ~40 rows/sec, generated server-side on demand. Slower than JSON.

Gotchas

Row granularity : Oregon = row per associated name, Pennsylvania = row per officer. Dedup on registry_number / filing_number before counting entities.

CSV labels ≠ SODA field names (Business_City vs billingcity). Map via /api/views/{id}.json → columns[].fieldName.

Connecticut agents live in companion datasets (Agent/Principal Details), joined on accountnumber.

States you can't get

CA/TX/DE: paid. Delaware has no bulk product at any price. FL: free but fixed-width FTP (Sunbiz), needs its own parser. OH: bot wall, monthly files. IA: migrated off Socrata, legacy endpoints 404. HI: data exists (~442k) but no explicit license tag — left disabled.

Background & full write-up: https://www.deciqai.com/blog/us-business-registries-open-data

Raw

fetch-us-business-entities.py

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.<br>Learn more about bidirectional Unicode...