Package Name Prefixes | Andrew Nesbitt

PEP 752 was accepted at the end of June. It lets an organisation apply to PyPI to reserve a package name prefix, after which new uploads matching that prefix are rejected unless they come from the grant holder. Existing packages are grandfathered. The motivation is impersonation: someone publishing malware as google-cloud-something or apache-something gets the vendor’s name in front of it. The PEP names four example prefixes: google-cloud-, opentelemetry-, apache-airflow-providers-, and types-.

I pulled every package name from PyPI (912,665 of them, via ecosyste.ms) and counted prefixes. Those four account for 264, 271, 154, and 1,295 packages respectively, and each is published almost entirely by one organisation. Google publishes google-cloud-*, the OpenTelemetry project publishes opentelemetry-*, the Apache Airflow team publishes apache-airflow-providers-*, and the typeshed maintainers publish types-*. Prefix reservation formalises what those publishers already do; one of the PEP’s authors, Jarek Potiuk, is on the Airflow PMC.

There are 13,934 prefixes on PyPI shared by five or more packages. The biggest is django-* with 17,450, published by thousands of different authors at a rate of about a thousand a year since 2010. flask-* has 2,815, pytest-* has 2,137, mkdocs-* has 764, sphinxcontrib-* has 370. collective.*, the Plone community’s shared prefix using Python’s namespace package machinery, has 1,436 packages and has been in use since at least 2008. Anyone publishes a Django app as django-something, a pytest plugin as pytest-something, a Sphinx extension as sphinxcontrib-something, and the host project’s documentation tells them to.

An earlier draft of PEP 752 had a mode for this, letting a grant holder mark a prefix as open so that anyone could still publish under it while the grant holder’s own packages got a visual indicator. It was removed as having “insufficient motivation and the fact that repositories could technically satisfy such use cases with standard grant semantics.” Multiple organisations can share a grant, which handles a prefix split across a few organisations. Under the accepted design, if the Django Software Foundation reserved django-, the existing 17,450 packages would keep publishing new versions and the roughly one thousand new django-* packages that would otherwise arrive next year would be blocked. I compared namespace designs across registries earlier this year; NuGet, crates.io and npm have each attached something to prefixes already.

Seven registries

Splitting package names on runs of -, _ and . and grouping by the first token:

packages<br>prefixes with ≥5 packages<br>packages with a shared prefix<br>≥100<br>≥500

pypi.org<br>912,665<br>13,934<br>41%<br>427<br>66

crates.io<br>311,301<br>9,161<br>54%<br>166<br>19

rubygems.org<br>209,853<br>4,214<br>46%<br>132<br>18

hex.pm<br>22,373<br>382<br>32%<br>12

hackage.haskell.org<br>19,345<br>521<br>41%

npmjs.org (unscoped)<br>2,678,188<br>37,460<br>68%<br>2,604<br>468

nuget.org<br>828,361<br>22,377<br>70%<br>690<br>72

npm has had @scope/name since 2014 so its row counts only the 63% of packages that are unscoped. Between a third and two-thirds of every registry’s packages share a first token with at least four others. The counts here are first-token only; multi-token prefixes like apache-airflow-providers- are counted under apache-* in the table and separately when named in the text.

Open plugin conventions, where many authors publish extensions named after a host project, are the largest category among the biggest prefixes on every registry: django-* (17,450 pypi), react-* (86,142 npm), jekyll-* (1,658 rubygems), bevy-* (1,870 crates), phoenix_* (245 hex), servant-* (152 hackage). Some of these are functional, meaning the name has to match the pattern for the host tool to load the plugin: cargo runs cargo-foo as the cargo foo subcommand, ESLint’s config resolver matches eslint-plugin-foo, pytest loads pytest-foo via entry points.

Generated SDK families, where one organisation publishes hundreds of packages from an API spec, are the next largest. aws-sdk-* (473 crates, 481 gems, 92 pypi), google-cloud-* (258 crates, 510 gems, 264 pypi, 5 hackage), azure-mgmt-* (273 crates, 124 gems, 334 pypi). The Odoo Community Association publishes about 20,000 packages on PyPI across seven versioned prefix trees (odoo8-addon-* through odoo14-addon-* plus odoo-addon-*), which is the largest coordinated prefix use I found on any registry. On npm the generated families are the ones that adopt scopes: @types/* 11,388, @stdlib/* 5,555, @fontsource/* 2,122.

Language and framework tags account for another large block: python-* (6,278), rust-* (1,692), ex_* (1,123 hex, the Elixir convention), node-* (23,389 npm), is-* (3,754 npm, predicate functions), use-* (4,317 npm, React hooks). Hackage has a set of these that mirror module paths (data-*, network-*, language-*).

Getting-started guides produce their own prefixes: hola-* on rubygems has 764 packages from people following the make your own gem guide,...