Three ways to smuggle SQLite into Nix | Farid Zakaria’s Blog
Skip to content
The core of nixpkgs-multiverse, when you strip away the Nix API and the CLI, is an index. It is a map from (attribute, version) to the revision that shipped it as a JSON file.11There are actually a few other files that drive other features such as the statistics or “fast mode”, but they are all JSON as well.
$ ls -lh index/<br>-rw-r--r--. 1 fmzakari fmzakari 7.5M Aug 19 13:57 history.json<br>-rw-r--r--. 1 fmzakari fmzakari 5.3M Aug 19 13:57 versions.json
As of 9cc0209, versions.json is 5.3 MiB and history.json is 7.5MiB covering 305,492 package versions across 31,904 packages and 1,534 revisions.
The Nix API loads the JSON files lazily and are all read via builtins.fromJSON:
index = builtins.fromJSON (builtins.readFile ./index/versions.json);
I would like to enrich the data with even more information however it comes at a cost: mo’data, mo’problems.
The goal of the project is to minimize the number of Nixpkgs that are downloaded. If we merely swap fetching huge Nixpkgs for huge JSON, it’s not a clear win.
For now we have to be judicious about what we store in the JSON files and think of clever encoding schemes to make the data small and compact.
If we were not constrained to the Nix builtins, we would leverage established technologies to efficiently encode our dataset that allow multiple query access patterns: databases!
Let’s say we were not restricted to JSON, do we have any other options?
§One lookup costs the whole file
Why are large JSON files so problematic? builtins.fromJSON is eager. There is no lazy JSON in Nix, no streaming parse (i.e. “just give me this one key”). The moment you touch the result you have parsed all 5.3 MB and materialised all 305,492 values on the Nix heap.
In the case of the multiverse, asking for one package costs the same as what asking for all of them.
Note<br>The lookup itself is not the problem. Nix attribute sets are a sorted array,<br>so access is a binary search, not a scan.<br>The cost is entirely in the JSON parse and in allocating the values and downloading a large file.
If we want to do alternate questions over the index, we have to make sure we keep the answers efficiently stored to better match<br>the access pattern.
What we want is obvious. We want a way to efficiently encode the data and a declarative way to define queries: we want SQLite!22nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data.
$ sqlite3 index.db "SELECT version, rev FROM versions WHERE attr='hello'"<br>2.10|728<br>...<br>0.01s, 4 MB
Nix by default cannot do this. Unfortunately there is no builtins.sqlite, although I think there should be…
Turns out though there are knobs we can touch or sources we can patch to get what we want anyways, albeit each one has a caveat. 😈
§One: builtins.exec
I was surprised I did not know about this builtin, and it has been around since release 1.11.9 in April 2017. It is the ultimate escape hatch for a variety of use-cases when you simply can’t get them done with what’s available.
builtins.exec takes a list of strings, runs the program, and parses its stdout as a Nix expression .
It is gated behind a setting that makes it clear it’s unsafe.
$ nix eval --option allow-unsafe-native-code-during-evaluation true \<br>--expr 'builtins.exec [ "/bin/sh" "-c" "echo 42" ]'<br>42
For integration, SQLite is perfectly capable of printing the Nix syntax. We never need a serialisation format in between as we make SQLite emit the attrset directly:
let<br>versionsOf = attr: builtins.exec [<br>"${sqlite}/bin/sqlite3" "-noheader" "-separator" "" "./index.db"<br>''<br>SELECT '{' || group_concat(<br>'"' || version || '" = ' ||<br>COALESCE(CAST(rev AS TEXT), 'null') || ';', ' ')<br>|| '}'<br>FROM versions WHERE attr = '${attr}';<br>''<br>];<br>in<br>versionsOf "hello"
$ nix eval --impure -f query.nix \<br>--option allow-unsafe-native-code-during-evaluation true<br>"2.10" = 728; "2.12" = 822; "2.12.1" = 1369;<br>"2.12.2" = 1486; "2.12.3" = null; "2.7" = 0; "2.8" = 13;
The caveat is that every query is now a fork, an exec, a process image of SQLite, and a re-parse of the output through the Nix parser.<br>If you do not plan to execute many queries that overhead is likely acceptable given the simplicity of the integration.
§Two: builtins.importNative
From researching builtins.exec, I stumbled upon builtins.importNative. It takes a path to a shared object and a symbol name, dlopens it, and calls that symbol. It landed in 1.8 , December 2014.33The C++ field was originally called enableImportNative and was renamed to enableNativeCode for exec.
The shared object must implement the following signature:
extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v);
We can define a new native function that returns the versions for our input:
extern "C" void nix_sqlite_versions(EvalState & state, Value & v)<br>v.mkPrimOp(new PrimOp{<br>.name = "nix_sqlite_versions",<br>.args = {"dbPath", "attr"},<br>.arity = 2,<br>.impl =...