Extending MySQL with VillageSQL | Max De Marzi

Max De Marzi

Graphs, Graphs, and nothing but the Graphs

Home<br>About

Contact

Services

Videos

May 21 2026

Leave a comment

By maxdemarzi

database

Extending MySQL with VillageSQL

One of the things that made me fall head over heals for Neo4j so many years ago was just how extensible it was. If the database engineering team was busy rebuilding the clustering feature for the third time and didn’t have time to take care of my feature requests… I could just add them myself. Not to Neo4j directly, no that would have been a horrible mess. Instead I could add any feature I wanted as an "Unmanaged Extension". Later on they became Cypher Stored Procedures, but it was basically the same thing. You had access to the top level Java API that dealt with Nodes and Edges. You could use the Traversal API that dealt with Paths….and if you were feeling extra spicy that day you could go down to the Storage API that dealt with Cursors over raw bytes.

I had spent prior jobs working with Oracle and Microsoft SQL Server so I never had that kind of power and freedom before. Well, it took a long time, but that power has come to MySQL in the form of a change tracking fork called VillageSQL. There are already a bunch of extensions that add UUID, Network Address custom types, Cryptographic Functions, Multi-Dimensional Geometry as well as AI helpers. So of course I had to try it out. I decided to add an extension for one of my other great loves, the Roaring Bitmap data structure.

You don’t have to start from an empty github repository, they provide a template extension repository to get you going in the right direction. Check the docs on how to build extensions as well. There is a good chance by the time you read this they will be on a new branch so double check that before you dive in.

I made a clone of the template repository and brought up Visual Studio. I haven’t written any C++ in a little while, so I decided to Vibe Code this like all the cool kids. I don’t recall my initial prompt exactly but it was something like: "Take a look at Roaring64Map on https://github.com/RoaringBitmap/CRoaring . I want you to build a village sql extension using protocol 2 for roaring bitmaps that adds the common and set operation functions. See https://villagesql.com/docs/mysql-8.4/0.0.4-dev/extensions-or-plugins for more details." It didn’t magically do all that. It just created the set operation UNION. Gotta feed the thing more tokens for it to do more work. So after a few more prompts we were in business.

It was nice enough to add mysql tests and result files (even if they were not actually ran until a test script was created). I ran into a few problems. The first was thinking I could use the SQL CAST keyword. But that doesn’t work, instead I have to create my roaring bitmap from a string method like this:

mysql> SELECT CAST('{1,5,10,255,1000}' AS ROARING64) AS my_bitmap;<br>ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ROARING64) AS my_bitmap' at line 1

SELECT ROARING64::from_string('{1,5,10,255,1000}') AS my_bitmap; -- Works!

Once I got past that error, the second error was not displayed but eaten by the log instead.

[ERROR] [MY-010666] [Server] VillageSQL: 'field_length (0) != persisted_length (-1) for column val (type vsql_roaring_bitmap.ROARING64)'

The Roaring Bitmap data structure doesn’t have a set size. It changes depending on how much data it has and what the layout of the internals of it are. In this case we are converting the data structure to a String for display, so this was fixed by setting the size of the string to the StringResult out parameter before ending the function:

void roaring64_to_string(CustomArg in, StringResult out) {<br>if (in.is_null()) { out.set_null(); return; }

Roaring64Map bitmap;<br>std::string error_msg;<br>if (!deserializeRoaring64Map(in, bitmap, error_msg)) {<br>out.error(error_msg);<br>return;

std::string value = roaring64ToString(bitmap);<br>auto buf = out.buffer();<br>size_t len = value.size();<br>if (len > buf.size()) {<br>out.error("ROARING64: output buffer too small");<br>return;<br>memcpy(buf.data(), value.data(), len);<br>out.set_length(static_cast(len));

There were some minor issues on the VillageSQL side as well that Tomas Ulin took care of for me to get the Roaring Bitmap Village SQL extension to work with MySQL Stored Procedures. These have been merged so you don’t have to worry about that. I am using the protocol 2 include-dev headers so I had to configure my extension with:

-DVillageSQL_USE_DEV_HEADERS=ON

But by the time you read this, you may not need to do that.

So first we create our custom type and add a few required methods, the roaring64_to_string is a reference to the code above.

constexpr auto ROARING64_TYPE...