Scratch a simple data model, find a complex one | Jon Skeet's coding blog

Search for:

I seem to have a knack for discovering corner cases – or in many situations being a corner case.

One category of this is where I find myself using a data model which appears simple to start with – and then the real world interferes. I’ve always found this sort of thing interesting, and recently I’ve come across a pretty good example which I thought I’d share. This also gives an example of some of the thought processes I use when scoping how I model data.

I’m trying to model the text of the Bible, for reasons I won’t go into. BibleGateway is my "source of truth" here, and all the screenshots in this post are from that site (with appropriate links).

Initial assumptions, scoping and simplifications

Note: this section is not all true. It’s how I approached the data model to start with. Later sections will show where it breaks down.

There are multiple translations of the Bible. I’m scoping my model to only English translations. I’m not interested in "first edition" vs "second edition" etc, but I do want to be able to differentiate between "New International Version" and "New International Version – UK" etc.

Each translation is made up of several books (Genesis to Revelation). While we could split this into "Old Testament" and "New Testament" (and some other categories, potentially) I don’t need that categorisation, so it won’t be part of the model. Different translations consist of different books, as some translations include the Apocrypha and others don’t.

Different translations may refer to the same book with different titles. For exmaple, "Song of Songs" is also known as "Canticle of Canticles" or "the Song of Solomon"; likewise "the Book of Wisdom" is also known as "the Wisdom of Solomon" and "Sirach" is also known as "Ecclesiasticus" (and some other titles, apparently). I don’t particularly need to know which title is used in each translation, but I do need a canonical representation. (If a user says they want to see a passage in "Sirach" or "Ecclesiasticus" I want to give the same results in either case, regardless of what the translation calls it.)

Each book is split into chapters, and each chapter is split into verses. Each book starts with chapter 1 and proceeds in the obvious way; each chapter starts with verse 1 and proceeds in the obvious way. Different translations may have number of verses for the same chapter, but have the same number of chapters for the same book (modulo books which are augmented by the Apocrypha). Some verses have optional splits (e.g. 25a, 25b, 25c) which may not be consistent between translations. For now, I’ll deem the splits to be out of scope for the data model (at least to start with).

The text within each verse may have some formatting details such as indentation. Sometimes there are headings within chapters. Some translations may come with cross references and commentary notes. All of this is out of scope of at least this blog post.

In other words, I might expect a simple C# representation to look something like this:

// Note: BookId is an enum or equivalent, so that "Genesis" uses the same BookId in all translations.

public record Bible(string Id, string Description, ImmutableArray Books);<br>// Chapters[0] = chapter 1 etc.<br>public record Book(BookId Id, ImmutableArray Chapters);<br>// Verses[0] = verse 1 etc.<br>public record Chapter(ImmutableArray Verses);

The complex reality

Of course, this post wouldn’t exist if the data model were really that straightforward. The very top-level aspects – a Bible with an ID, a description, and a sequence of books – is fine. But when we get to the "a book is a sequence of chapters" and "a chapter is a sequence of verses" aspects, it turns out life is more complicated.

It seems unlikely that I’ve discovered all the ways in which the simple data model is broken, but here are the ones I’ve encountered so far.

Verse ranges

Not all Bible translations attempt to translate each verse directly, in the original textual order. For example, The Message translates "chunks" of text at a time. Here’s the start of John’s Gospel in The Message:

As you can see, each chunk of text is associated with a range of verses (1-2, 3-5, 6-8) rather than a single verse.

That already completely messes up our data model. We can’t just use a sequence of verses in the chapter with each verse being represented by a single string. We need a more complex representation with a dedicated type for "part of a chapter". It could look something like this:

public record Bible(string Id, string Description, ImmutableArray Books);<br>public record Book(BookId Id, ImmutableArray Chapters);<br>public record Chapter(ImmutableArray Sections);<br>public record ChapterSection(string Text, int StartVerse, int EndVerse);

Note that at this point, if a user performs a search based on a book, chapter and range of verses, they might see text that doesn’t really belong in that range.

For example,...