Breaking up (lines) is hard to do
elements just below this comment. Here's what they're for:
* A link with rel="canonical" tells a search engine what URL to use<br>for this content. That's important if there are two or more URLs<br>which could get the same content.
* Links with rel="prev" or "next" are to help out navigation. They<br>show what's available within a sequence of pages (like my blog<br>posts).
-->
Skip to content
Breaking up (lines) is hard to do
Published on:
August 10, 2026
Categories:
Python, Unicode
Here’s a seemingly simple question: given a chunk of multi-line text, how do you split it and return an array whose members are the constituent lines of the text?
Hopefully, your first instinct is to reach for some sort of standard-library function, maybe something like the splitlines() method of Python’s str type. Because it turns out this “simple” question is actually pretty complex to answer! For example, quite some time ago I read a post by William Woodruff pointing out the surprising discovery that Python treats up to eleven different Unicode code points or code point sequences as indicating a line break.
At the time I meant to write about that, but a lot of other things started fighting for my time, and it’s only now that I’m finally digging it out of my drafts. Still, better late than never, so today let’s dig into some of the many ways there are to break a line of text and how they’ve been standardized and specified and ultimately wound up in the set Python uses.
In the beginning…
Once upon a time, there was ASCII. Of course there were other things before ASCII, and alongside ASCII, but for today’s discussion we really only need to go back to ASCII; if you want the full history of physical teletypes, how they evolved from typewriters and influenced character sets for computing and so on, I suggest Wikipedia. Here, I’m just going to gloss over and simplify a lot of that to focus on the topic at hand.
So. Once upon a time, there was ASCII. And it wound up being incredibly influential and important in computing, to an extent other early character sets couldn’t match. And because it was used on computers which used teletypes (basically electronic typewriters connected as input/output devices) as a user interface, it contained control characters for sending commands to the teletype. Such as a LINE FEED (byte value 0x0A) to advance the paper vertically to the next line, and a CARRIAGE RETURN (byte value 0x0D) to re-align the print head/carriage with the horizontal start point of the line.
These are often abbreviated LF and CR (or by their C-family escape sequences \n and \r, respectively), and you might think that since physically advancing a typewriter-style device to be ready to print the next line requires both operations, that would have just become the universal way everybody did new lines. Or at least the universal way everybody did them in English, or in the US, where ASCII dominated. Right?
Well, nothing is ever that simple. Physical teletypes apparently benefited from the two-character approach (as opposed to a single “new line” character) because it gave them time to physically move everything into the right position. But as virtual teletypes—“printing” to a television-like display instead of to paper—became more common, that was less of an issue. So there were multiple possible options for representing line breaks, and several of them showed up in historical systems. For example:
CP/M used CR LF. And so MS-DOS, which aimed for compatibility with it, used CR LF too. And so Microsoft Windows, which wanted to be compatible with MS-DOS, also used it.
Meanwhile, Multics chose to use just LF with no CR, and Unix went along with that choice.
But Commodore and Apple and many others went yet another way and used plain CR , with no LF.
This meant “plain text” was not easily portable between these various systems, since none of them could agree on how to represent a line break. Which led to one of my all-time favorite programming jokes, in the infamous “NOT the comp.text.sgml FAQ” document:
Q. What’s an RE?
A. RE is an acronym for Record End, which is sort of like a newline, only different. Goldfarb’s First Law of Text Processing states that:
"… if a text processing system has bugs, at least one of them will have to do with the handling of input line endings."
[The Handbook, footnote p. 321]
The Record End concept was introduced to make sure that SGML parsers don’t violate Goldfarb’s First Law.
(for the uninitiated, Charles Goldfarb created SGML)
Anyway, over twenty years ago Python tried (in Python 2.3) to smooth this over by introducing “universal newline” mode for opening files, which accepts all three options: a plain \n (Unix), or a plain \r (classic Mac), or an \r\n sequence (DOS and Windows) will all be interpreted as line breaks.
But even in ASCII there there are other ways of breaking a line. For example, at byte value 0x0C ASCII includes the FORM FEED control...