Pandoc: What survives a conversion? – DerEuroMark
Skip to content<br>Skip to content
☾☀
I have been working on a new (post-markdown) markup language Carve.<br>And I wanted to know how it performs for input and output. Especially compared to Markdown and Djot, which mainly inspired me here.
And it happens that Pandoc released 3.10.2 recently.
Everyone who converts documents knows the process loses things.<br>Tables are fine in most formats. LaTeX keeps everything. Word is a black hole.
Some of that is true. Some is backwards. You cannot tell by looking, because the<br>failure is quiet – the file still opens, the paragraphs are all there, and the<br>one attribute that carried your meaning is gone.
Why the initial experiment failed
Write a document, convert it, read it back, diff against the original. That is<br>maybe the intuitive way – but it does not really work.
Every pandoc writer normalizes as it serializes. Hand the markdown writer a<br>document and it rewraps your lines, renumbers your ordered lists, moves your link<br>definitions, and writes emphasis with * even where you typed _. None of that<br>changed the meaning, and all of it shows up in a diff.
So the diff is mostly noise. Tighten it and you are reading reformatting; loosen<br>it until the reformatting drops out and you have loosened it past the thing you<br>were trying to detect.
The way around it is to stop comparing against the input at all.
The approach
Every probe is a pair of documents differing in exactly one feature: a rich one<br>that uses it, a degraded one that does not. Both go through the same writer, same<br>options, same run. Two questions follow.
expressed<br>Are the outputs different at all? A writer that emits byte-identical text<br>for a document with the feature and one without cannot express it. Nothing was<br>encoded, so nothing can be recovered.<br>unchanged<br>Read both outputs back. Compare the resulting syntax trees node by node. If<br>the trees are identical, the round trip ate it.
Both documents take the same path, so canonical styling lands on both and<br>cancels. When a writer scores “cannot express”, pandoc emitted the same bytes for<br>two different documents. No formatting preference produces that.
Two settings that were bugs first
--wrap=preserve on every writer. Without it a wrapping writer turns a soft<br>break into a space, and the probe reports that the format cannot express a<br>distinction it handles perfectly well.
--resource-path on every writer, and the same image handed to the pandoc<br>server. An image pandoc cannot resolve gets its src left alone, which<br>round-trips to a clean false pass. Once it resolves, epub embeds the file and<br>rewrites the path to media/file0.png, which is the answer and the<br>failing one.
Both turned up as results that looked too good.
The gap
Chart dataexpressedsurvives a round tripcarve6464markdown5946html6342commonmark_x5739epub6234djot5233latex5424gfm5024docx5323commonmark5621rtf4015pptx400<br>Figure 1: Light bar is what the writer can express, dark bar is what comes back<br>unchanged, and the distance between them is the loss.
Table 1: 66 probes, pandoc 3.10.2.<br>FormatExpressedUnchangedLost
native (the AST itself)66660<br>Carve, AST lane65650<br>Carve, source lane64640<br>markdown594613<br>html634221<br>commonmark_x573918<br>epub623428<br>djot523319<br>latex542430<br>gfm502426<br>docx532330<br>commonmark562135<br>rtf401525<br>pptx40040
HTML expresses 63 of 66 and returns 42. It is a rendering target: the writer<br>encodes for display, and the reader has to guess intent back out of presentation.<br>A third of what it writes does not survive that guess.
LaTeX drops 30. Word drops 30. CommonMark drops 35 – the specification written to<br>make Markdown unambiguous, which turns out to be a separate question from making<br>it expressive.
pptx is the clean case. Expresses 40, returns zero. Every distinction it can draw<br>is one no reader recovers. Converting out of PowerPoint is archaeology.
Where Carve lands
Carve expresses 64 and returns 64. Through the AST lane, 65 and 65.
The rank matters less than the shape: both columns are the same number. Almost<br>everything on that table can write more than it can read, and the difference is<br>where your document quietly changes meaning. Carve keeps what it can express.
That comes from one decision, made early and not revisited – awkward things get<br>syntax rather than becoming an extension somebody bolts on. Per-cell table<br>alignment. Colspan and rowspan. Captions. Attributes on nearly every node.<br>Footnotes, definition lists, line blocks, raw blocks addressed to one target.
A feature with a spelling can be found again. A feature without one gets degraded<br>into prose by the writer, and parser quality is irrelevant at that point.
The two it drops
ol_example, pandoc’s @-style example lists, and table_multibody, more than<br>one body group in a single table. The AST lane loses only the first.
Neither has a Carve spelling, but those are supposed to stay internal, afaik, anyway.
native sits above Carve on the table. That row is pandoc’s own syntax...