유루메 Yurume: Back in time I used to make a stupid little font called Unison. It was a bitmap…

svg]:size-4 [&>svg]:shrink-0 group-data-[collapsible=icon]:-mt-8 group-data-[collapsible=icon]:opacity-0 " >Timeline<br>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground h-8 text-sm inactive" link="true" >News<br>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground h-8 text-sm inactive" link="true" >Hackers' Pub<br>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground h-8 text-sm inactive" link="true" >Fediverse<br>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground h-8 text-sm inactive" link="true" >Search

svg]:size-4 [&>svg]:shrink-0 group-data-[collapsible=icon]:-mt-8 group-data-[collapsible=icon]:opacity-0 " >Account<br>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground h-8 text-sm cursor-pointer " >Use old UI

Invitation tree<br>Code of conduct &middot; Privacy policy<br>Android &middot; iOS/iPadOS<br>The source code of this website is available on GitHub repository under the AGPL 3.0 license. v0.2.0+dcf0d40a

유Y<br>@yurume@hackers.pub<br>34 minutes ago&middot;<br>Back in time I used to make a stupid little font called Unison. It was a bitmap-vector hybrid font defined by text-based font description files which then get compiled by Python script. During the development of Unison, I had so frustrating bug that I had to write this comment in addition to its workaround:

def custom_sort_key((name, _)):<br># what, the, real, fuck.<br># it seems that Uniscribe has some bug with Hangul and possibly more scripts:<br># some characters, when they are located in specific glyph indices, are correctly<br># mapped via ScriptGetCMap but considered to be missing via ScriptShape.<br># combined with SSA_FALLBACK it causes the wrong *and* inconsistent fallback behavior.<br># given that the range of those indices abruptly end with 2^n boundaries,<br># I strongly suspect that this is something to do with the internal lookup mechanism.<br># for now, reorder problematic scripts to (empirically) avoid the problem... *sigh*<br>if not name.startswith('uni'): return (2, name)<br>try: c = int(name[3:], 16)<br>except ValueError: return (2, name)<br>return (0 if 0x1100 c 0x11ff or 0x3130 c 0x318f or<br>0xa960 c 0xa97f or 0xac00 c 0xd7ff else 1, c)<br>Uniscribe isn't exactly a household name so here's a brief description. It's an internal name for Windows' font rendering---or more accurately, shaping---system. OpenType fonts consist of a list of zero-numbered glyphs and a character mapping (cmap) from a character code, typically Unicode, to the glyph index. The problem was that Uniscribe correctly found a glyph from the cmap but failed to actually make use of that glyph. It is easy to demonstrate with Windows Notepad; you can configure its font, but it will use a fallback system font if the glyph doesn't exist in the specified font. When the problematic character was typed in, every non-space character before that character would suddenly switch to the fallback font!

For a long time the bug was thought to be Windows' because of its peculiar behavior over the problematic glyph indices. The bug starts to appear around the glyph index 1200 and persists right until the glyph index 2048 at which the bug suddenly disappears. Also I found a workaround to relocate problematic glyphs to lower glyph indices. 2048 = 2^11, what a suspicious number. My gut reaction was that it should be something to do with binary search, as OpenType's cmap contains related fields, but my compiler internally used fontTools for font creation and they should have been automatically calculated. So the culprit must have been Windows instead.

As there was a workaround I had no reason to continue at this point, so I documented this weird behavior and moved on. Fast forward to today: after 11 busy years, LLM has completely transformed how we do programming and I realized that LLM should be able to solve this 11-year-old mystery. You know, in this agentic era any little idea is worthy to execute. So I asked and got some additional infos:

The bug was traced back to GetCharacterPlacementW, which is a GDI function. Note that GDI is a part of Windows kernel (due to the performance concern) so I ended up with kernel debugging! Dang it!

Workaround turned out to be incomplete, as it only moved problematic glyph indices from here to there.

More precisely, problematic glyph indices were two ranges: [1216, 1664) and [1792, 2048). Every single end point here are multiples of 64. So a hypothesis was set up: there must have been some list of 64-entry bitmaps, and for some reason they were partially initialized. A partially initialized bitmap causes other glyphs in that bitmap to fail, hence this behavior.

I was able to...