Amor Fati Entropy - by Hollis Robbins - Anecdotal Value

Anecdotal Value

SubscribeSign in

Amor Fati Entropy<br>Avoid eternal return

Hollis Robbins<br>Jul 07, 2026

Share

The way I used to teach Claude Shannon’s Mathematical Theory of Communication long ago is this: if someone slips a card with a heart on it under your dorm room door on February 14, it means something, but it may not mean a lot. People expect cards with hearts slipped under their doors on Valentine’s Day. But if someone slipped a card with a heart under your door on January 14, it would mean more!<br>When you pick up the card, the uncertainty about meaning is reduced when you remember it is February 14. You may have gone to bed with a certain expectation about notes, a conditional distribution that may or may not have taken Valentine’s Day into account. If you did, then on February 14, p(love note | date) is already high, so the note’s arrival collapses very little uncertainty. Few bits move. But on January 14, a card under your door is a low-probability event under the receiver’s distribution, its surprisal (−log p) is large, and the receiver is forced into a significant posterior update. Something occasioned the card, and the occasion is not the calendar. Information is carried by the improbability. Probability on February 14 is manufactured by the date; the improbability on January 14 is not exactly manufactured by a date but it is not unrelated to the date.<br>All of this to say: same channel (under the door), same words (card with a heart), completely different information, or bit count.<br>All of this went through my head on Saturday, July 4, with the scores if not hundreds of patriotic posts that one doesn’t want to completely ignore coming across my X feed. The calendar context is shared by every poster and reader, so everybody’s conditional distribution concentrates on patriotic content before opening the app. The per-token perplexity of a patriotic post on July 4 is, for any reader, near the floor. In compression terms, the post is maximally compressible.<br>Shannon estimated ordinary English at roughly 50 % redundant in the 1948 paper; his 1951 prediction experiments pushed the estimate toward 75% . My July 4 feed was 100% redundant, even including the newly canonical dissenting variant (Frederick Douglass “What to the Slave,” etc.). H(post | date, author) approaches zero for almost every account.<br>It may be that July 4 is at X’s annual entropy minimum, lower even than Christmas and New Year’s and Easter and certainly than Halloween. World Cup posting entropy is higher than expected, at least for now. July 4 posting entropy is measurable, in principle, if you look at topic-distribution entropy of posts by calendar day, or mean per-token perplexity of posts scored by a language model. That would show the July 4 trough.1<br>What does this mean? First, that patriotic posts on X (or elsewhere) are not in fact transmitting on the channel where they appear. A message a receiver can easily predict carries no bits about the sender’s beliefs, or at least beliefs about the U.S.A., whatever the post says. What the patriotic message transmits is roughly one bit of participation and belief in participation. The post confirms the account is live, that the poster is observing the convention of patriotic posts on July 4. Like the congregation’s “amen” at the end of a sermon, the post transmits nothing semantically beyond a commitment to membership and simultaneity.2<br>So, on a day when so much is fixed, say Valentine’s Day or Independence Day, the best information exists in deviations, so a good X reader should allocate attention the way a decoder allocates bits: entirely to the residual. Silence from an account—particularly a political account whose prior history predicts a post—is a high-surprise event. A patriotic post from an account that generally scoffs at rituals is interesting. But pretty much everything was redundant and spent channel capacity on a drumbeat of repeated confirmations that the poster is the kind of poster the reader already believed would post.3<br>Now I am not against patriotism (I love my country!) but I have discovered in myself an irrepressible pro-entropy streak that just won’t let me post expected things. I say my amens in synagogue or church, I keep the sabbath, and I send birthday greetings on Facebook, when I visit once a week or so. Facebook is very helpful about those reminders. But as years go by, I have a harder time bringing myself to participate in low entropy social media posting.<br>I blame Claude, of course. And Shannon. Claude tells me regularly I am asking questions outside its corpus. If everyone says XYZ and I’m saying no, it’s ABC, Claude is surprised. My questions diverge from its priors. This is cross-entropy. My work with Claude is largely about things I know more than Claude about and my distribution P diverges from Claude’s distribution Q learned from the corpus. Cross-entropy H(P, Q) is the average number of bits...