The Slot Machine in Your Pocket: Variable Reinforcement

⠋cat post.md

init0%

*]:pointer-events-auto<br>translate-x-0 opacity-100"><br>Back to Blog<br>"The organism is always right."<br>B. F. Skinner, 1977

Photo: Luis Dantas, public domain, via Wikimedia Commons.

In the mid-twentieth century, running pigeons and rats through the operant chambers that now bear his name, a psychologist named B. F. Skinner pinned down the single most powerful behavioral hook ever found. He did not invent it. He measured it. And the thing he measured is now in your pocket, refreshing when you pull it, lighting up when you swipe it, buzzing when you least expect it.

The discovery sounds like it should be backwards. Skinner found that if you want an animal to perform a behavior over and over, with maximum persistence and minimum chance of stopping, you should not reward it every time. You should reward it at random. A pigeon paid for every peck will peck steadily and, the moment you stop paying, give up almost immediately. A pigeon paid for a random fraction of its pecks will peck like a machine, and when you stop paying entirely, it will keep going far longer, sometimes thousands of times, refusing to believe the reward is gone.

A reward you can predict is a transaction. A reward you cannot predict is a compulsion. This post is about why, and the why turns out to be a clean piece of probability that every slot machine and every social app is built on.

1. The phenomenon

Compare two machines.

Machine A is a vending machine. You put in a coin, you get a snack, every time, reliably. You use it when you want a snack, and not otherwise. Nobody has ever been addicted to a vending machine. When it breaks and stops dispensing, you notice on the very first failure and walk away.

Machine B is a slot machine. You put in a coin and sometimes something happens. You cannot predict which pull pays. Most pay nothing; a few pay a little; rarely, one pays a lot. People will sit at Machine B for hours, feeding it continuously, and when it stops paying they keep pulling, because they cannot tell a dry spell from a dead machine.

Same physical action (insert, actuate, observe). Opposite behavior. The only difference is the schedule on which the reward arrives. Skinner gave these schedules names. Machine A is continuous reinforcement. Machine B is a variable-ratio schedule : reinforcement after an unpredictable number of responses. And the variable-ratio schedule produces, in his words, the highest rate of responding and the greatest resistance to extinction of any schedule known. It is the closest thing behavioral science has to a universal lever.

2. The mathematical model

What is actually happening on Machine B? Strip it to the bone: each pull is an independent trial that pays with some probability ppp. The number of pulls until your next win is then a geometric random variable . The probability that your first win comes on exactly pull kkk is

Pr⁡(K=k)=(1−p) k−1 p,k=1,2,3,…\Pr(K = k) = (1-p)^{\,k-1}\,p, \qquad k = 1, 2, 3, \dotsPr(K=k)=(1−p)k−1p,k=1,2,3,…<br>and the average wait between wins is

E[K]=1p.\mathbb{E}[K] = \frac{1}{p}.E[K]=p1​.<br>So a machine with p=1/10p = 1/10p=1/10 pays, on average, once every ten pulls. But "on average" hides the whole psychological payload. The geometric distribution has a defining property, and it is the source of the entire effect: it is memoryless .

Pr⁡(K>a+b∣K>a)=Pr⁡(K>b).\Pr(K > a + b \mid K > a) = \Pr(K > b).Pr(K>a+b∣K>a)=Pr(K>b).<br>In words: no matter how long you have gone without a win, your chance of winning on the next pull is exactly the same as it always was, ppp. The machine has no memory of your dry streak, and crucially, the streak carries no information about when the next win arrives. There is no "due." There is no building pressure that must release. Every pull is the first pull, forever.

This is why you cannot stop. With a predictable schedule, you always know how far you are from the next reward, so you can reason about whether to continue. With a memoryless schedule, the next pull always has the full original probability of paying, so there is never a moment when quitting is obviously correct. The reward is permanently one pull away, and one pull is cheap.

The deep dive: resistance to extinction and the prediction error

Two deeper facts explain why variable reinforcement is not just engaging but specifically hard to quit.

Why random schedules resist extinction. "Extinction" is the technical term for a behavior dying out once rewards stop. On a continuous schedule (pay every time), extinction is fast: the very first unrewarded response is a glaring anomaly ("the machine is broken"), and a few more confirm it. The learner has a sharp, low-variance expectation, so a violation is detected almost immediately.

On a variable-ratio schedule the learner's experience of "many responses with no reward" is indistinguishable from a normal dry streak. Formally, under the geometric model a run of mmm...