SCROLL

SCROLL

Synchronized Change Replication Over Line Logs

Because we can do better than webhooks as a synchronization mechanism

draft-scroll-protocol-00 · request for comments · no implementations in the wild, one document

TL;DR

Webhooks have well-known pitfalls when used as a synchronization mechanism. Instead, you open an HTTP connection and receive objects as<br>NDJSON (JSON objects separated by newlines), with a few clever conventions: every event carries a<br>cursor, so resuming from where you left off is trivial, and every provider follows the same rules,<br>so one client can consume many providers. SCROLL says nothing about how your data should be<br>shaped — it is a delivery mechanism.

Press play on each lane to watch the same six events, the same outage, delivered both ways. The<br>consumer goes down for a moment in each lane; only one of them notices anything went wrong —<br>and it's the wrong one that doesn't.

WEBHOOKS &middot; provider pushes to your endpointplay

SCROLL &middot; consumer reads the provider's logplay

speed &times;0.5

What it is

A provider exposes each collection (customers, subscriptions, …) as an<br>ordered, cursor-addressed change log of full-state events, served as NDJSON over HTTPS with the same<br>credential as the rest of its API. A consumer with no cursor reads that log from the beginning,<br>which is how the replica gets built; a consumer with a cursor resumes from it. Same URL, same<br>request. The consumer's entire sync state is one saved cursor.

connect<br>compact<br>drop connection

idle — no connection

GET /scroll/feed/customers<br>Authorization: Bearer

200 OK<br>Content-Type: application/x-ndjson

{"cursor":"01J9XP1A","operation":"upsert","object":{"id":"cus_120","plan":"pro"}}<br>{"cursor":"01J9XQ4R","operation":"upsert","object":{"id":"cus_123","plan":"free"}}

GET /scroll/feed/customers?scroll-cursor=01J9XQ4R<br>Prefer: stream

200 OK<br>Preference-Applied: stream

{"cursor":"01J9XR2M","operation":"upsert","object":{"id":"cus_123","plan":"pro"}}<br>your database ▸ saved cursor: — &middot; 0 objects &middot; bootstrap needed

With Prefer: stream the response never ends and each change arrives as it commits.<br>Without it, the same URL returns a bounded page you can poll on your own schedule — same events,<br>same cursors, one parser, one code path. Applying an event is a blind upsert keyed by object id<br>(or a delete, for tombstones), so duplicates and replays are harmless by construction.

Nothing ever calls your server. Every connection is one you open, so there is no inbound endpoint,<br>no signing secret, no dead-letter queue — and it works behind NAT, on localhost, and from a cron job.

Versus webhooks

Webhook-fed replicas fail in well-known ways, and none of them are bugs in anyone's implementation<br>— they are what happens when a consumer must reconstruct an ordered log from a lossy<br>notification stream. SCROLL has the provider serve the log itself, and most of the machinery on<br>both sides — delivery queues, retry schedules, dedup stores, reconciliation crons —<br>simply has no reason to exist. For both sides the diff is mostly deletions.

WebhooksSCROLL

Missed events<br>Retry schedules, dead-letter queues, replay tooling — and what still slips through is a silent gap.<br>Nothing is delivered; the consumer resumes reading from its saved cursor.

Order & duplicates<br>At-least-once and unordered; every consumer builds a dedup store and a reorder buffer.<br>The log is ordered and events are full-state upserts, so replays and reordering are harmless by construction.

Bootstrap<br>No backfill; an initial import races live deliveries.<br>Read the same feed with no cursor. The read that builds the replica is the read that keeps it current.

Downtime<br>A slow or down consumer trips failure thresholds; the provider disables the endpoint and the gap is silent.<br>Not an error. The consumer is merely behind, and catches up.

Deletes<br>Often never arrive.<br>Tombstones are first-class events in the log.

Drift<br>Reconciliation crons crawling list APIs, or a support ticket when someone notices.<br>The feed carries a count and checksum your replica checks itself against.

Security<br>A public inbound endpoint plus a second credential system: HMAC secrets to issue, verify, and rotate.<br>No inbound endpoint exists; the same bearer credential as the rest of the API.

Local development<br>A tunnel to fake a public HTTPS endpoint on your laptop.<br>Works anywhere outbound HTTPS works: laptops, containers, serverless, behind NAT.

ARCHETYPES.md walks through what this looks like for directory sync,<br>billing, and transactional email.

How it works

1 &middot; BOOTSTRAP

Read from nothing

Ask the feed without a cursor and it starts at the beginning of the log, which for a<br>compacted log is current state. Ask for a stream at the same time and that first request is<br>also your last: it fills the replica and then carries on into live changes. No second<br>endpoint, no handoff, and no import-vs-live race to lose.

&rarr;

2 &middot; SYNC

Tail

Hold a stream for...