Gmail Attachment IDs Are Not Stable (and How They Doubled Our Index) | Dioveo

Dioveo indexes Gmail mailboxes so people can search and bulk-download years of attachments. The index is one document per attachment, and from day one it was written with idempotent upserts keyed on (user_id, message_id, attachment_id). Re-run the pipeline, get the same index. Retry a failed batch, no duplicates. That was the theory, and nothing in testing ever contradicted it.<br>Then a customer's mailbox showed 117,949 attachments, and we knew from his onboarding a week earlier that the real number was about 59,000.<br>The wrong theory first#<br>Our own founder account showed the same pattern: 21,857 attachments before a maintenance re-backfill, 43,359 after. Both mailboxes had almost exactly doubled.<br>Our first theory was the opposite of the truth. We assumed the earlier, smaller counts had been truncated, that some page of results had been silently dropped on the first pass, and that the re-run had finally picked up everything. The numbers were bigger, and bigger felt more complete.<br>Two checks killed that theory. We rebuilt the founder mailbox from scratch on a fresh account and got 21,869, matching the small number, not the big one. And a verification pass that re-enumerates every year of mailbox history and diffs it against the stored message IDs came back clean: the first pass had seen every message. The small numbers were correct. The re-run had inserted the entire mailbox a second time.

Skip the manual way<br>Dioveo finds every attachment in your Gmail and downloads them in bulk. No opening emails one by one.<br>See my attachments Free to start · No credit card · 2-minute setup<br>How the Gmail attachment downloader works

The root cause#<br>The upsert key contained attachment_id, straight from the Gmail API. When you fetch a message, each attachment part carries an attachmentId you can pass to attachments.get:<br>"partId": "1",<br>"filename": "invoice-march.pdf",<br>"body": {<br>"attachmentId": "ANGjdJ_wZxT4iEc0...",<br>"size": 182044<br>Here is the part the documentation never spells out: that ID is not a stable identifier. Fetch the same message twice and you get a different attachmentId each time. It is an opaque, effectively ephemeral handle, not a name.<br>So our idempotency key was built on a value that changes on every fetch. The upsert never matched an existing row, because no re-fetched attachment ever produced the same key twice. Every "idempotent" write was an insert. The pipeline was perfectly deterministic in the worst way: run it N times, get N copies of the mailbox.<br>There is a second half to this trap. The stored IDs do not just fail to match, they also expire. We learned that in a different feature, when saved attachment IDs started returning 404s on download weeks later. And Gmail has a cousin of this behavior for drafts: every autosave destroys the draft message and creates a new one with a new ID, which once had us chasing phantom "deleted" messages that nobody had deleted.<br>The fix: stop upserting, start replacing#<br>You cannot dedupe on a key the provider will never repeat. Filename is not a substitute either, since one message can legitimately carry three files named scan.pdf.<br>The property that saved us: our ingest batches always carry the complete extraction for a message. Every attachment of a message travels together. So the write path changed from per-attachment upserts to per-message replacement: in one transaction, delete every indexed row for (user_id, message_id), then insert the fresh set.<br>That one change had three effects:<br>Re-runs converge. Re-fetching a message replaces its rows instead of adding to them. Backfills became safe to run any number of times.<br>Existing damage self-heals. The doubled indexes did not need a bespoke cleanup script. The next backfill replaced every message's rows with the correct set. The 117,949-row index came back as 59,038, which matched the first pass plus a week of new mail. Ours came back as 21,869.<br>Downloads stopped rotting. Because every re-index stores the attachment IDs from the latest fetch, the handles we hold are always the freshest ones, which is exactly what you want for a value that expires.<br>What we took away from it#<br>Never build identity on a field the provider does not promise is stable. An opaque handle is not an identifier, even when it sits in a field called attachmentId. If the docs do not say "stable", assume it is not.<br>Untested idempotency is a hypothesis. Our upserts were idempotent in code review and in unit tests. Nobody had run the full pipeline twice against the same real mailbox and diffed the counts. That test would have caught this on day one, and it is now part of how we verify sync changes.<br>When two counts disagree, verify before you pick a side. We spent real time on the truncation theory because the larger number flattered our assumptions. An independent recount settled it in an afternoon.<br>Replacement beats upsert when child identity is unreliable. If the upstream cannot...