The Version Number Is Not the Territory

Shorn1 pts1 comments

The Version Number Is Not the Territory — The Build

Here is a log line from a PostgreSQL 14 database that PostgreSQL 14 cannot produce:

Copy<br>1ERROR: role "postgres" cannot SET ROLE to "rds_superuser"

The context: Amazon RDS, a major-version upgrade done through logical replication, with a reverse subscription pointed back at the old primary so we could fall back if the cutover went badly. Standard practice. The reverse apply worker on the old instance started, threw that error, exited with code 1, and restarted seven seconds later to do it again. A tidy crash loop applying exactly nothing.

The message names the problem precisely. The apply worker, running as postgres, tried to SET ROLE to rds_superuser and was refused. Which is strange, because the apply worker on PostgreSQL 14 does not do that.

The error that can’t happen

Walk the versions.

Through PostgreSQL 14, the logical replication apply worker runs with the privileges of the subscription owner and never switches roles. The owner has to be a superuser, so in practice the worker writes every target table with superuser force and no per-operation permission checks. The 14 documentation still says it plainly: privileges are checked once at the start of the replication connection, and are not re-checked as each change is applied. No role switch. No SET ROLE. Nothing that can produce this message.

PostgreSQL 15 added a permission check: each INSERT, UPDATE, DELETE, and TRUNCATE verifies that the subscription owner is allowed to perform it. A check is not a switch. When it fails you get permission denied for table, not a complaint about SET ROLE.

The role switch itself arrived in PostgreSQL 16, along with the run_as_owner subscription option that turns it off. From 16 on, the apply worker switches to the owner of each table before touching it, which means the subscription owner has to be able to SET ROLE to every table owner. That is the exact behavior in the error message. It is a PostgreSQL 16 behavior, and it appeared on an instance whose SELECT version() says 14.

So one of two things was true. Either the version string was wrong, or PostgreSQL 14 was doing something PostgreSQL 14 doesn’t do.

The engine is not the version

The version string was not wrong. The engine is a fork.

RDS PostgreSQL is not community PostgreSQL with a support contract stapled to it; it is a patched build, and Amazon carries changes that never show up in the upstream release notes for that major. This particular change is not exotic. Search the AWS forums and you find the same error hitting other people’s logical replication, phrased as role "rdsrepladmin" cannot SET ROLE to "rds_superuser" during a blue/green deployment. Same switch, same refusal; the only difference is which internal role does the applying. Blue/green uses rdsrepladmin. We had built the reverse subscription by hand, so ours ran as postgres.

The instance wasn’t secretly version 16. It really was v14, and the fork carries the apply-time role switch that community v14 does not.

What actually broke, and how to fix it

The switch fails on rds_superuser because nothing can SET ROLE to a reserved role. The trigger is mundane: one or more replicated tables are owned by rds_superuser. On RDS that happens more easily than it should, usually as a side effect of how a schema gets restored or carried across an upgrade.

Find all of them at once instead of discovering them one crash-loop at a time:

Copy<br>1SELECT n.nspname, c.relname, pg_get_userbyid(c.relowner) AS owner<br>2FROM pg_class c<br>3JOIN pg_namespace n ON n.oid = c.relnamespace<br>4JOIN pg_subscription_rel sr ON sr.srrelid = c.oid<br>5WHERE pg_get_userbyid(c.relowner) IN ('rds_superuser', 'rds_admin')<br>6ORDER BY 1, 2;

Then:

Reassign the tables to an ordinary application role: ALTER TABLE sch.tbl OWNER TO app_role; for each. postgres can do this through its rds_superuser membership, and tables owned by a reserved group role are the actual defect. This is the durable fix, and it is the only one that also survives a future managed blue/green, which uses rdsrepladmin and will not let you touch the subscription at all.

The part to actually remember

On managed PostgreSQL, the major version is not a complete description of your database’s behavior. The provider carries patches you can’t see, and one of them can hand you an error message the version documentation swears is impossible. When your premise and your evidence disagree, and the premise is a version number on a managed platform, believe the evidence. The fork is the tiebreaker.

Related

All your GUCs in a row: allow_system_table_mods

All Your GUCs in a Row: autovacuum

What Else Is In There?

role postgresql version owner rds_superuser subscription

Related Articles