BigLake's Iceberg REST Catalog: What Changed and How OLake Go Writes Into It | Fastest Open Source Data Replication Tool

Skip to main content

What is Changed and Why it Matters?​

Apache Iceberg was built around one core idea: separate the data from the metadata. The data files sit in object storage, and a catalog keeps track of the tables and their current state. That separation is exactly what makes Iceberg portable across query engines in the first place. But it also raises a question that's easy to gloss over: how does an engine actually figure out the current state of a table?

Google has supported Iceberg in BigQuery for a while now, but how BigQuery discovers Iceberg tables has changed. Today, Google offers a managed Lakehouse Runtime Catalog (BigLake) with an Apache Iceberg REST Catalog endpoint , giving any Iceberg-compatible engine a standard way to discover and work with the same tables.

This post walks through what changed, why the REST Catalog actually matters, and how OLake uses it to write Iceberg tables that BigQuery and other Iceberg-compatible engines can read straight off the shelf.

How BigQuery worked with Iceberg Tables without a Shared Catalog​

Before getting into the REST Catalog, it helps to look at how BigQuery worked with Iceberg tables in the first place.

1. Pointing BigQuery at a metadata file directly​

One option was to just hand BigQuery the location of an Iceberg metadata file:

orders/<br>├── data/<br>│ ├── file-001.parquet<br>│ └── file-002.parquet<br>└── metadata/<br>├── 00001.metadata.json<br>├── 00002.metadata.json<br>└── 00003.metadata.json

We had to point BigQuery at the latest one:

gs://bucket/orders/metadata/00003.metadata.json

The interesting part here is that BigQuery never had to talk to whatever catalog originally created the table. The table could've been created through a completely different Iceberg catalog and BigQuery didn't care, because it was just being handed a file path, not going through the catalog at all.

The problem is that Iceberg tables update by writing new metadata files, not by editing the old one. So the pointer you gave BigQuery gets stale the moment the table changes, and it has to be updated manually every time. If any query engine writes a new snapshot and adds 00004.metadata.json, BigQuery will still be pointing at 00003.metadata.json until the external table definition is updated to gs://bucket/orders/metadata/00004.metadata.json.

2. Catalog Based Access​

A catalog fixes that specific problem. It's the thing that keeps track of a table's current state, so nobody has to manually chase down the latest metadata file. When the table changes, the catalog's record of it changes right along with it, so now the consumers don't have to maintain their own pointer.

BigQuery has actually supported catalog-based integrations for a while too. Google documents the AWS Glue Data Catalog as a supported way to create Iceberg external tables, right alongside the direct metadata-file approach mentioned above.

So here's the thing worth being precise about. It was never that BigQuery couldn't read Iceberg tables. It could, one way or another, for years. The real change that came with the REST Catalog endpoint is that now BigQuery and other engines can now talk to Google's own catalog directly through the standard Iceberg REST Catalog interface.

Lakehouse Runtime Catalog: One Shared Catalog​

Google's Lakehouse Runtime Catalog (BigLake) is a managed metadata service for Iceberg tables. Its Apache Iceberg REST Catalog endpoint gives you a standardized interface, based on the open-source Iceberg REST Catalog API, and Google recommends it for any new workload that needs interoperability between open-source engines and BigQuery.

Instead of every engine needing its own custom integration with Google's catalog, any Iceberg-compatible engine can just speak the standard REST Catalog protocol to it. One interface, and everyone on the other end of it is understood the same way.

The easiest way to think about Lakehouse Runtime Catalog (BigLake) is as a managed metadata and governance layer. The actual data never moves. It stays in Cloud Storage, exactly where it was written. The catalog's job is just to keep track of the tables and their metadata, Google Cloud IAM controls who can touch what at the catalog level, and the REST endpoint is the standard door every compatible engine walks through to get to it.

What Iceberg 1.10 added​

Apache Iceberg 1.10.0 shipped a native BigQueryMetastoreCatalog implementation, along with a BigQueryMetastoreClient. Together, these let standard Iceberg clients use Google's BigQuery Metastore as an Iceberg catalog, the same way they'd use any other catalog implementation.

The same release also added GoogleAuthManager, which handles Google credential based authentication inside Iceberg's REST authentication framework. It can pick up Application Default Credentials, or it can use a service account you've configured yourself.

Together, that...