Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

A Spark job commits a table update. The catalog writes the change to Postgres. Then the network drops between the catalog and the client, and the client never sees the response. The client does the sensible thing and retries. This time the catalog sees that the table has already moved past the base snapshot in the request, so it returns 409 Conflict. The client reads that 409 as a failed commit and deletes the metadata files it just wrote. The commit is now recorded in the catalog, and the files it points at are gone.

That is data loss. It comes from a network blip, not from a bug in anyone's query engine.

Apache Polaris 1.7.0 shipped on August 2, 2026, tagged by JB Onofré at commit 4ac2f05. It fixes that specific failure and a long list of others in the same family. If you skim the changelog you see hundreds of entries, most of them dependency bumps, and it looks like a maintenance release. It is not. Underneath the noise there are four real stories: idempotent writes, a new beta API for semantic models, a much stricter approach to credential vending and location validation, and a serious pass over orphan file cleanup.

I am going to walk through all four, and I am also going to tell you which parts of this release create work for you rather than saving you work. Both kinds show up here.

This piece assumes you know what a table is and roughly what a data lake is. Everything past that gets defined as it comes up.

What a catalog actually does, and why its bugs are expensive

Apache Iceberg is a table format. It describes how to lay out data files and metadata files in object storage so that many engines read the same table the same way. Iceberg tracks a table's current state through a chain of files: a metadata file points at snapshots, snapshots point at manifest lists, manifest lists point at manifests, and manifests point at the actual Parquet data files.

One question that chain does not answer is: which metadata file is current right now? Every change to a table writes a brand new metadata file. Something has to record the swap from the old one to the new one, and it has to do that atomically so two writers cannot both think they won.

That something is the catalog. In its smallest form, a catalog is a pointer store. It maps a table name to the path of the current metadata file, and it swaps that pointer atomically on commit.

Apache Polaris is an open source implementation of that pointer store, speaking the Iceberg REST protocol. The REST protocol matters because it moves catalog logic out of the client. With older designs like Hive Metastore, every engine embedded its own catalog client code, and every engine had its own opinions about credentials and connection handling. With a REST catalog, the engine speaks HTTP to a service, and the service handles storage credentials, access control, and the commit protocol. Polaris was co-created with Snowflake, donated to the Apache Software Foundation, and graduated to Top-Level Project on February 18, 2026.

Because the catalog owns the pointer swap, catalog bugs have a nasty property. They do not corrupt one query. They corrupt the table. A dropped pointer, a prematurely deleted metadata file, or a credential scoped one character too wide affects every engine that reads that table afterward. This is why a release like 1.7.0, which is heavy on correctness fixes and light on flashy features, deserves more attention than a release full of new endpoints.

Here is the shape of what changed, before the details.

Idempotent writes, the headline feature

Go back to the failure I opened with. The root cause is that HTTP gives the client no way to distinguish "your request never arrived" from "your request succeeded and the response got lost." Those two cases demand opposite responses. In the first case the client should retry. In the second case the client should stop and treat the commit as done.

The Iceberg community's answer is an Idempotency-Key header on mutation endpoints, following the same design as the IETF draft for idempotency keys that payment APIs have used for years. The client generates a unique key per logical operation and sends it with the request. The server remembers the key and the outcome. On a retry with the same key and the same payload, the server returns the original result without executing anything again.

Polaris 1.7.0 implements the server half of that for two operations. Huaxing Gao's work landed entity-property idempotency for createTable in .

Read the word "opt-in" carefully, because it is the whole story for operators. Idempotency on updateTable is not on by default. You turn it on, and clients have to send the key. Nothing about upgrading to 1.7.0 makes your existing writers retry-safe by itself.

How a client finds out

The interesting design choice is capability discovery. A client has no business guessing whether a catalog honors idempotency keys, because guessing wrong in the unsafe direction produces exactly the corruption we are trying to prevent. So the catalog advertises it.

Every Iceberg REST catalog exposes a GET /v1/config endpoint that clients call at connection time. It returns two property bags: defaults, which the client applies unless it overrides them, and overrides, which the server forces. removed an unused IdempotencyStore and an idempotency_records table.

That is the sound of a design being reconsidered before release. An earlier approach kept idempotency records in a dedicated table, which means a separate write on every mutation and a separate cleanup job to expire old rows. The shipped approach attaches idempotency state to the entity itself, which is what "entity-property idempotency" in the createTable PR title describes. Fewer moving parts, no second table to vacuum, no second failure domain.

If you were tracking this feature from the development branch and built tooling against idempotency_records, that table is gone. Check before you upgrade.

What to do about it

Turning this on is a two-sided change, and the client side is not entirely in your hands yet.

1. Upgrade Polaris to 1.7.0 and confirm the config endpoint reports the lifetime you expect.


2. Check which of your engines send an Idempotency-Key. Support arrives engine by engine as the Iceberg client work lands, so verify rather than assume.


3. For engines that do not send one yet, nothing regresses. You get the same behavior you have now.


4. Watch for 422 responses after you enable it. Under the design, a repeated key with a different payload is a client bug, and the server rejects it rather than guessing which version you meant.

The last point is the one that surprises teams. Idempotency keys make a class of client bugs visible that used to hide inside retry loops. That is a feature. It also generates support tickets in week one.

The catalog starts learning what a metric is

The second story in 1.7.0 is smaller in code and larger in implication. marked it beta.

OSI stands for Open Semantic Interchange. It is an industry specification effort, convened by Snowflake with a broad group of analytics and BI vendors, that defines a YAML format for semantic models. A semantic model in this sense holds the things a table does not: datasets, the relationships between them, dimensions, and metrics. The example everyone reaches for is revenue. Every dashboard defines it, no two definitions agree, and the finance number never matches the sales number.

The OSI spec gives that definition a portable form. A semantic model contains datasets, relationships, and metrics, with SQL expressions attached and optional context annotations written for language models to read.

So why is this landing in a table catalog?

Because the catalog is the one component every engine already talks to. If your metric definitions live in your BI tool, they are available to your BI tool. If they live next to the tables, in the service that Spark and Trino and Flink and your agent framework all authenticate against, they are available to everything. The same argument that moved credential vending and access control into the catalog applies to semantics.

The AI angle is the forcing function. An agent writing SQL against a lakehouse has the schema and nothing else. It sees a column named amt_net and guesses. Give it a metric definition that says net revenue excludes returns and intercompany transfers, and the guessing stops. That is the thin part of the problem that semantic models solve, and it is the part where wrong answers are most expensive because they arrive fluent and confident.

Two supporting changes matter more than they look. . A registry for config endpoints is how a server grows optional API surfaces without every extension hard-coding itself into the core request path. Semantic models are the first tenant of that mechanism. They will not be the last.

The honest assessment

Beta means beta. The PR titles say scaffolding, the API is explicitly marked as unstable, and the OSI core spec itself is young. Do not build a production metric layer on this in August 2026.

What to do instead: read the OSI spec, write a semantic model for one domain you already argue about internally, and see whether the format holds your actual business logic. The feedback loop for a young specification is people trying to express real definitions in it and reporting where it breaks. That is worth more to you and to the project than waiting for version 1.0 of the API.

One more reason to care, independent of which platform you run. Nearly every analytics vendor ships some form of semantic layer, and each one holds your metric definitions in its own format. A portable definition format means those definitions survive a change of vendor. That is worth something regardless of who you buy from today.

Credential vending got stricter, and one fix was a real hole

Credential vending is the feature where the catalog, rather than the engine, holds the cloud storage credentials. An engine asks for a table, the catalog checks whether that principal is allowed, then calls AWS STS or Azure or GCS to mint a short-lived credential scoped to just the paths that table needs. The engine gets a token that opens a narrow door instead of a bucket-wide key.

The security of the whole arrangement rests on one thing: the scoping has to be correct. A credential scoped one prefix too wide hands a reader access to a neighbor's data. 1.7.0 fixes three separate ways that went wrong.

fixed a GCS downscoped credential prefix boundary problem for locations without a trailing slash. This is the classic prefix bug. A credential scoped to gs://bucket/data/sales with naive prefix matching also opens gs://bucket/data/sales-archive and gs://bucket/data/sales_pii, because both start with the same characters. The trailing slash is what makes the boundary a boundary.

added a session policy parameter to SigV4 connections, which lets you attach an additional IAM policy that further narrows an assumed role. propagated storage HTTP client settings to S3FileIO for table operations, so proxy and timeout configuration finally applies to the catalog's own file reads rather than only to the vending path.

Location validation tightened everywhere

Alongside vending, 1.7.0 tightened where a table is allowed to say its data lives. This is the same class of protection viewed from the other end.

• 
  validates locations when registering tables and views.


• 
  fixed ALLOW_EXTERNAL_METADATA_FILE_LOCATION not being overridable at catalog level.


• 
  made default table and view locations unique, and fixed a gap in that input: the realm identifier was missing. A realm in Polaris is a tenant boundary, the mechanism that keeps separate organizations on one deployment from seeing each other. If your OPA policy receives a request that names a catalog and a table but not the realm, and two realms happen to use the same catalog name, your policy has no way to tell them apart. Any operator running multi-tenant Polaris with OPA should treat this as the reason to upgrade.
  
  
  

  The rest of the authorization work is structural. Y Sung's refactored PolarisPrincipal to hold generic attributes, followed by an AttributeMap interface in put the missing privilege and the target entity into 403 messages. A denial that says "access denied" starts a thirty-minute investigation. A denial that names the privilege and the object it applied to ends in thirty seconds. clarified principal role selection semantics, fixed view grants on federated catalogs, and , titled as fixing data corruption via premature metadata deletion in commitTransaction. Deleting a metadata file that is still referenced is the exact failure I opened this article with, arriving from the server side rather than the client side. Paired with it, cleans up orphan metadata files on failed table and view commits. Together they draw a clear line: on failure, delete the files the failed attempt created, and never touch anything else.

  
  
  

  Several fixes target the cleanup tasks themselves.

  
  
  
  
  
  • 
    taught the manifest cleanup handler to handle delete manifests. Delete manifests track row-level deletes in merge-on-read tables. Skipping them means a specific category of file was never cleaned.
  
  
  • 
    fixed a duplicate setId() in the table cleanup handler that burned entity IDs on every run.
  
  
  • 
    changed TaskHandler.handleTask to return void so success and failure travel through exceptions instead of a boolean nobody checked consistently.
  
  
  
  
  

  Then there is a performance thread with a direct line to your cloud bill. removed a redundant existence check before deleteFile, and , from first-time contributor hkwi, added an OpenTelemetry event listener. OpenTelemetry is the vendor-neutral standard for traces, metrics, and logs, and nearly every observability backend ingests it. Emitting catalog events as OpenTelemetry data means a table commit shows up in the same trace view as the query that caused it, without a custom bridge.

  
  
  

  fixed PolarisEventMetadata.eventId() returning a different UUID on every call, which is exactly the bug that breaks deduplication in any consumer built on at-least-once delivery. avoids setting up metrics persistence when events are only buffered in memory, which removes a startup cost for deployments that never enabled persistence.

  
  
  

  If you are building anything that reacts to catalog changes, the Kafka extension is the piece to look at first. Start with a consumer that does nothing but log, run it for a week, and read what your catalog actually emits before you design around it.

  
  
  

  
  
  
  Persistence: several queries stopped reading more than they needed
  

  
  
  

  The JDBC persistence layer, which for most people means Postgres, got a focused optimization pass. The pattern repeats across the fixes, and the pattern is the lesson.

  
  
  
  
  
  • 
    did the same for lookupEntityGrantRecordsVersion.
  
  
  • 
    bounded the JDBC hasChildren existence check with LIMIT, and eliminated redundant metastore lookups when resolving principal roles.
  
  
  
  
  

  Every one of these is the same mistake in a different place. The code needed one small fact, a version number or a yes/no answer, and asked the database for entire rows to get it. On a catalog with a few thousand entities nobody notices. On a catalog with hundreds of thousands, resolving principal roles on every single request while reading full rows is how a p99 latency graph develops a shelf.

  
  
  

  hasChildren is the clearest example. The question is "does this namespace contain anything," and the answer is yes the moment one row exists. Without a LIMIT, the database happily returns all of them, and the cost of asking scales with the size of the namespace instead of staying constant.

  
  
  

  One more in the same family: changed concurrent rename to return HTTP 503 instead of 500. The distinction is not pedantic. 500 means the server broke and a retry is pointless. 503 means the server is temporarily unable and a retry is sensible. A concurrent rename is a transient contention event, so 503 is the honest answer, and every well-behaved HTTP client already knows what to do with it.

  
  
  

  centralized drop-failure error mapping and fixed misleading messages. Scattered error mapping produces the situation where the same underlying condition surfaces as three different messages depending on which code path found it. added a readiness check for reflection-free serializers. Reflection-free serialization is what lets a Quarkus application start fast and compile to a native image, and a startup check that verifies it is actually in effect prevents a silent fallback to the slow path.

  
  
  

  The test infrastructure moved from localstack to Floci testcontainers for AWS, GCP, and Azure emulation, with integration tests migrated to a shared server runner and pushed down into the extensions they belong to. Faster and better-isolated tests sound like an internal concern. They are the reason the next release ships with fewer regressions.

  
  
  

  Operationally useful odds and ends:

  
  
  
  
  
  • 
    added HTTP histogram buckets, which gives you real latency distributions instead of averages.
  
  
  • 
    removed the schema version option from the admin bootstrap command.
  
  
  • 
    fixed credential exposure in Python CLI debug logs and hardened profile secret handling and config storage.
  
  
  • 
    added non-HTTP scheme support to the CLI.
  
  
  • 
    .
    Vollständiger Original-Bericht

    Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

    ↗ Original-Artikel auf dev.to lesen