A few years ago the question about Apache Iceberg was whether open table formats could replace proprietary warehouses. That question is closed. Iceberg won. The new question is sharper and more interesting. What do we do with it next?
That is the question driving Iceberg v4.
At Iceberg Summit 2026 in San Francisco, more than 600 people gathered for two days and over 70 sessions. Not one talk tried to convince the room to adopt Iceberg. Every session assumed you already run it in production. The energy went somewhere else. It went to the limitations that success created, and to the spec changes that fix them.
This post walks through the state of v4 as of June 2026. It covers each major proposal, how the proposal works at a technical level, and why it matters for the people who run Iceberg at scale. It also covers the live debates, since v4 is not finished and the arguments on the dev list tell you as much as the design documents do.
How Iceberg metadata works today, in plain terms
To understand the proposals, you need a quick mental model of how Iceberg tracks a table right now. Skip this section if you already know it cold.
Iceberg replaced the old Hive approach of tracking data by directory. Hive mapped each partition to a folder and treated every file in that folder as part of the table. That worked on HDFS where directory listings were fast. It broke on object storage like S3, where listing millions of files across nested partitions got slow and expensive, and where request-rate throttling caused real outages.
Iceberg fixed this by tracking individual files through a tree of metadata. The tree has a few layers.
Data files hold the actual rows, usually in Parquet. Manifest files list groups of data files along with per-file statistics like row counts and the min and max value of each column. A manifest list collects all the manifests that make up one snapshot. A metadata file, written as JSON, points to the current snapshot and stores table-level details like schema, partition spec, sort orders, and snapshot history.
Every commit produces a new immutable snapshot. Readers get a consistent point-in-time view. Writers add data through atomic swaps of the metadata pointer. This is what gives Iceberg time travel, rollback, and snapshot isolation on cheap object storage.
The payoff of this tree shows up at query time. An engine reads the metadata, checks the per-file statistics, and skips any file whose min and max values cannot match the query filter. It does this without listing directories or opening data files. Scan planning becomes a metadata lookup rather than a full scan of the storage layout. A single table can hold tens of petabytes, and an engine can still plan a query against it quickly, since it reads metadata instead of crawling files. That property is the core architectural advantage of Iceberg, and every v4 proposal is careful to protect it.
The spec has grown in clear stages. V1 set the foundation with immutable data files, snapshots, hidden partitioning, and safe schema evolution. V2 added delete files, which let engines mark rows for removal without rewriting whole data files. That made row-level updates and merge-on-read practical, and it powered change data capture and GDPR deletions. V3, shipped across the 1.8 through 1.10 releases in 2025, added binary deletion vectors, the variant type for semi-structured data, native geometry and geography types, nanosecond timestamps, row lineage, default column values, multi-argument partition transforms, and table encryption keys.
Each version solved real problems. And each version exposed the next set of problems. That brings us to v4.
The pattern behind v4 is consistent. Iceberg was built for large, slow-moving analytical tables. The workloads people run on it now are anything but slow-moving. Streaming pipelines commit every few seconds. Machine learning feature tables carry thousands of columns. Disaster recovery plans demand that a table can move between buckets and regions. The metadata design that served batch analytics well becomes the bottleneck under these new patterns. V4 attacks that bottleneck from several angles at once.
Proposal two: storing metadata in Parquet instead of Avro
Since the early versions, Iceberg has stored its metadata files in Apache Avro. Avro is row-based. That choice was sensible when manifests were small and engines read them as whole records.
Tables grew. Manifests grew with them. A wide table can carry hundreds of columns, and each manifest entry then carries hundreds of per-column statistics. The problem is that Avro forces an engine to deserialize an entire record even when it needs only a sliver of it. During query planning, an engine often wants just the file path and the min and max of a single column. With Avro it pays to read everything.
The v4 proposal moves metadata to a columnar format using Apache Parquet. This is the same format that already stores the data in most Iceberg tables. The win is direct. An engine can read only the columns of metadata it needs. Column pruning and predicate pushdown, the same tricks that make Parquet fast for data, now apply to metadata queries too. Memory use drops. Planning gets faster on wide tables.
There is a pleasing symmetry here. Metadata storage starts to look like data storage. The same engine machinery that scans Parquet data files can scan Parquet metadata files. And this proposal pairs naturally with the adaptive metadata tree. As the metadata gets richer and more expressive, columnar reads keep planning fast. You get more detail in the metadata without paying to read all of it on every query.
The change does raise a compatibility question that the community has to handle with care. Every existing engine reads Avro metadata today. A move to Parquet metadata means every reader and writer needs to learn the new format, and tables written under v4 with Parquet metadata will not open in an engine that only knows the older layout. This is the normal cost of a format version bump, and it is why v4 is a new spec version rather than a patch. Engines will add v4 support over a period of months, the same way v3 support rolled out across the ecosystem during 2025. The reward is worth the transition. Metadata reads stop being a tax that grows with table width.
The dependency runs both ways with the column statistics rework, which is the next proposal. Columnar metadata is the container. Better-typed statistics are part of what fills it.
Proposal four: relative paths and relocatable tables
This proposal fixes an operational headache that has annoyed teams for years.
Iceberg stores file references as absolute URIs. Every manifest and metadata file embeds the full path to the files it points at, including the bucket and region. That was a deliberate early decision. Absolute paths solved real consistency problems on eventually-consistent object stores, where a stale or ambiguous reference could corrupt a read.
The cost shows up the moment you need to move a table. Copy a table to a new bucket, a new region, or a different storage system, and every embedded path is now wrong. You have to rewrite the metadata to point at the new location. For a large table with deep metadata, that rewrite is slow and expensive. It turns routine operations into projects. Replication, disaster recovery backups, and multi-region deployments all run into this wall.
The v4 proposal adds support for relative paths inside table metadata. References get stored relative to the table root rather than as absolute URIs. Move the table root, and the internal relationships between metadata and data files stay valid without a rewrite. Copy the whole directory tree somewhere else, and it just works. Absolute paths remain available where you still need them, such as references to external data that lives outside the table root.
The payoff is portability. A table becomes a self-contained, relocatable unit. You can replicate it to another region for disaster recovery and not pay a metadata rewrite tax. You can clone it for testing. You can migrate it between storage systems during a cloud transition. The Summit framing put it plainly. Relative paths eliminate entire categories of expensive metadata rewrites.
This is the proposal that is furthest along in the spec text. The spec already describes how table location works for "v4 and later," and the model assumes a catalog will provide the table's location rather than baking it into every file reference. That is a clean separation. The catalog knows where the table lives. The metadata describes the table's internal structure in terms relative to that location.
Other proposals in the conversation
The five proposals above carry the most momentum, but they are not the whole v4 conversation. Several other ideas show up in the design documents and the dev list, and they are worth knowing about even if they are earlier in the process.
Multi-table transactions and catalog-level semantics come up often. Today an Iceberg commit is atomic for a single table. A pipeline that writes to several tables and needs all of them to commit together, or none of them, has to build that coordination itself. Many teams want a way to commit across tables atomically, so that a fact table and its related dimension tables move as one unit. This kind of catalog-level transaction would be transformative for complex pipelines, and it has been flagged as one of the most-watched horizon features. It is also one of the hardest to design, since it pushes transactional guarantees up from the table into the catalog, and the REST catalog spec would have to carry the new semantics. Expect this one to take time.
Refinements to the v3 types also continue. The variant type, added in v3 for semi-structured data, has room for richer operations and better statistics, and the column statistics rework feeds directly into making variant queries faster. The geospatial types added in v3 invite extended capabilities for spatial indexing and filtering. Row lineage, the feature that gives each row a persistent identity across commits, has open discussion about making incremental processing even cheaper. None of these are headline rewrites of the format. They are the steady tightening that happens once a feature ships and real workloads reveal the rough edges.
There is also ongoing work at the file-format layer that v4 depends on, even though it lives outside the Iceberg spec. The Parquet community is working to make the footer cheaper to read, including a proposal to replace it with FlatBuffers for faster metadata access. Parquet and Arrow are evolving for the AI era in parallel with Iceberg. The Summit paired the Iceberg metadata talks with sessions on evolving Parquet and Arrow for what comes next, since the table format and the file format have to move together. A faster Parquet footer makes columnar Iceberg metadata faster to read. Better Parquet support for column-level updates makes the column families proposal cleaner. The layers are coupled, and the communities coordinate.
Keep the maturity levels straight when you read about these. Single-file commits, Parquet metadata, typed statistics, relative paths, and column families have concrete design documents and active pull requests. Multi-table transactions and the type refinements are real conversations with less settled design. Treat the first group as the likely core of v4 and the second group as candidates that may land in v4 or may slip to a later version.
Why this is happening now: streaming, AI, and a maturing ecosystem
Step back and the pattern across all five proposals is one story. Iceberg outgrew its original design assumptions, and v4 is the format catching up to its own success.
The workloads tell the story. Streaming pipelines commit every few seconds, and the old metadata tree cannot tolerate that commit latency. The adaptive tree and single-file commits answer streaming. Machine learning produces tables with thousands of columns and constant small updates, and the old layout forces full rewrites. Column families and efficient column updates answer ML. AI retrieval needs index structures the old stats map cannot hold, and the column statistics rework answers vector search. Disaster recovery and cloud migration need portable tables, and relative paths answer portability. Each proposal maps to a workload that was rare or nonexistent when v1 shipped.
The ecosystem reached the maturity to support this push. A spec is only as useful as the tools that implement it, and Iceberg's tooling crossed a threshold. The REST catalog turned from a convenience into the connective tissue of the open lakehouse. Any engine, JVM-based or not, can work with Iceberg tables through one common interface. Apache Polaris graduated to an Apache top-level project on February 18, 2026, after incubating for 18 months with contributions from Google, Microsoft, Confluent, and many others. The catalog is becoming the control plane for governance, security, and multi-tenant access.
Iceberg is also no longer a JVM-only project. The Rust implementation now powers the native scan operator in DataFusion-Comet, bypassing Spark's JVM overhead. A C++ implementation is emerging for engines that need predictable memory and SIMD-optimized execution. PyIceberg crossed 500,000 daily downloads on PyPI, and teams run it in production without ever spinning up Spark. These are production-grade implementations, and they widen who can build on Iceberg and where it can run.
Multi-engine access became routine rather than aspirational. Spark handles ingestion while Snowflake, Trino, DuckDB, or Flink serve queries, and teams describe this as established architecture. The interoperability promise Iceberg made years ago is now operational reality across cloud boundaries. The net effect is that adopting Iceberg no longer demands a single monolithic technology choice. You pick the catalog that fits your governance model, the engine that fits your latency needs, and the language that fits your team, and the spec keeps them composable.
V4 is the format growing to match that reality. The proposals support AI and streaming workloads as first-class citizens, not as workarounds bolted onto a batch design.
The shape of what comes next
Iceberg v4 is not one feature. It is a coordinated redesign of the metadata layer, broken into proposals that each solve a concrete operational problem. The adaptive metadata tree makes commits cheap and fast. Parquet metadata makes planning fast as metadata gets richer. Typed statistics make stats reliable and extensible, and they open the door to vector search. Relative paths make tables portable. Column families make wide AI tables practical to update. The Delta convergence proposal asks whether two formats can share one foundation.
These proposals reinforce each other. Cheap commits enable column updates. Columnar metadata holds typed stats. The pieces fit because they came from the same insight. Iceberg succeeded so completely that people now push it far past its original design, and the format has to evolve to hold that weight.
The debates are not noise. They are the system working. The questions about scan cost in the adaptive tree, about whether column families belong in Parquet or Iceberg, about whether the community accepts convergence, these are the conversations that turn a good proposal into a durable spec. V4 will arrive after those arguments resolve, not before. That is slower than a single vendor shipping a feature, and it is exactly why the result will be worth building on.
For now, the practical advice holds. Run v3. Watch v4. Choose your catalog with care. And follow the work in the open, since the people building it are doing it where everyone can see.
Go deeper
If you want to understand the data lakehouse and the AI workloads reshaping it at the level this post only gestures at, the best next step is to read the books that cover it end to end. Alex Merced has written multiple hands-on books on Apache Iceberg, the agentic lakehouse, modern data architecture, and AI-assisted data work. They take you from the metadata internals through to building and operating real systems.
Pick them up at books.alexmerced.com and turn the concepts in this post into working knowledge.
SOCIAL SHARE CARD GENERATOR