The open lakehouse stack spent this week arguing about what belongs in a spec and what belongs in a release. Iceberg contributors opened a formal push to remove equality deletes from the V4 spec, Parquet closed its vote on a brand new File logical type, and Polaris wrestled with how much consistency its persistence layer owes its users. Underneath the design debates, release trains kept moving: Iceberg Rust, an Iceberg Terraform provider, Arrow JS, Arrow Rust Object Store, DataFusion, Ballista, and Comet all had votes in flight. This is a week where the community showed both sides of its personality, big structural bets for the future and steady, unglamorous shipping for the present.
By the numbers, the six dev lists carried 268 messages across roughly 80 distinct threads in the past week. Polaris led with 84 messages, Iceberg followed at 71, Parquet posted 51, DataFusion 26, Arrow 20, and the young Ossie project added 16. Those raw counts undersell the range: the week included two format-level votes, five release candidates, one new committer, a persistence redesign proposal, and at least four threads that will shape spec decisions months from now. Grab a coffee. There is a lot to cover.
Apache Iceberg
The most consequential thread of the week came from huaxin gao, who , merged across six commits. The feature ships as a new table maintenance task called ConvertEqualityDeletes, integrated with IcebergSink. Writers keep producing data files and equality deletes as before. The converter reads them, resolves the deletes into deletion vectors using a primary key index backed by RocksDB in Flink state, and commits data files plus DVs to the target branch. Teams can stage writes on a separate branch so readers never see equality deletes at all, or run in-place conversion on a single branch. This is the workable alternative that was missing in 2024, and it lands right as the V4 deprecation push begins. Read the two threads together and you see a community clearing a path before it closes a door.
Release work stayed busy. Danny Jones and Shawn Chang from Kevin Liu, Amogh Jahagirdar, and others earlier in the week. Sung Yun, Renjie Liu, L. C. Hsieh, and Maximilian Michels verified RC4, which includes a check that pyiceberg-core builds and tests cleanly against the release. The Rust implementation now sits underneath the Python ecosystem, so each Rust release carries weight well beyond Rust users. That dependency chain is worth pausing on. PyIceberg increasingly delegates its performance-critical paths to pyiceberg-core, which is compiled from this Rust codebase. A bug in iceberg-rust becomes a bug in every Python notebook and Airflow DAG that touches Iceberg through PyIceberg. That is why the release checklist now explicitly verifies the Python bindings, and why voters from the Python side of the community show up on Rust release threads. The 0.10.0 line also continues the project's steady march toward feature parity with the Java reference implementation, which lowers the barrier for teams that want Iceberg without a JVM anywhere in the stack.
Infrastructure as code arrived as a first-class citizen this week. Matt Topol as issues surfaced. Once this lands, teams can declare Iceberg resources in the same Terraform plans that manage the rest of their infrastructure. Think about what that unlocks in practice. A platform team can define namespaces, tables, and their properties in version-controlled HCL, review changes through pull requests, and roll environments forward and back with the same tooling they use for VPCs and Kubernetes clusters. Catalog drift, the gap between what the catalog says and what the last runbook did, becomes a solved problem instead of a recurring incident. It took years for databases to get credible Terraform support. Iceberg is getting there in its first decade.
AI showed up on the dev list in a very concrete form. Gang Wu for Iceberg. Embeddings are everywhere in AI workloads, and today they live in Iceberg as list, which cannot express the invariant that every value shares one dimension. The proposal prefers a dense numeric vector type with compact schema encoding, something like float[768], with fixed dimension, non-null elements, and nullability controlled at the field level. It points at parallel work in the Parquet community on fixed-size lists, which matters because the table format and the file format need to agree for the type to pay off. Meanwhile the , and smartly split the draft into two parts: what criteria any candidate file format should meet, and how Vortex meets them. That framing turns a single-format request into a durable policy, which is exactly what a spec-driven project needs as more formats knock on the door.
Quality and correctness threads kept coming. Priyadarshini Mitra , a standalone language-neutral repository modeled on parquet-testing, so every Iceberg implementation checks its reading of the spec against a shared answer key instead of only against itself. Working proofs of concept already exist for pyiceberg, iceberg-rust, and iceberg-go. And Russell Spitzer , where adding LocalTimestamp support changes what convert returns for local-timestamp-micros, from Long to TimestampType.withoutZone(). His position: the old behavior is a bug, and Iceberg should not preserve incorrect legacy behavior behind a flag for outside consumers. Ryan Blue engaged on the thread, and the precedent cited is the earlier NanoTimestamp change that did the same thing.
On the operational side, Oleksii Omhovytskyi . Iceberg tables already record their sort orders in metadata, but engines rarely exploit that knowledge at plan time, so there is free performance sitting on the table. A CDC practitioner opened a thread on in the API and core modules, small plumbing that makes REST catalog auth failures debuggable instead of mysterious. Andrei Tserakhau also , which bridges Iceberg tables into enterprise AI and RAG pipelines, one more signal that retrieval workloads now treat the lakehouse as a first-class source.
Rounding out the week: Adam Szita , favoring a basic implementation in Iceberg itself that replaces a view with a table read, usable without engine APIs, with engines layering smarter freshness decisions on top over time. Alexandre Dutra . And Talat Uyarer , prompted by PRs from Ayush and Prithvi that surfaced real consistency gaps in the JDBC backend. Dmitri's framing is deliberately broad: rather than patching individual windows, the community should design one approach that covers concurrent validated commits, independent but consistent RBAC changes, atomic multi-entity updates, authorization-filtered listings, credential vending rooted in exact catalog state, and server-side retries for transient failures. Prithvi S made the problem concrete in . The resolution matters for anyone running external authorizers: the authentication layer will forward user information to PolarisPrincipal as optional attributes, which decouples OPA and Ranger authorizers from both Quarkus classes and PrincipalEntity. Authorizers then work with any identity provider and survive Quarkus upgrades untouched. Alexandre Dutra also , Yufei Gu clarified the two motivations, runtime loading of JDBC drivers for ASF binaries and runtime datasource creation as a building block for per-realm datasources. His proposed contract keeps Quarkus and Agroal as the default, with Polaris-managed Hikari as an escape hatch. Alexandre Dutra pushed back hard on the hybrid: alternating pools based on configuration means bugs and performance characteristics vary across deployments purely by pool choice, so if the goal is a runtime-driven architecture, commit fully and switch to Hikari unconditionally. Romain Manni-Bucau, JB, and Dmitri also weighed in. Nobody has yet closed the gap between "escape hatch" and "all or nothing."
Process and culture got real attention too. Dmitri opened a thread titled when a table or view rename fails with TARGET_ENTITY_CONCURRENTLY_MODIFIED. The reasoning: 409 already means "target identifier exists" in the Iceberg REST spec, and 429 wrongly implies rate limiting, so 503 is the retryable option without semantic conflicts. The vote closes at 14:00 UTC on Sunday, July 19, with Robert Stupp, Alexandre Dutra, Nándor Kollár, and Dmitri participating. Nándor had teed up the choice in , Robert Stupp supported Polaris hosting Ossie semantic-model documents as a beta foundation, then drew a sharp line: the merged API is namespace-and-name CRUD over opaque documents, and the project should not describe it as enabling AI tools, BI tools, or semantic discovery until clients can actually find models by table, metric, domain, or capability. His larger point is architectural. The client consumption model should drive the persistent data model, because once semantic models become durable Polaris entities, identity, versioning, indexing, and freshness semantics get very hard to change. This debate matters well beyond Polaris. The industry is converging on the idea that AI agents need a semantic layer to query data correctly, and catalogs are the natural place to host one. Whoever defines how agents discover the right semantic model, by table, by metric, by domain, by trust level, defines a big piece of how agentic analytics works. Robert's insistence on honest labeling, calling document CRUD what it is until discovery exists, protects users from building on promises the API does not yet keep. EJ Wang also : the persistence refactor splits into its own PR, the metrics SPI stays in core with a no-op default, and the REST query API plus JDBC implementation land as optional extensions.
A cluster of smaller operational threads rounded out the Polaris week. Eundo Lee, Alexandre Dutra, and Yufei Gu discussed , trading environment fidelity for speed and debuggability in CI. EJ Wang and Dmitri discussed and . None of these is glamorous. All of them are the difference between software that demos well and software that operators trust.
Polaris is also getting its own Terraform provider. Alex Stephen explained that the Iceberg provider community chose to focus exclusively on Iceberg resources, so the Polaris resources need a new home. Sung Yun , , insisting on explicit usability and operational criteria for the local lineage store before schema work merges.
Apache Arrow
Arrow's headline discussion was about making schemas travel well. Matt Topol, David Li, and Dewey Dunnington continued , with David Li, Kent Wu, Bryce Mecum, and Hyukjin Kwon verifying. Andrew Lamb .
Apache Parquet
Parquet delivered the week's biggest format decision. Burak Yavuz drew 19 messages and capped months of design docs, biweekly syncs, and a discussion thread, with reference implementations already open in parquet-java and arrow-rs. A File logical type lets Parquet columns carry file-like binary payloads with defined semantics, and the breadth of the voter list, spanning Iceberg, Arrow, and Parquet maintainers, says a lot about who plans to use it. The use cases are easy to picture. Multimodal AI datasets carry images, audio clips, and documents alongside tabular features today, usually as raw binary columns with semantics living in tribal knowledge or sidecar metadata. A File logical type gives readers a standard way to know that a column holds file content, opening the door to smarter tooling, previews, and type-aware processing across engines. Burak is holding the format PR open a few more days for remaining reviewer feedback from Rok, Antoine, and Gang, and a follow-up design conversation with Talat and Gaurav on version identifiers continues on the doc.
The long-running versioning debate produced structure this week. Micah Kornfield , using an extended apples-versus-oranges analogy to argue the community already spent a month choosing between numbered releases and time-based presets, and the vote simply affirms that choice. Julien Le Dem , saying he plans to start the process next week after multiple requests for accumulated features, fixes, and patched CVEs. Aaron Niskode-Dossett flagged two performance PRs as candidates, a FileStatus cache in the footer path and a faster RunLengthBitPackingHybridDecoder.
Encodings advanced on two tracks. Prateek Gaur , answering Antoine Pitrou's nullability question with fresh benchmarks. His numbers show a hint-aware reader on a plain LIST landing within noise of the non-compatible vector option, around 1,430 nanoseconds per row versus 2,600 for a full Dremel decode, and the optional outer array costs nothing when no nulls are present. That is the kind of measurement that turns a format argument into a format decision, and it pairs directly with the Iceberg vector type proposal above.
The encoding pipeline has more behind ALP. Prateek also floated in the column metadata, trimming redundant bytes from footers that large tables repeat thousands of times. Aaron Niskode-Dossett suggested , where several of the threads above got live discussion time.
Two compatibility threads deserve attention from anyone running mixed reader fleets. Kevin Liu , arguing for FLBA<9> over FLBA<8> because BigQuery and Trino are already moving toward picoseconds, and one width handles nanoseconds through picoseconds with the same code.
Apache DataFusion
DataFusion ran a clean release week across three subprojects. Matt Butrovich after RC1 failed, and the vote drew verification from Andrew Lamb, Marko Milenković, Martin Grigorov, and L. C. Hsieh before through its vote to , and the congratulations thread filled quickly with notes from Kumar Ujjawal, Jeffrey Vo, Matt Butrovich, and Martin Grigorov. Committer announcements are easy to skim past, but they are the truest health metric an open source project has.
The shape of the release week says something about how the DataFusion family now operates. The core engine, the Ballista distributed scheduler, and the Comet Spark accelerator each cut releases on their own cadence while tracking the same 54.x line, so downstream users get a coherent version story across very different deployment models. A failed RC1 followed by a clean RC2 within days is also a sign of healthy release muscle: problems get caught by verification, not by users. With DataFusion increasingly serving as the query engine inside other lakehouse tools, that discipline pays dividends far outside the project's own repositories.
Apache Ossie
Ossie, the young semantic interchange project, spent the week doing the unglamorous work that decides whether a project scales. Yong Zheng, fresh to the codebase, on an apache-ossie-xxxxx pattern and to introduce a shared base converter abstraction, since only three of five converters follow the same file structure today.
On the spec side, Will Pugh and the financial services semantic stack, reading FIBO as a reference ontology layer and Ossie as the interchange layer that maps to it. The question lands at the right moment, given the Polaris semantic model hosting debate above. The ecosystem is deciding, in real time, which layer owns which promise.
Why does converter housekeeping deserve newsletter space? Because Ossie is a specification project, and a spec lives or dies on its converters. If exporting a dbt project, a Snowflake semantic view, or a GoodData workspace into Ossie feels inconsistent, adoption stalls no matter how elegant the core model is. A shared base converter class and a uniform packaging story lower the cost of writing converter number seven, and converter number seven is how a new tool joins the ecosystem. Yong volunteering to do this work in his first weeks on the project is exactly the kind of contribution that turns an incubating spec into infrastructure. New contributor Dragos Crintea also to keep the growing contributor base aligned.
Cross-Project Themes
Three threads of connective tissue stood out this week. First, infrastructure as code went from wish to reality across the stack: Iceberg voted on its first Terraform provider release while Polaris reached lazy consensus on creating its own provider repository. Declarative catalog and table management is becoming table stakes, and both communities are honoring the registry naming rules rather than fighting them.
Second, the AI workload is reshaping formats from both ends. Iceberg debated a native vector type, Parquet benchmarked fixed-size lists to store those vectors well, Iceberg considered Copilot for code review, and Polaris debated how to host Ossie semantic models for AI and BI consumption. The table format, the file format, the catalog, and the semantic layer are each answering the same question at their own layer: what does an agent or an embedding pipeline need from open data infrastructure?
Third, correctness culture is compounding. Shared conformance fixtures in Iceberg modeled on parquet-testing, a VARIANT forward compatibility fix with backports in Parquet, spec language tightening on manifest uniqueness, and integrity validation actions all point the same direction. As implementations multiply across Java, Rust, Python, Go, and C++, the projects are investing in shared answer keys instead of trusting each implementation to grade itself.
A fourth pattern hides in plain sight: governance maturity. Parquet is writing down its versioning rules as an RFC instead of relying on tribal knowledge. Polaris is debating whether review norms should be documented rules or reviewer discretion. Iceberg is defining criteria for admitting new file formats before evaluating any specific one. DataFusion promoted a committer. These are the habits of projects planning to be around in a decade, and the fact that all four surfaced in one week suggests the lakehouse stack is entering its institutional phase. Institutional does not mean slow. The same week produced five release candidates and a passed format vote. It means the projects are building the decision-making machinery that lets them move fast without breaking the ecosystems that depend on them, and that machinery is the least visible, most valuable output of this community.
Looking Ahead
The Polaris 503 vote closes Sunday, July 19, and the Polaris community sync on July 23 takes up the tag spec. The same day, Iceberg contributors gather in person in Austin. Watch for the Parquet ALP encoding vote to open, for Fokko to kick off the Parquet 1.18.0 release process, and for results on Iceberg Rust 0.10.0 RC4 and the Terraform provider RC1. The equality deletes deprecation thread will keep growing, and the answers there will define a good chunk of what Iceberg V4 becomes.
Further out, keep an eye on three slow burns. The Iceberg collation discussion has to reconcile cross-engine consistency with ICU upgrade freedom, and whatever it decides will echo in every engine that sorts strings. The Polaris persistence redesign will take months, and the SPI shape it lands on determines how hard NoSQL backends are to build. And the Parquet versioning RFC, once merged, becomes the template other format projects copy when they outgrow informal release habits. None of these resolves next week. All of them reward following the threads as they develop, and the permalinks above will take you straight to the source.
If this is your first issue, a note on method: everything above links to the public Apache dev list archives, and every claim traces to a thread you can read yourself. The dev lists are where the real decisions happen, before the blog posts and the conference talks. Subscribing to even one of them changes how you understand this ecosystem.
Resources & Further Learning
Get Started with Dremio
: Learn how Dremio brings the open lakehouse stack together
Free Downloads
: O'Reilly book, free download
Books by Alex Merced
- Browse the full catalog of 50+ books at books.alexmerced.com
SOCIAL SHARE CARD GENERATOR