The lakehouse community spent this week arguing about versions, and the arguments mattered. Parquet contributors produced the single largest thread across all five projects with a 40-message debate on what Parquet versioning should even mean, while Iceberg shipped four release candidates of its C++ implementation in seven days and locked in a patch release plan for its two production lines. Underneath the release activity, a quieter theme connected everything: how these projects make decisions. Polaris debated merge button mechanics and HTTP status codes, Parquet contributors insisted that working group syncs cannot replace mailing list consensus, and Arrow wrote down rules for AI-generated code reviews. The formats are maturing, and so is the governance around them.
Before getting into each project, the raw numbers set the scene. The five dev lists combined for 358 emails this week. Iceberg led with 135 emails across 34 threads from 51 distinct participants, followed by Polaris at 114 emails across 23 threads from a tight group of 14 regulars. Parquet concentrated 72 emails into only 7 threads, which tells you its conversations ran deep rather than wide. Arrow posted 24 emails across 11 threads from 18 participants, and DataFusion rounded things out at 13 emails across 6 threads. The shape of those numbers matters as much as the totals. Iceberg's breadth reflects a project with a dozen parallel workstreams from spec evolution to language implementations to community events. Polaris's depth from a small group reflects a project where a core team is hammering out operational fundamentals. And Parquet's concentration reflects a community wrestling with a handful of existential questions all at once.
Apache Iceberg
The Iceberg dev list logged 135 emails across 34 threads from 51 participants this week, and the headline work happened in the spec.
Ryan Blue's vote to evolved into a full conversation about a v2 loadTable endpoint. Ryan Blue laid out the case for v2, including optional locations, optional snapshots, and moving credentials out of properties. Russell Spitzer agreed those are good reasons but questioned whether a v2 endpoint actually changes the capability negotiation problem the header was meant to solve. The sharpest pushback came from Christian Thiel of the Lakekeeper project, who challenged the sentiment that a v2 loadTable should mandate that clients fail when they encounter unsupported restrictions. His argument is grounded in adoption reality: a v2 endpoint gets adopted for many reasons, and strict failure semantics create friction for clients that have nothing to do with the restriction features. Kurtis Wright backed the v2 direction after missing the original community meeting discussion. This thread is the one to follow if you build or operate REST catalogs, because the outcome shapes how every engine negotiates features with every catalog for years.
Step back and the stakes become clearer. The REST catalog spec is now the contract that binds the entire commercial Iceberg ecosystem together. Every managed catalog service, every query engine, and every standalone tool implements some slice of it, and those slices increasingly diverge in subtle ways. A capabilities header gives clients a standard way to declare what they understand, which lets catalogs make informed decisions about what to return. A v2 loadTable goes further by fixing accumulated design debt in the most heavily trafficked endpoint in the protocol. The tension Thiel identified is the classic protocol evolution dilemma: strict semantics protect correctness for new features like fine grained access control, where a client silently ignoring a row filter is a security incident, but strictness also slows adoption by punishing clients for capabilities unrelated to their workload. How the community threads that needle will determine whether v2 arrives as a clean upgrade path or a compatibility minefield. The fact that catalog implementers like Thiel, engine maintainers like Spitzer, and spec authors like Blue are all in the same thread arguing in good faith is the system working as designed.
Prashant Singh's . The read side of the proposal binds row filters and masks to field IDs so they survive schema evolution safely, but Yun points out that the write path has no equivalent story yet. Securing reads while leaving writes unguarded is a half-finished lock, so expect this gap to get attention as the proposal matures.
Security work continued on a second front. Adam Szita published a after encountering a bug where the Spark rewrite manifests procedure fails to carry over first row IDs correctly. The thread gathered 13 messages and quick consensus. Steven Wu pointed to the existing 1.11.1 milestone, Yufei Gu and Daniel Weeks added their support, and Weeks made the operating principle explicit: keep the 1.10 backports narrow so the release stays easy and helps anyone who has not yet moved forward. Meanwhile Neelesh Salian conclusion over on the Polaris list crossover thread, where Alexandre Dutra summarized the community's honest feedback by recommending the feature branch experiment not be repeated.
The C++ implementation provided the week's endurance story. Junwang Zhao proposed , by June 11. Each candidate fixed issues the previous one surfaced. Matt Topol's RC2 verification caught real gaps in the release tooling, including undocumented meson and gtest requirements and an SSL workaround needed for the curl dependency, and Gang Wu called for improving the release script to catch similar issues automatically. By RC3, verification reports were coming in clean from macOS and Ubuntu environments across multiple contributors including Steven Wu, Raúl Cumplido, and Tanmay Rauth. Four release candidates in a week is not a failure story. It is what a healthy verification culture looks like when a young implementation is still hardening its release process.
Spec precision got its own dedicated attention. Andrei Tserakhau called a shows why this dry-sounding clarification matters: Tserakhau noted the ambiguity bit someone again just last week on the Go side, where compacting a Spark-written table produced incompatible manifests. Kevin Liu suggested keeping the spec explanation format agnostic, and the fix landed in PR review. Small spec ambiguities compound into real interoperability bugs once five language implementations write the same metadata.
The function catalog work crossed a milestone when huaxin gao's , and the follow-up targets small file workloads where Iceberg currently issues two GETs for the footer and one for data when a single GET could fetch the whole file. The companion idea to , building on the earlier expressions proposal to let defaults be computed rather than constant. Xiening Dai and Maninder Parmar continued working through , which has been waiting since mid-May. Robert Kruszewski noticed that , a small operability win for anyone who has stared at an opaque commit failure at 2 AM.
Community infrastructure had a moment too. Bob Thomson from ASF Infra reported that to halve PR runner minutes, and Ajantha Bhat pointed to his open PR doing exactly that plus incremental CI builds, which has been waiting for review. On the events side, the this October already gives the EMEA community a major gathering, co-located with Community Over Code and with its agenda now live. Kessler also announced the with namespace and table management now supported, and huaxin gao posted notes from both the series.
Apache Polaris
Polaris generated 114 emails across 23 threads this week, and the volume tells you something: this project is in the thick of working out what a production catalog service owes its operators.
The biggest thread by message count was, surprisingly, about the merge button. Jean-Baptiste Onofré opened a PR to when a conflicting operation is in progress. The current behavior returns a 500, which Dmitri Bourlatchkov reviewed and declared most certainly not correct, since 5xx codes signal fundamental service failure beyond the client's control. The candidates each have problems: 503 implies the whole service is unhealthy, 429 means rate limiting and is not defined for rename in the Iceberg REST spec, and 409 traditionally signals a conflict the client should not blindly retry. Seventeen messages in, the thread had become a genuinely useful seminar on REST semantics for catalog operations. The resolution matters beyond Polaris, because whatever convention Polaris adopts will influence how clients across the ecosystem implement retry logic for concurrent catalog operations.
Operational maturity drove a cluster of related threads on events and metrics. Yong Zheng raised the need for a , sketching expressions that match on catalog, namespace, and table name. Bourlatchkov floated CEL as the filter language before recalling that prior community consensus leaned toward removing CEL, leaving include and exclude lists with glob patterns as the likely landing spot. The largest design question in this cluster came from Yufei Gu, who proposed , prompted by a PR that shifted delivery to a blocking executor. The previous behavior implicitly relied on Vert.x event bus semantics that nobody had written down. Kollár argued listeners should be documented as thread-safe and that strict ordering rarely matters as long as every event arrives, and Gu took the pragmatic position: keep ordered delivery as the only behavior now, and introduce unordered delivery only if a real need appears. Documenting implicit guarantees before users depend on them accidentally is exactly the kind of unglamorous work that separates production infrastructure from promising prototypes.
JB's . EJ Wang reported no must-have blockers and plans to cut from main, while Adnan Hemani asked to land one PR first, a fix for a documentation versioning issue that had gone unreported for a while. JB updated the release process documentation to match. In parallel, the project took a step toward friendlier adoption when Yong Zheng proposed won quick support from Russell Spitzer, who endorsed taking determinism out of table creation paths as a safety improvement. Bourlatchkov raised an important operational catch: with randomized locations, long-running staged create operations like CTAS face a credential refresh problem, connecting to the , with authorization aspects deferred. And the , Russell Spitzer shared a proof of concept he implemented with Claude's help, comparing approaches for testing file operations without real cloud containers. Robert Stupp agreed the POC clarifies the layering problem and they converged on a split: synthetic FileIO for generated listings and pure file operation behavior, real containers where fidelity matters. Bourlatchkov also opened threads on through a REST spec change, and a subtle by working through what Polaris should do when lineage events reference non-Polaris datasets on both ends, with Stupp calling for broader community input because the options on the table represent materially different commitments. And Sankaran proposed a after RC1 hit a verification problem that Ed Seidl fixed. Verification reports came in from Seidl on RHEL 8, Raúl Cumplido on Debian 14 with Rust 1.96, Adam Reeve on Fedora 44, and L. C. Hsieh, and Lamb produced one of the more thoughtful AI governance conversations in the ASF right now. After two weeks of testing, Cumplido found the reviews useful for ready PRs but wants them disabled for drafts, since a draft signals work in progress and an immediate bot review adds noise. Lamb agreed they help as an initial pass and pushed for documenting what contributors are expected to do with bot feedback. Sutou Kouhei synthesized the feedback into a PR with a pragmatic split: first-time contributors get one policy, returning contributors another. Alenka Frim asked the practical question nobody had answered, which is when Copilot actually considers itself satisfied with a PR, since nobody had seen it grant an approval. Arrow is writing down norms for AI participation in code review while most projects are still improvising, and other communities will likely copy this homework.
The format itself saw movement on two fronts. The surfaced a coordination problem: Gang Wu pointed out that several duplicate efforts are underway on variant support in Arrow C++, including work by his colleague Zehua that iceberg-cpp already depends on. Micah Kornfield confirmed community interest and pointed to the freshly opened tracking issue. Duplicate implementations of the same type are wasted effort the dev list exists to prevent, so expect consolidation here.
The Arrow family also grew. Following the donation vote, Benjamin Philip for prepared statement results with four binding +1s and work proceeding on Go, Java, ADBC, and JDBC implementations, while Richie Black's , Arrow's continuous benchmarking project, with Mihevc interested in having his agents work on it and Keane happy to see anyone pick it up. The phrase "having your agents work on it" passing without comment in an ASF dev thread says plenty about where 2026 is.
Arrow's quieter week should not be mistaken for a quiet project. The format has reached the stage where its biggest contributions happen downstream, in arrow-rs powering DataFusion and a growing share of the Rust analytics ecosystem, in ADBC and Flight SQL steadily replacing bespoke wire protocols, and in the C++ library serving as the substrate for iceberg-cpp and the engines built on it. That last dependency is why the variant duplication issue deserves a faster resolution than it might otherwise get. With Iceberg, Parquet, and Spark all converging on variant as the standard answer for semi-structured data, Arrow C++ sits in the critical path for every engine that wants to read shredded variant columns efficiently, and two parallel implementations means review attention split exactly where the ecosystem can least afford it. Wu naming the problem publicly, with a disclaimer about his colleague's involvement, is the dev list doing its job.
Apache Parquet
Parquet packed 72 emails into just 7 threads, and one of them was the week's heavyweight across the entire lakehouse ecosystem.
The . Pitrou pushed back with a precise objection: the page invents an a posteriori meaning for V1 and V2, and he questioned why parquet-format 2.0.0 deserves to be singled out as a meaningful boundary. Lamb conceded that earlier drafts did try to invent definitions and revised toward describing what shipped rather than what the labels should have meant. This exchange is the versioning debate in miniature. The community is discovering that before it can design future versioning, it has to agree on a truthful account of past versioning.
While the philosophy unfolded, the release train kept moving. Gang Wu confirmed in the collected binding +1s from Kornfield and others, with Seidl's vote carrying the best line of the week: we have waited long enough for usable float statistics. Sortable floating point statistics have been a known gap for years, and 2.13.0 finally closes it.
The footer redesign work formalized its process. Jiayi Wang scheduled from design doc to pull requests against parquet-format and the reference implementation, after the Parquet sync aligned on keeping the field simple and minimalistic. Daniel Weeks followed up with additional context from the sync discussion. A File logical type gives engines a standard way to represent file references inside Parquet data, which matters for multimodal and document-heavy workloads where tables increasingly point at external binary content. And Divjot Arora closed the loop on the long-running featured the kind of drama that proves verification works. Matt Butrovich cast a -1 after Comet, the Spark accelerator built on DataFusion, showed large performance regressions on TPC-H and TPC-DS at scale factor 1000 that appeared related to Parquet metadata parsing. Andrew Lamb connected it to a similar report from Adam in the Vortex project tied to new metadata cache size limits. Butrovich investigated further, found Adam's issue went through the ListingTable API that Comet does not use, could not reproduce the regression in DataFusion alone, and retracted his -1 while deferring the Comet upgrade for more investigation. Lamb then after crowdsourcing input from the community, and opened the , citing the 54.0 release, the new Java bindings, and a remarkable growth trajectory. Meanwhile Bob Thomson's infra review brought good news on the resource front: — Build your lakehouse on Iceberg with a free trial
— O'Reilly book, free download
SOCIAL SHARE CARD GENERATOR