The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened a formal vote to adopt versioned releases for breaking changes, borrowing a governance model that Iceberg refined over years. Apache Polaris canceled a vote on its semantic model API so it can align with Apache Ossie, the freshly incubating semantics project that opened its dev list this week. And across Iceberg, Arrow, and Parquet, contributors debated who owns statistics, which format features deserve to survive, and how far the specs should bend to serve AI and machine learning workloads. The connective tissue this week is governance. These communities are building the rules for evolving open formats without breaking the millions of tables that already depend on them. That work is invisible when it goes well, which is exactly why it deserves a close read while it happens. Releases also kept pace with the design debates: Polaris shipped 1.6.0, Arrow Rust shipped 59.1.0, and both Iceberg Rust 0.10.0 and Arrow 25.0.0 entered their final voting rounds.
Apache Iceberg
The Rust implementation dominated release activity this week, and the path was not smooth. For readers who mostly touch Iceberg through Spark or a query engine, iceberg-rust deserves a moment of framing. The Java implementation remains the reference, but the Rust library has become the foundation for a second generation of tooling: pyiceberg-core binds it into Python, DataFusion integrates it for query processing, and a wave of lightweight services use it to read and write tables without a JVM. When this library ships a bug, the blast radius crosses language ecosystems, which explains the ceremony you are about to read. Danny Jones and Shawn Chang . The willingness to cut a third candidate rather than wave through a flawed second one says something about how seriously the Rust community takes its release checklist, which covers everything from ASF license headers to a clean build of the pyiceberg-core bindings. The Rust library now sits underneath a growing stack of Python and query engine integrations, so the caution pays forward.
The week's most consequential design debate concerned table statistics, a topic that sounds dry until two engines start fighting over the same files. Some background helps here. Query engines rely on statistics, things like distinct value counts and data distributions, to choose good execution plans. Join two tables in the wrong order and a query that should take seconds takes minutes, so statistics quality translates directly into compute cost. Iceberg lets engines write statistics files and attach them to a snapshot in table metadata. That design works cleanly when one engine owns a table. The trouble starts in the multi-engine deployments Iceberg was built for, where Spark handles ingestion, Trino or Dremio serves interactive queries, and Impala or Flink sits somewhere in the mix, each with its own idea of which statistics it wants. Dzeri96 raised concerns about , with Kevin Liu joining the exchange. Today, delegated access in the REST catalog operates at table scope, and that granularity is the limitation Hyun wants to fix. If a consumer should only see a subset of partitions, administrators either over-provision access or fragment tables to match access boundaries, and both options create operational pain. Hyun's proposal uses pre-signed URLs during scan planning to make partition-scoped sharing practical without restructuring tables. Concretely, a catalog planning a scan for a restricted consumer returns signed links only for the files that consumer is entitled to see, and the storage layer enforces the boundary because unsigned paths simply fail. The idea builds on an earlier delegation thread and moves Iceberg closer to the fine-grained sharing models that commercial platforms offer on top of the format. That last point carries strategic weight. Fine-grained access control is one of the few remaining capabilities where proprietary lakehouse platforms hold a clear edge over the open spec, and standardizing it in the REST catalog narrows that gap for every open implementation at once. Nevin Zheng added a related thread on , describing production experiments with primary-key-oriented semantics for CDC and mutable-data workloads. His team treats the primary key as table metadata and then builds behavior on top of it: key-aware write semantics, storage organization, compaction strategy, and changelog generation for incremental processing. He framed this as complementary to the existing constraints proposal, which limits PRIMARY KEY and UNIQUE to informational metadata and leaves enforcement out of scope. The distinction matters because it sketches a two-layer future. The spec standardizes how keys are declared, and engines opt into richer key-aware behavior above that line. Anyone watching the upsert and CDC space, where formats like Apache Hudi and Paimon built key handling in from the start, should track this thread closely. A related note from , showing the same pressure arriving through the Rust door.
Geospatial work is quietly becoming one of Iceberg's most active frontiers. Sunmin Lee proposed , with Tanmay Rauth engaging on the bbox thread. Together these ideas sketch a spatial stack for the format: partition data by spatial cells on write, prune by bounding boxes on read. The economics mirror what min-max statistics did for numeric columns years ago. A query asking for events inside a city polygon should never open files whose contents sit on another continent, and today that pruning depends on engine-specific tricks rather than portable metadata. Iceberg v3 introduced geometry and geography types, so the type system groundwork exists, and these proposals show the community moving from type support to the performance engineering that makes spatial workloads economical. Fleet telemetry, logistics, climate data, and location-based applications all stand to benefit, and they represent exactly the data volumes where file pruning changes the bill.
The variant type effort keeps a steady drumbeat. Variant is the shredded, binary-encoded type for semi-structured data that lets JSON-shaped payloads live in tables with columnar performance, and it spans both the Iceberg and Parquet specs, which makes coordination the hard part. A field promoted to a shredded column by one writer has to remain readable by every engine, and the type touches file format, table format, and engine layers at once. That is why the community runs a dedicated sync for it. Neelesh Salian , which touches how partial column updates get written to storage.
Spec hygiene rounded out the Iceberg week. Daniel Weeks opened threads on . Sung Yun raised a . These threads rarely make headlines, but they are the reason independent implementations of Iceberg in Java, Rust, Python, Go, and C++ produce the same answers on the same tables. Every ambiguity closed on the list is a bug that never ships, and the current volume of clarification work reflects the v3 and v4 feature waves pushing the spec into corners nobody had to define precisely before. Spec text that a single reference implementation can leave fuzzy becomes load-bearing the moment a second implementation reads it differently.
A few more items deserve a sentence each. Szehon Ho called a , which matters for anyone whose sort order needs to survive engine boundaries. Daniel Weeks flagged an to expose property history for audit and debugging. And Renjie Liu suggested , the proposal that would let Polaris store and serve business metric definitions alongside its catalog duties. Jean-Baptiste Onofré asked the community to pause. His reasoning: the Open Semantic Interchange specification is transitioning into the Apache Ossie project, which entered incubation on June 22, and an initial Ossie spec release is expected soon. Implementing the OSI spec now and the Ossie spec later means doing the work twice. Robert Stupp and Adam Christian weighed in across the thirteen message thread, and Gu canceled the vote after gathering the feedback. He noted the practical middle path along the way: most of the semantic model work in Polaris can proceed in parallel, since the main dependency on Ossie is a JSON validator that arrives later, and the API can ship marked as beta with an explicit plan to converge on the Ossie specification.
Gu then did exactly that kind of parallel work, opening a focused design thread on the , which drew verification from Dmitri Bourlatchkov, Yong Zheng, Francois Papon, Robert Stupp, and Ajantha Bhat. The , which lets Polaris create its own Hikari connection pool from configuration instead of depending on the Quarkus-managed datasource. Three details make this more interesting than typical plumbing. Different configurations can create independent datasources, which lays groundwork for future per-realm datasource routing, a meaningful capability for multi-tenant deployments. JDBC drivers can load at runtime from a jar rather than living on the build-time classpath, which solves a genuine licensing problem for drivers like MySQL that carry Apache-incompatible licenses. And the change is opt-in, since Polaris keeps the existing Quarkus path when no JDBC URL is configured. Dmitri Bourlatchkov, Robert Stupp, and Onofré worked through the design across seven messages.
Configuration cleanup and operational clarity threaded through the rest of the week. Gu proposed , trimming legacy persistence code, and Dmitri Bourlatchkov opened a discussion on , building on the earlier discussion Gu, Bourlatchkov, and Dutra held about across six messages. The same contributor raising Terraform providers on two project lists in one week is worth pausing on. It signals that lakehouse components have crossed a maturity threshold in the eyes of platform engineering teams, who now expect to manage catalogs, principals, grants, and table definitions the way they manage VPCs and Kubernetes clusters: declared in code, reviewed in pull requests, applied by CI. When the infrastructure-as-code ecosystem starts building first-class providers for your project, it means production adoption arrived ahead of the tooling, and the tooling is catching up.
Community texture rounded things out. Rich Bowen interviewed project members for the ASF's , Ayush Saxena asked about , which matters once catalogs hold thousands of entities and clients need server-side filtering rather than full listings.
Apache Arrow
Arrow entered release season on two fronts at once. Raúl Cumplido and . His case is thorough. The Tensor message arrived in 2017 and SparseTensor in 2019, both remain marked experimental, neither appears in the cross-implementation integration test suite, neither is reachable from Flight RPC or the C++ Dataset API, and a GitHub code search turned up no third-party usage. Meanwhile the messages carry maintenance cost, including recent security reports about missing validation on untrusted input. Arrow now has canonical extension types that carry dense tensors as ordinary RecordBatch columns, with room to add sparse variants if demand appears. Rok Mihevc and Weston Pace joined the discussion. Deprecating unused format surface is unglamorous work, but every retired feature shrinks the attack surface and the implementation burden for the ecosystem.
Kent Wu opened a discussion that fills a long-standing usability gap: , drawing responses from Felipe Oliveira Carvalho and Weston Pace. Pitrou and Mihevc discussed the , and Pitrou collected contributions for the quarterly board report. Ian Cook hosted the July 1 community call.
Apache Parquet
Parquet produced both the busiest thread and the most consequential vote of the week, and the two stories reinforce each other.
The busiest thread first. The . The mechanics are straightforward. Forward-incompatible changes accumulate against the next major version of the Parquet spec, new breaking changes automatically target the version after the current one, and the community votes to close and adopt each major version as a unit. Blue noted the lineage with a smile: Iceberg contributors call this the Iceberg model, though Iceberg originally inherited the idea from Parquet. His , which concerns versioning the specification artifacts themselves.
Why does versioning deserve this much ink? Because Parquet sits at the bottom of nearly every analytics stack on earth, and it has historically evolved through feature flags and reader capabilities rather than clean version boundaries. Forward incompatibility is the scary direction of change. Backward incompatibility breaks old files, which communities avoid at all costs. Forward incompatibility breaks old readers, meaning a file written with a new feature fails, or worse, silently misbehaves, in software that predates the feature. With Parquet readers embedded in everything from Spark clusters to embedded databases to decade-old ETL jobs nobody dares touch, the community needs a way to ship new capabilities without playing compatibility roulette. Recent additions like Variant, Geometry, and new encodings strain the old model past its limits. A predictable major-version mechanism means query engines can advertise Parquet 3 support as a coherent unit, vendors can test against a fixed target, and users can reason about compatibility without memorizing a feature matrix. The INT96 story this week shows the flip side of format ambiguity: Micah Kornfield . Hardwood targets a gap that has annoyed JVM users for a decade: parquet-java carries heavy Hadoop dependencies, and plenty of applications want to read Parquet without dragging in a distributed filesystem stack. A microservice that reads a few Parquet files from S3 should not inherit hundreds of megabytes of Hadoop transitive dependencies and their security advisories. The C++, Rust, and Go ecosystems solved this with clean standalone readers years ago, and Hardwood brings the JVM the same option. Community response was warm and practical. Pritam Pan asked about Spark integration, and Morling sketched a plausible path where an engine keeps its own decoders but adopts a Hadoop-free metadata, IO, and filtering layer underneath. Steve Loughran connected Hardwood to his pending parquet-testing fixtures for malformed files, and Morling reported the results candidly: Hardwood rejected nearly everything, but several rejections happened incidentally rather than for the right reasons, a few fixtures exposed missing validation, and one surfaced a latent bug. That exchange is open source working as designed, with a new implementation and a shared test corpus hardening each other in public. It also echoes the security thinking in Loughran's separate thread asking , another compatibility question the new versioning policy will make easier to answer in the future.
Encodings research added a forward-looking note. Prateek Gaur opened threads on . ALP, short for adaptive lossless floating point, exploits the fact that most real-world floats are decimals in disguise, encoding them as scaled integers that compress far better than raw IEEE bits, with an exact fallback for values that resist the trick. It has shown strong results in the research literature and in modern engine formats, and floats dominate ML feature data, so the pairing with the FIXED_SIZE_LIST discussion is natural. Between the fixed-size type, float-native encodings, and the footer redesign work, a picture emerges of Parquet systematically retooling for the vector era rather than ceding that ground to specialized formats. The healthy sign is that each piece arrives through the normal proposal process, with benchmarks attached, rather than as a rushed response to competitive noise. Divjot Arora proposed , Jiayi Wang convened in parquet-java. Julien Le Dem gathered input for the July board report. Add it up and Parquet had one of its most substantive weeks of the year.
Apache Ossie (Incubating)
Apache Ossie opened its doors this week, and readers of this newsletter got a preview of why it matters in the Polaris section above. Ossie comes from Open Semantic Interchange, and it defines a vendor-neutral specification for expressing business metrics, dimensions, and their relationships, so a definition like monthly active users means the same thing to every tool that touches it. The design philosophy matters as much as the format. Rather than point-to-point field mappings between tools, Ossie standardizes the ontology and lets systems read semantic metadata straight from the source, which means the meaning travels with the data instead of being retranslated at every boundary. The project ships two main components, the specification itself plus bindings and converters from existing formats, so teams with definitions locked inside today's tools have a migration path. Ossie entered the Apache Incubator on June 22 with a mentor bench that signals how seriously the data community takes it: Jean-Baptiste Onofré, Zili Chen, Russell Spitzer, and Holden Karau, names readers of this newsletter will recognize from Iceberg, Polaris, and Parquet threads.
The dev list came alive on July 8. Onofré posted the has been populated. He followed with a : Build your lakehouse on Iceberg with a free trial
: O'Reilly book, free download
SOCIAL SHARE CARD GENERATOR