TL;DR — DuckDB, DataFusion, Polars, and Postgres each compute and store table statistics their own way, so a histogram built in your ELT pipeline is invisible to your query engine's optimizer. samkhya fixes this by serializing sketches into versioned Iceberg Puffin sidecars that any adapter can load unchanged, and clamps whatever corrections come from that shared data with a provable pessimistic bound so a stale or wrong sketch can never make a plan worse.
Here's a scene that plays out in more data platforms than anyone admits. Your ELT pipeline runs nightly, and as a courtesy it computes real statistics on the tables it writes — a HyperLogLog for distinct counts, an equi-depth histogram for the skewed customer_id column, maybe a 2D correlated histogram for the pair of columns everyone joins on. Good stats. Expensive to build. Then a completely different process — DataFusion in your production query service, DuckDB on an analyst's laptop, Postgres serving a dashboard — runs a query against the same table and estimates cardinality using none of that work. Each engine falls back to its own sampling, its own defaults, its own idea of what "recent enough" statistics means. The nested-loop join that looked fine in DuckDB during development scans four billion rows in DataFusion at 3 a.m., because DataFusion never saw the histogram your pipeline already paid to compute.
The stats you built are not the stats your engine trusts
The naive fix is "just run ANALYZE everywhere." Postgres gets its own ANALYZE pass. DuckDB samples on read. DataFusion pulls whatever statistics live in Parquet footers or gets set explicitly on a TableProvider. Polars does its own thing on its own DataFrame. You end up computing the same statistical facts about the same data four separate times, in four separate formats, on four separate schedules — and they drift. The Postgres stats are fresh because autovacuum just ran. The DataFusion side is stale because nobody re-registered table stats after the last ingest. The optimizer that matters for your actual production query path is working from the oldest, cheapest information in the stack, and it doesn't know it.
The deeper reason this is hard isn't laziness — it's that statistics have always been treated as engine-local cache, not as a shared data asset with its own lifecycle. There's no standard on-disk format for "here is a histogram, here is what it's a histogram of, here is when it was built" that a query planner from a different vendor can just read. Every engine invented its own internal representation because none of them needed to talk to each other. The moment you run a lakehouse with more than one query engine on the same Iceberg or Parquet data — which is now the normal case, not the exception — that assumption breaks.
A sidecar file, not another engine-specific cache
samkhya's answer is to stop treating statistics as engine property at all. It serializes classical sketches — HyperLogLog for distinct counts, Bloom and Count-Min for membership and frequency, equi-depth histograms, and 2D correlated histograms for join-key pairs — into versioned, KIND-tagged blobs inside an Iceberg Puffin file. Puffin is already the Iceberg-native container for exactly this kind of sidecar metadata; samkhya just fills it with a defined, portable payload instead of leaving each engine to invent its own encoding. The ELT pipeline that builds the histogram writes the Puffin sidecar once, next to the table. The DataFusion adapter and the client-side DuckDB adapter load that same sidecar unchanged — no re-derivation, no re-sampling, no format translation. The sidecar owns the stats; no engine does.
That portability is the whole point, but it would be reckless on its own — portable stats can still be stale, and a sketch built from yesterday's ingest can mislead an optimizer just as easily as no sketch at all. This is where samkhya's second mechanism matters: it never lets a correction — whether sourced from the portable sketches, a gradient-boosted-tree corrector, or an LLM-backed corrector over HTTP — reach the optimizer un-clamped. Every proposed row-count estimate is capped from above by an LpJoinBound, a provable pessimistic ceiling computed via LP relaxation over the ℓp-norms of the join's degree sequences (an idea drawn from Zhang et al.'s LpBound work, not a reimplementation of it). There's no machine learning inside that ceiling — it's pure combinatorics on the actual data structure, so a corrupted sketch or a hallucinating model can suggest whatever it wants and still can't push a plan past a bound the system can defend mathematically.
The v1.1 release makes this concrete for the DataFusion path specifically: SamkhyaPreJoinRule and install_pre_join_corrector apply a corrector immediately before DataFusion's join-selection step, with native estimates kept as the safe default floor — below-native estimates require you to explicitly opt in. That's a deliberate design choice: the system defaults to trusting the engine's own numbers and only overrides them when you've said it's allowed to, which is exactly the posture you want when you're introducing a shared, cross-engine statistics source into a planner that didn't ask for one.
Correctors are swappable; the clamp isn't
Because the sketches and the clamp are separate from whatever produces the "improved" estimate, samkhya lets you swap correctors without touching the safety guarantee. The default is a sub-megabyte gradient-boosted-tree model with no GPU requirement. An LLM-pluggable corrector is available over HTTP, with a Python FastAPI reference server and a parity Node/TypeScript port speaking the same wire contract — v1.1 adds end-to-end tests enforcing that contract across malformed input, batch limits, and lossless 64-bit integer handling in Node, which matters more than it sounds like once you've debugged a JSON round-trip that silently truncated a row count. You can put a language model in your query optimizer's estimation loop specifically because the clamp means it structurally cannot make your plan worse than the bound allows — it can only fail to help.
What this doesn't fix — and the honest numbers
Be clear about the limits. The Puffin sidecar solves the portability problem — one set of stats, read consistently by DuckDB, DataFusion, and other adapters — but it doesn't solve every optimizer difference between engines. Each engine still has its own cost model, its own physical operator choices, its own notion of memory pressure. Shared cardinality estimates make the input to those decisions consistent; they don't make the decisions identical.
And on the actual numbers: samkhya pre-registered ambitious speedup targets — at least 1.35x geomean, higher for join-heavy queries — and the measured result across 55 real queries was a geomean speedup of 1.038x. That's statistically real (95% CI [1.026, 1.056], Wilcoxon p = 3×10⁻⁶) but it is a falsified result against what was pre-registered, and the project reports it that way rather than rounding it into a win. The results were 17 wins, 38 ties, 0 losses — the never-regress clamp does exactly what it promises, zero regressions — but the caveats are real too: warm-cache only, CSV rather than Parquet, a small query budget, and an out-of-memory failure past one particularly heavy query. If you adopt this, adopt it for the guarantee — never worse, sometimes better — not for a speedup you haven't verified on your own workload.
The project used as the worked example in this column is independent open-source work (Apache-2.0). Every number above is reproducible from a fresh clone.
SOCIAL SHARE CARD GENERATOR