samkhya v1.0: Plug Claude, GPT-4o-mini, or Local Ollama Into Your SQL Query Optimizer

TL;DR

samkhya v1.0 ships the first LLM-pluggable corrector backend for an embedded SQL query optimizer. Plug Anthropic Claude (claude-opus-4-7, claude-sonnet-4-6), OpenAI GPT-4o-mini, or local Ollama (llama3.2:1b) into the cardinality-estimation slot of DataFusion, DuckDB, or Polars via a 4-line HTTP wire contract. Two reference servers ship in the box: Python FastAPI (canonical) and Node TypeScript (broader operator appeal). It's all wrapped in a provable safety envelope so a hallucinating LLM cannot make your plan worse than the engine's native estimate. Apache-2.0, sole author. 13-crate Rust workspace.

• cargo add samkhya-core --features llm_http · pip install samkhya




• + does it. The crime is this: every time the process dies, eight years of incredible cardinality-estimation research dies with it, and the next session starts over from zero. The HyperLogLog your midnight ELT job built? Gone. The Bloom filter that knew exactly which customer IDs lived in that 400 GB Parquet partition? Gone. The histogram that took twelve seconds to compute over the join key? Gone. The optimizer wakes up at 9 a.m. with no memory of anything it ever learned and goes scanning the same columns again.

  
  
  

  That waste isn't a bug. It's the inevitable consequence of a missing library— a library so obvious in hindsight that the absence is almost embarrassing. Apache DataSketches gave the world sketches but never wired them to a query optimizer. Iceberg's Puffin specification gave us a sidecar format but no producer/consumer library to fill it. AQO, the PostgreSQL-only adaptive estimator, has feedback but no portability between engines. The three pieces have been sitting on three different shelves for years, and nobody has bolted them together.

  
  
  

  I shipped samkhya v1.0.0 today to bolt them together. And to do four other things while I was at it.

  
  
  

  Repository: . This is the engineering tour.

  
  
  

  
  
  
  The five-layer stack, in plain English
  

  
  
  

  samkhya is सांख्य— Sanskrit for "enumeration, counting"— the name of the classical darshana whose entire discipline is counting reality's constituents honestly. The library has exactly that job. It is a 13-crate Cargo workspace under a single Apache-2.0 license with the explicit §3 patent grant, structured as five replaceable layers, each failing safely toward the engine's native plan when the layer above is missing.

  
  
  

  Layer 1— portable stats. Five classical sketch families ship in v1.0: HyperLogLog (precisions 4–18, measured RSE 0.676% at p=14 / n=10⁶— comfortably below the Flajolet 2007 0.8125% envelope); Bloom filters via Kirsch-Mitzenmacher double hashing; Count-Min; equi-depth histograms; and a 2D correlated histogram that captures the pairwise column dependencies the four scalar sketches miss. Each sketch carries a stable KIND tag (samkhya.hll-v1, samkhya.bloom-v1) and a to_bytes / from_bytes serialization contract. A Puffin sidecar produced by the Python wheel is byte-identical to one produced by the DuckDB extension and fully readable by the DataFusion adapter. That byte-identity is the moat. No engine owns the stats. The sidecar does.

  
  
  

  Layer 2— feedback recorder. The recorder hooks query execution at the adapter boundary, captures (plan template, estimated rows, actual rows) triples, and writes them to a SQLite sidecar keyed by template. A per-template residual model— a gradient-boosted tree under 100 KB on disk in the default backend— learns the systematic bias between what the planner thought and what actually happened, then surfaces the correction as a hint. This is the observe-and-hint pattern from Stillger's LEO (IBM, 2001), Marcus's Bao (SIGMOD 2021 Best Paper), and Anneser's AutoSteer (VLDB 2023)— the only learned-QO pattern with documented production deployment, full stop. Cold start sees the native plan. The recorder fires only when it has evidence.

  
  
  

  Layer 3— LpJoinBound. The never-regress guarantee, made provable. This is the non-negotiable contract every other layer must honour. Every corrected estimate is bounded from above by a pessimistic ceiling derived from LpBound (Zhang et al., SIGMOD 2025 Best Paper)— LP relaxation over ℓp-norms of degree sequences, no machine learning involved. The samkhya refinement is strictly tighter than the Atserias-Grohe-Marx AGM bound (PODS 2008) on the star-5 join family at p=1. Not approximately tighter. Strictly tighter on every cell of the 30-cell evaluation grid. Wilcoxon signed-rank test gives W=0, p=1.73×10⁻⁶— a complete dominance result. Translated to wallclock: 40.95× speedup over native DataFusion 46 LpBound tightness, BCa 95% CI [30.93, 47.45]. A correction that breaches the LpJoinBound ceiling is rejected and the native estimate is used in its place. The worst case is silently degenerate. Never catastrophic. That is the whole point.

  
  
  

  Layer 4— GPU batch inference (opt-in, via gpudb). Subplan enumeration is embarrassingly parallel: each candidate is an independent forward pass through a small GBT or PFN. CPU does this serially. The GPU collapses what would be a thousand-iteration loop into one CUDA or Apple Silicon Metal kernel launch. When samkhya is paired with my gpudb extension, the correction model scores thousands of subplan candidates in a single launch. Strictly opt-in: the default cargo build --release --workspace links no CUDA, no Metal, no GPU runtime of any kind.

  
  
  

  Layer 5— the pluggable Corrector backend. This is the headline. One Rust trait (Corrector), four shipped backend slots:

  
  
  
  
  

  • GBT (default). Gradient-boosted-tree, sub-MB on disk, sub-millisecond inference, no external dependencies. The conservative classical bet, and what the cold-start safety analysis assumes.

  
  

  • TabPFN-2.5 (opt-in, tabpfn_http feature). Hollmann et al. ICLR 2023 + Prior Labs 2026— foundation tabular model. Measured P95 31.15 ms at batch size 8, sequence length 128, on RTX 4090 Laptop (BCa 95% CI [29.39, 35.32]); q-error reduction vs GBT on synthetic 7.84% (BCa 95% CI [2.21, 14.62], p=1.04×10⁻⁵).

  
  

  • LLM-via-HTTP (opt-in, llm_http feature). The viral one. samkhya-core::residual::llm::LlmHttpCorrector calls an HTTP server you control. Two reference servers ship in samkhya-gpudb/scripts/— llm_infer_server.py (Python FastAPI, canonical) and llm_infer_server.ts (Node TypeScript port, broader operator appeal). Documented backends: Anthropic Claude (claude-opus-4-7, claude-sonnet-4-6), OpenAI GPT-4o-mini, local Ollama (llama3.2:1b via http://127.0.0.1:11434). Plug your own foundation model in ~50 lines of glue. Wire contract is dead-simple: POST /infer {"features": [...], "baseline_estimate": <u64>} → {"estimate": <u64>}.

  
  

  • Dummy (transport-floor). Echo backend for measuring wire overhead independent of model latency. Produced the H1-A PASS— P95 0.07–0.11 ms across batch sizes 1/4/8/16/32, proving the LLM plug works at sub-millisecond cost when the model isn't the bottleneck.

  
  
  
  
  

  The safety story is what makes this sane. Every backend— Claude, GPT, Ollama, GBT, TabPFN, anything you write tomorrow— is clamped from above by the LpJoinBound ceiling at Layer 3. If a hallucinating LLM returns "the join is 10¹² rows," LpJoinBound says "the provable ceiling is 4.2 million" and the planner sees 4.2 million. A miscalibrated TabPFN, a stale GBT, an LLM with a cosmic-ray bit-flip— none of them can break never-regress, by construction. That's the contract the field has been waiting for: let foundation models help with query planning, without giving them the keys.

  
  
  

  What's measured, what's projected— being explicit. The transport-floor latency (0.07–0.11 ms P95) and the wire contract are MEASURED on the dummy backend (full receipts in bench-results/19_llm_corrector.md). The live-LLM end-to-end latency cells— Claude (~1.2s P95 paper-projection), GPT-4o-mini (comparable), Ollama (latency-bounded by local hardware)— are PROJECTED pending API-key budget and the 30-trial measurement campaign. The mechanism ships in v1.0; the headline live numbers are next-revision work. I would rather you know that now than discover it when you read the bench-results dossier.

  
  
  

  The LpJoinBound clamp, visualised. Claude (amber), GPT-4o-mini (cyan), and local Ollama (emerald) each contribute a cardinality estimate; the crystalline lens— the provable pessimistic envelope— refracts and bounds every stream before it reaches the query-plan tree below. A hallucinating LLM cannot exceed the ceiling. The worst case is the engine's native plan.

  
  
  

  
  
  
  Now the part nobody else in this field tells you
  

  
  
  

  The standard playbook in learned cardinality estimation— for about eight years now, across roughly fifty papers— has been: hand-tune a system against a target workload, report the geometric mean of the wins in the headline, mention the losses in a §9 disclosures section that nobody reads. The credibility deficit that resulted from this playbook is exactly as large as the cumulative gap between those papers and the production deployments that followed them. Which is to say large. Naru is dead. NeuroCard is dead. MSCN is dead. DeepDB is dead. BayesCard is dead. None of them shipped into a production database. The 2021–2022 critique papers (Are We Ready For Learned CE?, In-depth Study of Learned CE) wrote the obituary collectively. The field has been quiet about why ever since.

  
  
  

  Before the WAVE4-F head-to-head against native DataFusion 46 on the IMDb Join-Order-Benchmark Slow subset (n=55 paired warm-cache, scale factor 1), I pre-registered three upper-bound performance claims. ≥1.6× geometric mean on join-heavy queries. ≥1.35× on aggregate-heavy. ≥1.50× overall headline.

  
  
  

  All three were falsified.

  
  
  

  The measured geometric mean is 1.038× wallclock— BCa 95% CI [1.026, 1.056], Wilcoxon W=212 p=3.00×10⁻⁶. The Benjamini-Hochberg FDR procedure at q=0.05 rejects the null on 24 of 55 cells. The full record across the suite is 17 wins / 38 ties / 0 losses. Never-regress holds— that is the whole point of the LpJoinBound clamp— but the magnitude of the wins is small, and I am reporting it as such, in the headline section, where this sentence lives.

  
  
  

  The TabPFN-2.5 backend had two pre-registered hypotheses. H1-A (P95 below 50 ms) passed comfortably. H1-B (≥15% q-error reduction over GBT on synthetic) failed on magnitude— measured reduction is 7.84%, statistically real, half the pre-registered effect size. The paper reports H1-B as falsified.

  
  
  

  I am telling you this here, in the launch post, on the first day, in the second-most-prominent section, because the alternative is the §9-disclosures playbook that hollowed out this field's credibility in the first place. The cost of admitting a falsified pre-reg in public is smaller than the cost of having someone discover it on their own three months later. The 17/38/0 record is the appropriate evidence to weigh: a real, modest, statistically significant, never-regress improvement, in the only kind of public paired-warm-cache head-to-head whose results would actually deserve to be believed.

  
  
  

  The interesting number in this release is not the JOB-Slow geomean. It is the 40.95× LpJoinBound tightness over AGM on the star-5 family. That is a theoretical-tightness result, with a wallclock translation on the synthetic topology, and it is the right way to communicate what this library actually contributes.

  
  
  

  
  
  
  The 1000 → 42 demo (thirty seconds to feel the mechanism)
  

  
  
  

  The repository ships a stats_propagation_demo example that proves the end-to-end path in plain Rust. A 1000-row table wrapped in DataFusion 46's default TableProvider reports num_rows = 1000 to the physical plan. Wrap the same provider with SamkhyaTableProvider plus the optimizer rule, and the physical plan reports num_rows = 42. The example prints, verbatim: "without rule: 1000, with rule: 42."

  
  
  

  The DataFusion integration is intentionally a five-line change:
  
  

  
  
  
  
  

  CODE Kopieren

  use datafusion::prelude::SessionContext;
  
  use samkhya_datafusion::{SamkhyaTableProvider, SamkhyaOptimizerRule};
  
  
  
  let ctx = SessionContext::new();
  
  ctx.state().add_optimizer_rule(Arc::new(SamkhyaOptimizerRule::default()));
  
  let provider = SamkhyaTableProvider::wrap(inner_provider).with_puffin_sidecar("orders.puffin")?;
  
  ctx.register_table("orders", Arc::new(provider))?;
  
  

  
  
  
  
  
  
  
  

  The samkhya_leaves_seen diagnostic on the optimizer rule confirms the corrected stats reached the physical plan. No fork of DataFusion required. DataFusion 46's Distribution framework already accepts external column statistics— samkhya simply supplies better ones.

  
  
  

  
  
  
  What is in the v1.0 box, plainly
  

  
  
  

  13 crates. Approximately 266 #[test] blocks across the workspace. 17 property tests. ~31 million cargo-fuzz executions, zero crashes. Criterion microbenchmarks for sketches and Puffin I/O. clippy -D warnings clean. The full workspace builds in under two minutes on a laptop with no network access. An ACM Artifact Evaluation v1.1 reviewer entry ships in , the falsified pre-regs are in the headline section because that is where they belong, and the bench-results/ directory in the repository is where the receipts live— I would rather have the argument there than in the abstract.

  
  
  

  The library's whole job is to count reality's constituents honestly. The release is doing the same thing about its own measurements. Both are deliberate. Both are how this field starts to look like the production field it always wanted to be.

  
  
  

  samkhya v1.0.0— Apache-2.0. Repository: . Companion to the prior gpudb release— the two share the embedded-tier engine target, the GPU-optional architecture, and the single-Apache-2.0 license posture.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen