Three TODOs, three weeks, one weekend: finishing pq v0.14

$ pq sales.parquet 'group_by .country | sum .revenue | top 3 by sum_revenue'

┌─────────┬─────────────┐

│ country ┆ sum_revenue │

╞═════════╪═════════════╡

│ US      ┆ 19065.00    │

│ FR      ┆ 999.99      │

│ DE      ┆ 312.00      │

└─────────┴─────────────┘

Where it started. I work in adtech. I look at parquet files dozens of times a day — campaign deliveries, partner exports, audience snapshots. Every existing option was painful:

pq is the tool I actually want — single binary, no JVM, no Python, jq-style syntax for piping into the rest of the unix toolbox. It's been my default cat for parquet since v0.5.

Demo

• 
  Repo: (this submission)


• 
  Install: brew install thehwang/parq/pq
  


• 
  Tutorial: ) closes #2: streaming JSON via a hand-written incremental array writer ([, then row, then ,\n + row, then ]). Memory stays flat, head -c 200 returns instantly.


• 
  PR #9 () closes #4: pq diff as a new subcommand, markdown by default, JSON for tooling, exit 1 on drift. Detects added / dropped / type-changed columns, plus nullability changes (which are breaking for downstream consumers and easy to miss).


• 
  PR #8 () closes #5 (the v0.14 tracking issue): bump Cargo.toml to 0.14.0, README v0.14 section, full reference manual entry (doc/reference.md §14), tutorial Lesson 6.

By the numbers:

• 5 PRs merged, 5 issues closed


• Test count: 204 → 215 (+8 unit, +3 integration)


• All CI green: macOS, Ubuntu, tui smoke (vhs)
  


• 
  v0.14.0 tagged, release workflow building macOS arm64/x86_64 + Linux musl + Windows binaries + Homebrew bottle

What I'm proudest of isn't the line count — it's that the milestone is empty. No "ship it and clean up later" comments left in the code. README's "What's coming" section now has nothing in the v0.14 row to delete. That itch is gone.

Postscript: the cover image caught DuckDB lying

The cover image at the top of this post? It almost wasn't.

After tagging v0.14.0 I tried to record a custom cover showing off the new pruning gauge — a 5M-row parquet file with id < 1M, expecting a satisfying green pruned: 80% line. What VHS captured instead was a wall of JSON debris over the panels, plus this:

● pruned: 0% (25.0M/5.0M rows)

25 million scanned rows out of a 5 million row file. That's not a typo — it was wrong twice over.

Bug #1: a silent PRAGMA. The pruning code sets DuckDB's enable_profiling='json' to grab the JSON profile, then "resets" with PRAGMA disable_profiling afterward. Against DuckDB 1.10.501 that pragma is a silent no-op — accepts the call without erroring but doesn't actually flip the bit. Subsequent EXPLAIN ANALYZE calls (and the TUI runs one on every preview tick) kept returning JSON in column 1, which then bled into the rendered panel as garbage. The documented inverse enable_profiling='no_output' is what actually works. I only found this out by writing a Python probe and trying every reset spelling DuckDB's docs hinted at.

Bug #2: the wrong field. With JSON no longer leaking, the cover redrew — and the numbers still made no sense (25.0M/5.0M). Turns out operator_rows_scanned from DuckDB's JSON profile is roughly 10× a parquet scan's actual row count (likely an internal multi-pass / per-thread accumulator). The correct field for "rows out of scan after pushdown" is operator_cardinality. Same query against the same file: cardinality 1.0M, total 5.0M, ratio 0.8 — the green 80% I was expecting in the first place.

Both went out as , closes · brew install thehwang/parq/pq · pq --help