This is Part 9 of a 15-part covered embedded catalogs. This article explains the five ways Iceberg table storage degrades and how to detect each problem before it impacts query performance.
An Iceberg table that works well on day one will not work well on day 365 without maintenance. Every append, update, and delete operation adds files and metadata. Without periodic cleanup and reorganization, query performance gradually deteriorates until someone notices that a dashboard that used to load in 2 seconds now takes 30.
Table of Contents
1. The Small File Problem
) leaves orphan files.
Expired snapshots: When snapshots are expired, the metadata references are removed, but the underlying data files remain in storage until explicitly cleaned up.
Compaction: When . If it exceeds 1,000, configure snapshot expiry to keep the count manageable.
4. Sort Order Decay
If a table has a declared sort order (e.g., sorted by
customer_idfor efficient lookups), new data written by different engines or pipelines may not respect this sort order. Over time, the min/max statistics per file widen as new unsorted data is mixed with sorted data.
Impact: File skipping becomes less effective. As described in to rewrite files in the correct order and restore tight min/max ranges.
5. Partition Skew
Some partitions grow much larger than others. An event table partitioned by
day(event_time)might have 10 GB on a normal day but 500 GB during a promotional event. The oversized partition contains files that are too large or too numerous for efficient processing.
Impact: Queries against skewed partitions are slower because they must process disproportionately more data. Parallel execution becomes unbalanced when one partition's task takes 50x longer than the others.
Real-World Degradation Timeline
Consider a table receiving 100 small appends per day from a streaming pipeline:
Day 1: 100 small files (3 MB each), 300 MB total. Queries are fast.
Day 30: 3,000 small files, 9 GB total. Query planning starts to slow noticeably.
Day 90: 9,000 small files, 27 GB total. Every query scans all 9,000 manifest entries. Dashboard queries that took 2 seconds now take 15 seconds.
Day 180: 18,000 small files plus thousands of orphan files from expired snapshots. Metadata file is 50+ MB. Planning alone takes 10 seconds before any data is read.
Without compaction, the table becomes nearly unusable for interactive analytics within 6 months. With daily compaction, the same table stays at 40-50 well-sized files regardless of how many commits happen each day.
How to Diagnose Table Health
that let you inspect table health. Here are the key diagnostic queries:
Check File Sizes (Dremio / Spark)
CODE-- Average file size
SELECT AVG(file_size_in_bytes) / 1024 / 1024 AS avg_mb
FROM TABLE(table_files('analytics.orders'))
If average file size is below 32 MB, you have a small file problem. Target: 128-512 MB.
Check Snapshot Count
CODE-- How many snapshots exist?
SELECT COUNT(*) AS snapshot_count
FROM TABLE(table_snapshot('analytics.orders'))
If snapshot count exceeds 1,000, you should expire older snapshots.
Check File Count Growth
CODE-- Files per partition
SELECT partition, COUNT(*) AS file_count
FROM TABLE(table_files('analytics.orders'))
GROUP BY partition
ORDER BY file_count DESC
Partitions with hundreds of files are candidates for compaction.
covers all three approaches in detail.
The cost of not maintaining Iceberg tables is both direct (wasted storage from orphan files) and indirect (slow queries leading to poor user experience, excessive cloud compute costs from reading unnecessary data). Organizations with hundreds of Iceberg tables often find that a single data engineer dedicated to table maintenance saves more in compute and storage costs than their salary. Automated maintenance through by Alex Merced (Manning)
by Alex Merced
by Alex Merced
SOCIAL SHARE CARD GENERATOR