When does Iceberg beat Parquet+projection on AWS Glue, and when doesn't ?

because I didn't have one of this kind yet and, having worked on data ingestion with Glue for a while, I wanted to gather in one place three things: how to structure code so it stays testable, which Firehose and Glue features to use and on what criteria, and a few Docker and Terraform gems I'd always promised myself to slot in somewhere.

Plus, I had never set up Glue streaming from scratch, and for a personal project I needed a test bed to compare Iceberg and Parquet + partition projection on the same data flow and under the same Athena queries, to figure out when one solution wins over the other and why.

This project mixes a lot of the experience I've gathered over the years with a couple of curiosities I hadn't had a chance to test. So there are no real challenges here: I already took those hits long ago. What I'm sharing is deliberate choices, driven by knowing these services inside out.

The architecture in the image describes exactly this project: a Python producer simulating stock tickers, a Kinesis Data Stream as the single entry point, two Firehose streams persisting the same flow in two different formats (Iceberg and Parquet), two Glue jobs that write to both formats (one batch for OHLC computation on 1m and 5m, one streaming for anomaly detection via z-score on a sliding window), and Athena querying both databases.

The choices and why

The goal was to compare Glue batch and Athena on top of an Iceberg-based database and a Parquet + partition projection one.

The record schema the producer writes (ticker_symbol, sector, price, change, event_timestamp) isn't something I made up: it's the one from the official AWS Firehose demo. That demo configures a single Firehose; this PoC configures two in parallel, one for Iceberg and one for Parquet+projection, to compare both storages on top of the same source. The Kinesis Data Generator is the tool the demo uses to produce the dataset, but rewriting it as a Python producer with boto3 gave me control over the scenarios (stable, trend, spike, mixed) and made them testable in pytest. The scenarios feed Glue streaming, which handles anomaly detection: spike injects controlled price spikes to validate z-score detection on anomalies, stable and trend act as baseline to avoid false positives.

As a lazy developer, the criterion is always the same: less effort, in terms of time, code or cost. Two rows of the table deserve a deeper look: --LOAD_DATA_MODE raises the question of read modes, the 3-line wrapper carries the code organization that makes TDD possible. I'll cover them one at a time, starting with reading.

Performance and read modes

To understand why the three LOAD_DATA_MODE exist, you have to start from the choice of , letting you read them from Glue with from_catalog and leverage the push-down predicate, , S3 LIST instead scales because it's :

Partition projection is usable only when the table is queried through Athena. If the same table is read through another service such as Amazon Redshift Spectrum, Athena for Spark, or Amazon EMR, the standard partition metadata is used.

So a Glue job reading the Parquet+projection database via from_catalog would fall back to standard partition metadata, which for a projection table aren't registered in the Catalog: no partition info available on the Glue side, full scan that goes nowhere, dead end. You have to go straight to S3 with spark.read.parquet, leaving Spark to handle :

A DynamicFrame is similar to a DataFrame, except that each record is self-describing, so no schema is required initially. Instead, AWS Glue computes a schema on-the-fly when required, and explicitly encodes schema inconsistencies using a choice (or union) type.

The access pattern shifts the balance between spark/parquet and iceberg as volume grows:

In practice, on large volumes lists two equivalent ways to apply the conf in the right place:

Create a key named --conf for your AWS Glue job, and set it to the following value. Alternatively, you can set the following configuration using SparkConf in your script.

Under the hood, the two configurations achieve the same result:

• 
  SparkConf in Python code:
  

  sc = SparkContext()

  conf = sc.getConf()

  conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")

  conf.set("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog")

  # ... other conf ...

  sc.stop()

  sc = SparkContext.getOrCreate(conf=conf)

  glueContext = GlueContext(sc)  # the SparkSession is born here with the right conf

The configuration lives in the code. The sc.stop() + recreation of the SparkContext is when the configuration gets "injected" before SparkSession init.

• 
  --conf in Terraform's default_arguments:
  

  locals {

    iceberg_spark_conf = join(" --conf ", [

      "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",

      "spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog",

      "spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog",

      "spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO",

      "spark.sql.catalog.glue_catalog.warehouse=s3://${data.aws_s3_bucket.main.id}/iceberg/",

      "spark.sql.defaultCatalog=glue_catalog",

    ])

  }

Glue parses the concatenated string, applies the configurations at SparkSession boot, and then hands control to the Python script.

I chose to configure the PoC via Terraform: why ? Three reasons:

• 
  a single source of truth: the iceberg_spark_conf local is defined once in Terraform and reused by both the Glue batch and the streaming via --conf = local.iceberg_spark_conf in their respective default_arguments. No per-job duplication, and if I add a third Glue job tomorrow I reuse the same local with a single line


• 
  separation of configuration and code: the catalog setup lives in Terraform alongside --datalake-formats=iceberg; the Python code of the jobs doesn't know an Iceberg catalog exists, it imports glue_common, takes spark and glue_context as parameters and runs


• 
  low-cost configuration changes: a different warehouse, catalog implementation or IO is touched only in Terraform, with no need to rebuild and re-upload the wheel

The configuration in code, on the other hand, stays handier when the catalog config depends on arguments the job receives at runtime (for instance a warehouse derived from the input bucket name passed as --ARG): in that case the conf is built naturally in the code, since you already have the resolved arguments there. In this PoC the warehouse is fixed per environment, so the configuration in Terraform wins on simplicity.

What else is there to add ?

Once the PoC has been signed off, you start to get serious: there's what was simulated to integrate, and other services and approaches to evaluate:

• 
  Real APIs: replace the simulated scenario with a real ingestion. It changes the producer's nature, not the architecture


• 
  Apache Flink as an alternative to Glue streaming: it makes sense when you need stricter guarantees on how many times an event is processed (Flink natively supports exactly-once, i.e. each event processed exactly once; Glue streaming is at-least-once and duplicates are handled at the application layer), or when the required latency is sub-second (Glue streaming, working in micro-batches, typically lands in the 5-10 second range; Flink drops to hundreds of milliseconds)


• 
  Multi-environment deploy: in a PoC, a single environment is enough. In production you need to separate so you can test feature rollouts without touching live data. So you introduce Terraform Workspaces or per-env modules, with all the implications for account management


• 
  CI/CD: in a PoC, manual make test and terraform apply are enough. Working in a team or on mission-critical pipelines you need automation (lint, test, build wheel, terraform plan automatic on every PR) to catch regressions before merge


• 
  Cross-account Data Catalog sharing: Lake Formation + RAM + KMS + assume_role. When the data lake aggregates flows from branches, departments, partners, the centralized schema changes everything


• 
  Data Management: the evolution of centralized Data Catalog sharing is DataZone or SageMaker Unified Studio, with lineage, asset-level permissions and per-asset documentation


• 
  Extra time frames in the batch as roll-up from 5m (1h, 1d), not from raw: each level computes on top of the previous level's output, hence on less data. It's a classic approach (cascade ETL) and works when the higher-level aggregate can be recomputed from the lower level (the high of one hour is the max of the highs of the 5 minutes). It doesn't work if the calculation needs to go back to the original values, like medians or exact distinct counts