Solution overview
I’ll use a hypothetical Orders table to demonstrate running analytical queries with Athena across various order-related dimensions:
We’ll look at how Zero-ETL integration handles nested fields, sets, and lists of maps but first let setup the configuration.
Integration configuration
Let’s walk through the steps to configure the integration.
1- Configuring the DynamoDb source table
Before getting started, Point in time recovery (PITR) must be enabled on the source table:
to trigger a lambda function that creates or deletes the integration whenever a the database is created or deleted—pretty much like a CloudFormation custom resource.
import { GlueClient, CreateIntegrationCommand, CreateIntegrationResourcePropertyCommand, DeleteIntegrationCommand, CreateIntegrationTablePropertiesCommand } from "@aws-sdk/client-glue";
import { SSMClient, PutParameterCommand, GetParameterCommand } from "@aws-sdk/client-ssm";
export const handler = async (event) => {
let glueClient = new GlueClient({ region: process.env.AWS_REGION });
let paramStore = new SSMClient({ region: process.env.AWS_REGION });
if(event.sourceArn == null || event.targetArn == null || event.roleArn == null) {
throw new Error("SourceArn, TargetArn and RoleArn are required");
}
if (event.tf.action === "create") {
const integrationResult = await glueClient.send(new CreateIntegrationCommand({
IntegrationName : event.integrationName,
SourceArn : event.sourceArn,
TargetArn : event.targetArn,
}));
const integrationResourcePropertyResult = await glueClient.send(new CreateIntegrationResourcePropertyCommand({
ResourceArn: event.targetArn,
TargetProcessingProperties: {
RoleArn: event.roleArn
}
}));
await glueClient.send(new CreateIntegrationTablePropertiesCommand({
ResourceArn: integrationResult.IntegrationArn,
TableName: event.tableConfig.tableName,
TargetTableConfig: {
PartitionSpec: event.tableConfig.partitionSpec ? event.tableConfig.partitionSpec : undefined,
UnnestSpec: event.tableConfig.unnestSpec ? event.tableConfig.unnestSpec : undefined,
TargetTableName: event.tableConfig.tableName ? event.tableConfig.tableName : undefined
}
}));
await paramStore.send(new PutParameterCommand({
Name: event.integrationName,
Value: JSON.stringify({
integrationArn: integrationResult.IntegrationArn,
resourcePropertyArn: integrationResourcePropertyResult.ResourceArn
}),
Type: "String",
Overwrite: true
}));
return;
}
if (event.tf.action === "delete") {
const integrationParams = await paramStore.send(new GetParameterCommand({
Name: event.integrationName,
}));
const { integrationArn } = JSON.parse(integrationParams.Parameter.Value);
await glueClient.send(new DeleteIntegrationCommand({
IntegrationIdentifier: integrationArn
}));
return;
}
};
I’m using the .
4- Glue resource policy
Since I’m using the Glue catalog for the integration, I made sure to include the following permissions in the glue catalog resource policy. This allows for integration between the source DynamoDB table and the target Iceberg table:
data "aws_iam_policy_document" "glue_resource_policy" {
statement {
effect = "Allow"
principals {
type = "AWS"
identifiers = [
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:root",
aws_iam_role.manage_zero_etl_integration_role.arn
]
}
actions = [
"glue:CreateInboundIntegration",
]
resources = [
"arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog",
"arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}",
]
condition {
test = "StringLike"
variable = "aws:SourceArn"
values = [data.aws_dynamodb_table.this.arn]
}
}
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["glue.amazonaws.com"]
}
actions = [
"glue:AuthorizeInboundIntegration"
]
resources = [
"arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog",
"arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}",
]
condition {
test = "StringEquals"
variable = "aws:SourceArn"
values = [data.aws_dynamodb_table.this.arn]
}
}
depends_on = [
aws_iam_role.manage_zero_etl_integration_role,
aws_lambda_function.manage_zero_etl_integration_fn
]
}
resource "aws_glue_resource_policy" "this" {
policy = data.aws_iam_policy_document.glue_resource_policy.json
}
You can find this configuration in the official docs
You can view the details. By default the refresh interval from the source DynamoDb table to the Iceberg table is set to 15 minutes, it is not editable for now:
.
☝️ Note that the shippingAddress was un-nested and deliveryPreferences was replicated as an array. That’s very convenient. However,items property was inferred as string. Since it’s a list of maps in DynamoDB, I expected it to map cleanly to a list of structs in Apache Iceberg, but it didn’t quite get the schema right.
The items property ends up as a plain JSON string in this DynamoDb list format, It’s not perfect, but we can work around it by using
Here’s an example query using Athena to get the number of orders grouped by city:
Smooth Sailing? Not Quite!
I encountered some limitations while experimenting with Glue Zero-ETL. Since it’s still relatively new (at the time of writing), I expect there may be updates and improvements over time. I’ll keep this blog post updated as things change:
IaC support
Deploying services through the console is not my preferred approach. As mentioned earlier in this post, currently, neither CloudFormation nor the AWS Terraform provider supports Glue Zero-ETL. I used the AWS SDK to create the integration and configure table properties. While this approach works for now, it’s not ideal. I expect that support for CloudFormation and Terraform will be introduced soon.
Handling DynamoDb List of Maps
Lists of Maps aren’t supported (yet?). Since Apache Iceberg tables can handle lists of structs, the lack of support for this feature could complicate more advanced use cases with complex table schemas. In such cases, running a custom ETL job remains a better solution.
Custom partitioning configuration
When setting up the integration, you can configure target table properties, such data partitioning as using the primary key from the DynamoDB table or specifying a custom partition:
await glueClient.send(new CreateIntegrationTablePropertiesCommand({
ResourceArn: integrationResult.IntegrationArn,
TableName: event.tableConfig.tableName,
TargetTableConfig: {
PartitionSpec: event.tableConfig.partitionSpec ? event.tableConfig.partitionSpec : undefined,
UnnestSpec: event.tableConfig.unnestSpec ? event.tableConfig.unnestSpec : undefined,
TargetTableName: event.tableConfig.tableName ? event.tableConfig.tableName : undefined
}
}));
However, while I was able to define custom partition configuration through both the console and the AWS CLI, it didn’t seem to take effect:
isn’t very clear on this point, but hopefully, it gets updated soon!
Support for AWS services other than DynamoDb
Thank you for reading and may your data be clean, your queries be fast, and your pipelines never break 😉
Resources
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/glue/command/CreateIntegrationCommand/
SOCIAL SHARE CARD GENERATOR