AI-powered metadata correction and harmonization

As data collection and data generation accelerate, the gap between our ability to produce raw data and our capacity to standardize it continues to widen. Without automation, this gap becomes a critical bottleneck that delays analysis, complicates interpretation, and limits the global value of shared datasets.

Metadata harmonization (standardizing labels, identifiers, and formats so datasets from different sources can work together) remains largely manual. AI-powered metadata correction and harmonization offers a way forward, transforming metadata management from a time-consuming responsibility into a process that scales with your data volume and supports open science rather than obstructing it.

In this post, we demonstrate how AI-powered metadata correction works in practice, explore two implementation approaches (from human-in-the-loop validation to fully autonomous agent-driven workflows), and provide governance considerations for deploying these solutions in your organization.

Metadata correction and harmonization workflow

To address this challenge, we developed a centralized metadata correction and harmonization workflow built on AWS, designed to support consistency, interoperability, and accuracy across disparate metadata sources. The system uses for schema and result storage, for authentication, and and frontend development.

node -v          # 18 or higher

python --version # 3.11 or higher

aws --version    # AWS CLI configured with credentials

Getting started

Step 1: Clone and install dependencies

Clone the repository and install all Python and TypeScript dependencies:

git clone https://github.com/aws-samples/sample-intelligent-metadata-harmonization.git

cd metadata-harmonization



# Create and activate Python virtual environment

make createPythonEnvironment

source .venv/bin/activate



# Install all dependencies (Python packages, TypeScript packages, CDK, frontend)

make install

The make install command installs the processor, API, agent, and evaluation packages as editable Python packages using uv, and runs npm install for the infrastructure and frontend projects.

Step 2: Configure the deployment

Copy the configuration template and edit it with your AWS account details:

cp config.yaml.example config.yaml

Edit config.yaml:

appName: "metadata-harmonization-app"

env: "dev"



dev:

  profile: "your-aws-profile"

  deploymentName: "metadata-harmonization-dev"

  accountNumber: "123456789012"     # Your 12-digit AWS account number

  region: "us-east-1"

  deploymentStage: "dev"

  removalPolicy: "destroy"

  logLevel: "INFO"

  targetPlatform: "linux/amd64"

Validate the configuration:

make validateConfig

Step 3: Deploy infrastructure

Bootstrap AWS CDK (first time only), then deploy:

# First-time setup: prepare your AWS account for CDK

make bootstrap



# Deploy all AWS resources

make deploy

make deploy builds the API container image, pushes it to Amazon ECR, and deploys the AWS CDK stack. This creates Amazon DynamoDB tables, Amazon S3 buckets, an Amazon Cognito user pool, and an Amazon ECS Fargate service.

After deployment, generate a local .env file with the deployed resource identifiers:

make createLocalDotEnvFile

Step 4: Create Amazon Cognito users

After deployment, create users in the Amazon Cognito user pool so they can log in to the application:

1. Open the to get started with the solution.
   
   

   ---

   
   

   About the authors

   
   
   Vollständiger Original-Bericht

   Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf aws.amazon.com.

   ↗ Original-Artikel auf aws.amazon.com lesen