🦄 Making great presentations more accessible.

This project aims to enhances multilingual accessibility and discoverability while maintaining the integrity of original content. Detailed transcriptions and keyframes preserve the nuances and technical insights that make each session compelling.

Overview

📖 AWS re:Invent 2025 - Data Processing architectures for building AI solutions (ANT328)

In this video, Sakti Mishra and Radhika Ravirala discuss transforming data architectures for AI readiness. They address the challenge that 89% of CDOs prioritize generative AI but over half feel their data foundations aren't ready. The session covers unlocking enterprise data for AI agents through Retrieval Augmented Generation (RAG) and Model Context Protocol (MCP) servers, demonstrating how to expose data lakes and warehouses to AI assistants. Key demos showcase auto-generating Spark code and visual ETL pipelines using Amazon Q Developer in SageMaker Unified Studio. Radhika details AWS enhancements including Trusted Identity Propagation for single sign-on, Lake Formation fine-grained access control, S3 Access Grants, SageMaker Notebooks with Spark Connect, and the Spark Upgrade Agent that migrates applications from version 2.4 to 3.5. The presentation emphasizes identity-based access control, integrated AI/ML environments, and productivity improvements in data processing engines like Amazon EMR, AWS Glue, and Amazon Athena.

; This article is entirely auto-generated while preserving the original presentation content as much as possible. Please note that there may be typos or inaccuracies.

Main Part

Introduction: The Data Foundation Gap for AI Initiatives

Hello, all. First of all, thank you for joining us. I hope all of you are enjoying re:Invent. My name is Sakti Mishra. I work as a Principal Data and AI Solutions Architect, and I'm joined with Radhika Ravirala, who is a Principal Product Manager for Amazon EMR. Today, we are going to talk about data processing architectures that can help you build AI solutions.

So this is a very high-level agenda that we will cover. First, we will try to highlight what is the role of data in AI. Then we will try to revisit what are the foundational pillars for a modern data foundation. Then we'll go deeper into how do you unlock your existing enterprise data for your AI agents. And then Radhika will go deeper into what are the enhancements we have done to our AWS data processing engines that are related to security and productivity.

Agentic AI Use Case: HR Onboarding System and Data Requirements

Now, let's understand the role of data in AI with a real-world use case. Let's assume you have an agentic AI HR onboarding system where you are sending a message saying, we have just hired Amy, please start onboarding. And with that, a network of agents are activated through a supervisor agent or through an orchestration agent. The first agent is a task planning agent, which is building a tailored onboarding plan for Amy. The second one is an onboarding buddy agent, which recommends Amy to meet the coworkers based on her profile, role, and background. And the third one is the exception handling agent, which gets kicked in if something doesn't go as planned. For example, maybe Amy's joining date is delayed, or maybe Amy's device gets delayed. Then the exception handling agent can rerun approvals, update the agents to make sure everything is going as per plan. For the HR onboarding agent to work effectively, it needs to get access to the structured datasets, unstructured datasets, vector stores, and real-time streaming datasets with the right level of security and governance.

Now let's understand what is holding our customers back. We can think of top three priorities. The first one is more related to people and prospects. We have different engineering roles in the organization, but now with the AI landscape, the distinguished line between these roles are starting to blur, where the data engineers, machine learning engineers, data scientists, and AI engineers all need to closely work together to build AI solutions. Now, the second part is customers have years of investment to build the data platform they have today, and they need to be agile to make it AI ready. But it's not easy to replace what they have. So how do we enable them to extend what they have to be exposed to the AI agents in an efficient way? And the third is we have a traditional way for the data processing that takes your data from raw layer to the lakehouse through a series of transformations. How do you bring in efficiency in the data processing layer itself that can improve your efficiency and productivity?

Retrieval Augmented Generation (RAG): Real-Time Data Integration for AI Models

How can you auto-generate Spark codes? How can you auto-generate your visual ETL pipelines? How can you automate your data quality or data lineage jobs? So let's try to touch base on the first part, which is basically unlocking your existing enterprise data for AI agents. To understand that, let's go through a real-world use case. Let's assume you have a customer AI chat assistant that is getting built by a customer service team of a bank. They have fine-tuned their model based on their previous data, and the end user comes and the end user asks, what is today's thirty-year fixed mortgage rate?

As I explained, RAG helps build or augment your prompts with the latest information, and it provides several benefits. One of them is improved accuracy, which is what we talked about. The second one is reducing hallucinations. Sometimes AI models, when they do not have the actual information, try to give wrong information. By providing the latest information, you are reducing that risk.

The next benefit is that you can bring in flexibility for domain adaptation. Let's assume you are trying to build a use case for the financial industry. You do not have an AI model which is specifically designed for the financial industry, so you create a knowledge base with financial industry data so that the model can use that financial data to give domain-specific answers.

Model Context Protocol (MCP) Servers: Connecting AI Agents to Enterprise Data

So let's expand our use case now. The AI model has answered the end user, saying the current mortgage rate is 6.25%, and it may change depending on your credit history and the down payment you're paying. Now, the credit history might be in your enterprise data, stored in your data lake or data warehouse. How do you expose your AI model to get access to your data lake or data warehouse? That's where we are introducing a concept called Model Context Protocol servers, or MCP servers.

These MCP servers will be registered as tools for your AI agents, and the AI agents will invoke the execution of SQL queries or maybe jobs that will get executed in your data lake and data warehouse. It will get the response back and answer the end user query. Let's understand what Model Context Protocol is and what its benefits are.

Now let's look at a high-level architecture for MCP. As you can see here, the end user is interacting with the MCP host. The MCP host can be an Amazon Q developer CLI or a Kiro CLI or a cloud desktop or a custom agent that you have built. The agent is trying to interact with the MCP server through a client. The MCP server can be many. I talked about data processing MCP servers.

Similarly, if you are trying to interact with Redshift, you have Redshift MCP servers. If you want to interact with S3 tables, you have S3 tables MCP servers. But beyond that, you do have flexibility to integrate custom MCP servers on other services. Now, this architecture highlights some of the AWS analytics services, as I said, and it's not limited to analytics services. You will also be able to invoke third-party models through APIs by defining MCP servers.

This is the second use case I highlighted where you are trying to build a data pipeline. You do have the option to use Python-based DAG design, or you can use drag-and-drop design in an interface, but this AI chat assistant is helping you to give a quick start, where you are giving a prompt that says, this is my source, this is my target, and these are the transformations I want, and with a single prompt, you will be able to see that it is automatically generating an ETL pipeline for you. Let's look at a demo for both of them.

It scans the data, automatically derives the schema for it. It'll give you a preview of the schema which you can verify, and then click Create Table. Once you do create table, you can refresh your catalog. So you expand the lakehouse, then you go to AWS Data Catalog, you see a database called sales marketing, then you see the diabetes data table created. When you click preview of the data, an Athena query gets executed, and it'll give you a sample of 10 records.

So once you verify that your data is onboarded, now you can navigate to the Jupyter Lab notebook by clicking a menu under Build. In the Jupyter Lab notebook, we have the Q chat assistant, where you can see on the top, it gives you an option to configure MCP servers. So we click that, and then we click the Add icon, which gives us the form. If you remember in the previous slide I have shown, this is where we are adding the MCP server. We specify the name. We specify the command and the argument which will be pulling the latest of it. When you click Save, it'll activate the MCP server. It'll also list the tools it has, which you can verify. It might be a Glue database or listing a Glue table or the connections.

So now the Q chat assistant is scanning the command you have given. It is breaking that into multiple tasks, and you will see in the bottom, it is going step by step. First, it is listing your local directory because it needs to create a notebook. Then it will try to list Glue connections. It is asking your human permission. You are executing run. After that, it'll ask you to execute the list of AWS Glue databases. When you click that, it lists the databases.

So for this demo to work, we needed a few data sets to be available. Now, that is already created in Amazon S3, which is basically a customer behavior CSV and a customer dimension CSV. Once you verify those two CSVs are available, we will navigate to the Amazon SageMaker Unified Studio. We have a corporate domain already created, and we will be navigating to the Amazon SageMaker Unified Studio portal. We have done previously this step where we select the project first, and then under the build, we navigate to visual ETL jobs.

Once you are on that page, you click Create Visual ETL Job. And you do have an option here to create a pipeline with a drag and drop interface, but we will show the generative AI capability with this prompt. And in this prompt, we are highlighting, use one data set as the customer behavior, the second data set as customer dimension, do a little bit of transformations by typecasting, changing, or renaming columns, then join these two data sets by state. And then create two aggregated columns, that is total purchase amount and total page views, and then finally store that into a new target table.

When you click submit, it will take some time to analyze the prompt and then creates the nodes automatically for you, where you have flexibility to click and edit the nodes. So now the pipeline is created, as you can see, and I explained you can click each node to edit, verify, and update. So we are first verifying this is the customer behavior CSV. We specify the delimiter as comma, and also we specify this CSV has header.

So, the next step will be joining these two data sets, but before that, we are verifying the second data set, which is the customer dimension CSV. Again, we are specifying comma as the delimiter, and we are specifying it has header. And once you do that, you will be able to see that you have proper column names. And then finally, the next step, which is basically joining the data set through a join, and we are using customer ID as the common column. And the next one is aggregation. When you are doing aggregation, you are aggregating by state,

Before we get there, let me just give you a quick recap of how customers are building AI applications. So here you see that the end user interacts with the generative AI application, typically by posing a question. The application then loads the relevant context and the conversation history. Now, before this can happen, when the user poses a question, it actually loads the prompt templates that can be applied to it. It could be one template, it could be multiple templates, and those templates refer to your Q&A or whether it is a code completion task or whether it is a summarization task, and so on and so forth.

Once it's done that, it loads all the conversation history if the user had already been engaged with the generative AI app on this topic. Now once the conversation history is loaded, the question posed by the user is then used to get more additional context based on the user profile. The user himself or herself can have preferences, specific permissions, and other settings that are situational in nature, such as the project that the user belongs to or maybe additional business rules that need to be applied. Now all this state information is stored in our data stores such as DynamoDB, and some of it comes from the data stores that Sakti was talking about earlier.

Once you have loaded your conversational history and you have the relevant context, the application then tokenizes this original question. To tokenize it, it sends it to, for example, Amazon Titan embeddings or OpenAI embeddings. Once the original question is tokenized, it derives the vector representation of that, and then it is sent back to vector stores such as S3 vectors or OpenSearch. Then, with those original question embeddings, the application performs a similarity search in the vector data store, and this is using some form of an approximate nearest neighbor search algorithm.

Once it applies that, it gets a bunch of results back, and it uses the top key results or document chunks as it calls from that. Once you have all those data available, it combines it with the original textual content, and then the data is synthesized and a prompt is engineered. That is the final prompt that will be sent to the LLM. The LLM uses that prompt along with all the embeddings and the original text content that was generated, and it processes the request and sends back a response.

The response is then sent back to the conversation history. It is sent back to other data stores as relevant, and then finally the response is sent to the end user.

Take the humble data lake. Your organization has been building data lakes for many years now. Data scientists and analysts access raw data in S3, but the environment faces some challenges, right? Like if you take this example, you have three different users: Al, the data analyst; Joe, the data scientist; and a group of users who belong to a BI group. And all these users are essentially building different applications, and they're trying to access the data in your central data governance account. Now to do that, they're all using different roles. The challenge here is that there are an explosion of roles because as the number of users increases, you have to add additional roles based on the permission set that you require for that role.

And then as the user base increases, you want to also make sure that they are able to access multiple roles to access multiple applications. So there's a pivot happening here from where Joe the data scientist has to go through several AWS accounts and assume different roles in order to get access to, let's say, pink data or green data, right? In addition to that, it is very difficult proving who accessed what data and when, especially for sensitive data.

And so there is an AI readiness barrier here for data scientists who spend 80% of their time trying to find, access, and prepare data for building models. Now, if you extend the example to transactional data lakes or lake houses that you're building with your BI applications, more challenges later on where you need to support open table formats and also be able to manage different catalogs. Maybe it is the Glue Data Catalog, or maybe you want to federate to a third-party catalog like Unity or the Horizon catalog.

Right, so a few themes have emerged looking at the challenges with all the existing data architectures that we have seen so far, and they include identity and access control. Customers have been asking for a seamless single sign-on capability with fine-grained access control so that you can do identity-based access control. They want support for open table formats with comprehensive auditing capabilities where you can track end user actions, and then you want a very good integrated AI experience while performing all your tasks with your data processing architectures. So let's see how we are working with our data processing engines to enable some of these features in the coming slides.

Trusted Identity Propagation and Fine-Grained Access Control with Lake Formation

So starting with identity, a couple of years back we introduced a feature called Trusted Identity Propagation. This is a feature from AWS IAM Identity Center which enables administrators to grant permissions based on user attributes such as user ID and group association. It is built on the OAuth 2.0 protocol and it allows you to add context to your existing IAM roles, and the IAM role with the embedded identity context can be passed to the downstream applications and services so that they can either propagate that identity or, if it's the end service which is doing the authorization, it can authorize based on that identity.

Now if we look at some of the benefits this feature has, you will notice that it allows for enterprises to build SSO-like experiences where data engineers and scientists access Apache Spark sessions in Jupyter Lab Notebook in SageMaker Unified Studio using their organizational or corporate credentials and not the IAM roles, eliminating the need for separate credentials and streamlining your authentication workflows. You have end-to-end traceability for the user actions, which means that you have comprehensive AWS CloudTrail logging that captures all activities from interactive Jupyter Lab sessions to the background processing jobs that are running on EMR and Glue and Athena.

You have a centralized security management where administrators can implement fine-grained access controls from Lake Formation and apply permissions based on that, and these permissions can go granular.

Here is how it works in the context of SageMaker Unified Studio. You have Charlie and Elle who are users of SageMaker Unified Studio. They're logging into the SageMaker Unified Studio portal, and their credentials are passed from their project role to, let's say, a SageMaker training job. It could be an EMR job as well, and the credentials for Charlie and Elle are then passed further downstream when they're trying to access an S3 bucket using their own credentials. So here what is happening is when Charlie logs into SageMaker Unified Studio, SageMaker Unified Studio through its integration with Identity Center is able to authenticate Charlie using a token, their job token or the token that they get from their corporate identity provider. That token is exchanged for an Identity Center token which gets embedded into their project role and passed on downstream as they pass it to the analytic services.

The idea behind S3 Access Grants is that when you're in scenarios or use cases where your simple IAM policies and bucket policies are hitting the limits, you want to switch to S3 Access Grants where you can overcome that. You can overcome the limits that you have on your bucket policies and simply use SQL-like grants to have a much simplified user-to-dataset permission mapping for your applications.

So here is how full table access works. There's an admin who actually enables full table permissions on Lake Formation for the tables that you're interested in. When the user submits a job, the EMR or AWS Glue jobs reach out to Lake Formation for getting the credentials. Now here, they don't go directly to S3. The credentials are rendered by Lake Formation, so these jobs reach out to Lake Formation. Lake Formation looks at the credentials for the runtime role for that EMR job, and then it vends the required scoped-down credentials for that user or the job role. Then those credentials are used by the EMR engine, the Spark engine, to read data from S3 and then return the results back to the user.

So here is how it works. You have an admin granting permissions to users or groups. Remember we talked about trusted identity propagation. So now, Lake Formation is also integrated with trusted identity propagation, which means that an admin can grant permissions at a user level. Once the permissions are granted in Lake Formation, when that same user submits a Spark job, the job, EMR takes the job role with the user's identity in it and reaches out to Lake Formation for credentials. Lake Formation evaluates the policies for that user, returns the scoped-down credentials on the table for that user or their group association, filters the data, and then it reaches out to S3, reads the data, filters the data, and then returns the results back to the end user.

So there is a whole slew of activities that are happening in the fine-grained scenario, and as a result there are stringent requirements on how you run fine-grained access control jobs on EMR clusters. As a result of that, because the filtering itself happens on the EMR engine, on the EMR Spark engine, we have to protect the area or the portion of the container which runs the filtering fleet or the filtering operation. As a result, there are some limitations here. We don't allow RDDs or anything that is invasive to securing the filtering fleet. So as a result, we have some considerations when you are using fine-grained access control.

There are no RDDs. We have limited support for debugging, and custom JARs and UDFs are also not supported in this scenario. So something to think about when you want to use fine-grained access control, and these are some of the limitations that we want to address in the future.

Integrated AI/ML Experience: SageMaker Notebooks, Spark Upgrade Agent, and Performance Enhancements

So with that, let me jump to a different topic, which is enriching our data architectures with integrated AI and ML experience. Now, earlier you have seen a few demos from SageMaker. Those were run in SageMaker Jupyter IDE. We also launched SageMaker Notebooks, which is an immersive experience for your data scientists, your data analysts, and your serverless fans who really want to get started within seconds. It intelligently plans and executes complex workflows. It has the capability to choose the language that is best suited for the task, and it allows you to scale your workloads effortlessly.

Let me talk a little bit about what this means. There's a Jupyter IDE experience which has the full set of features that an enterprise needs, which includes the Identity Center with its Trusted Identity Propagation, a full project construct that will allow you to organize your work, and a multitude of other features. But there are also a lot of customers who prefer to get started within minutes, and for those customers, SageMaker Notebooks is a fantastic tool to get started with. You can start quickly without having to pre-provision any data processing infrastructure.

The notebook gives your data engineers and your data scientists a place to perform SQL queries, execute your Python code, process large-scale data jobs, run machine learning workloads, and create visualizations without having to switch between tools. These notebooks are powered by Athena Spark, which is based on Spark version 3.5.6, which supports Spark Connect. I think some of you are already familiar with the decoupled architecture for Spark Connect where the client and server are decoupled. So the client code can reside or can be part of your IDE itself, and your clients can simply issue commands to a remote Spark running on a cluster, such as EMR, or it could be an EMR serverless application.

The Spark upgrade agent workflow works as follows. There is a planning and orchestration phase, a build and dependencies phase, a code modification phase, and a data quality guardrail phase. Let's see this in action through Sam, who's been tasked with urgently upgrading their production Scala-based Spark applications from version 2.4 to 3.5 after a critical security vulnerability was discovered. Normally this would take weeks or months to complete. Let's watch how the agent upgrades it very quickly.

I've posted the question into the queue agent. We have taken steps to launch the MCP, activate the MCP server as before, and we have posed the question. As you can see, it is building a plan very quickly to walk through the steps of the upgrade. There are multiple steps here including the upgrade for the dependencies, running all the unit and integration tests, validating on EMR on EC2, and it generates the full upgrade summary.

But in the interest of time, I'm just going to quickly close out on a couple other features that we have launched, which includes the fastest Spark. Now we are 4.4 to 4.5 times faster than open-source Apache Spark. We are 2 times better for writes compared to open-source Spark and Iceberg. We also introduced something called EMR Serverless storage provisioning. The idea behind this is to provide you with an external remote shuffle storage so that your EMR jobs can eliminate the disk bottlenecks and out of disk errors that occur.

There are multiple features that we have launched. This is just a glimpse of what we have launched in this year. There's more on catalog federation, materialized views that you can check out in various other talks and our blog posts. So with that, thank you all for coming, and I hope you'll give these features a try and share your feedback with us. Thank you.

; This article is entirely auto-generated using Amazon Bedrock.