🦄 Making great presentations more accessible.

This project aims to enhances multilingual accessibility and discoverability while maintaining the integrity of original content. Detailed transcriptions and keyframes preserve the nuances and technical insights that make each session compelling.

Overview

📖 AWS re:Invent 2025 - Unlock Advanced Model Training: Reinforcement Fine-tuning on Bedrock (AIM3327)

In this video, Amazon Bedrock introduces Reinforcement Fine-Tuning (RFT) with Nova Lite 2, democratizing advanced model customization without requiring deep ML expertise. Unlike traditional Supervised Fine-Tuning that demands massive labeled datasets, RFT learns from small examples through feedback-driven loops and reward functions. The feature offers three simple steps: upload data from multiple sources, define success metrics using model-as-a-judge templates or Lambda functions, and deploy with on-demand inference. Salesforce's Phil demonstrates real-world impact, showing their TaxEval model achieving 97% accuracy on instruction adherence—surpassing GPT-4o's 88%—while serving 130 million monthly queries. The demo highlights training metrics visibility, side-by-side base model comparison, and token-based pricing at $1/hour for training, with inference costs matching base model rates.

; This article is entirely auto-generated while preserving the original presentation content as much as possible. Please note that there may be typos or inaccuracies.

Main Part

Introduction to Reinforcement Fine-Tuning on Amazon Bedrock: Addressing the Limitations of Traditional Fine-Tuning

Hello everyone. Welcome. We are here today to talk about our newest feature launch, Reinforcement Fine-Tuning on Amazon Bedrock with Nova Lite 2, with support for additional models coming soon. Today we'll show you how we're going to democratize RFT and bring it to all developers on Amazon Bedrock without you needing to have deep ML expertise. My name is Shalendra and I'm a product leader on Amazon Bedrock. I'm here with my colleague Shreyas, Principal Data Scientist on AWS, and our customer Phil, SVP of Engineering on Agentforce team at Salesforce.

Let's start with some basics. What is fine-tuning and why do we need it? Fine-tuning is a way to adjust an existing base model so that it fits your specific use case. For example, teaching it your company's tone and style with your own examples like customer chats or documents, so it learns to give you more accurate and consistent answers for your tasks, for your domain-specific tasks. But why do we need it? Why is the base model not sufficient? It's because an out-of-the-box base model is trained on a vast amount of internet data. It knows a little about everything, but not the specific details or tone or style that your company needs.

For example, a base model may know how to write emails but not your company's specific style or format. It can summarize text but maybe not in your preferred tone or with the accuracy you need. And that's why fine-tuning is needed. It teaches the model to perform specific tasks for your domain more accurately, consistently, and efficiently than a generic base model can do.

Many of our customers also use traditional Supervised Fine-Tuning, also known as SFT. And we launched SFT in Bedrock a while ago. SFT means training a base model on a set of inputs and correct outputs that you provide. In simple terms, you show the model how you want it to respond. For example, in the slide for a classification task, you can see "I love this" is a positive sentiment, "it's OK" is neutral sentiment, and "terrible" is a negative sentiment. By training on many such examples, the model learns to follow your preferred tone and style, making the answers more reliable and task-specific.

Although SFT works great for certain use cases like classification and summarization, it does come with its own set of challenges. Number one, it's data hungry. Customers often struggle with high-quality labeled datasets. You must provide many clear and correct examples, and if your data is inconsistent or biased, the model will learn those mistakes as well. It's expensive and time-consuming. Collecting, cleaning, and labeling training data takes effort, and training large models requires compute cost. In other words, it's very data hungry and requires a large amount of high-quality labeled datasets, which is not possible for everyone.

Number two, it is rigid. Models may memorize but may not adapt. The model may become too focused on your examples. And three, model drift, where performance degrades over time. If your data or rules change often, then you have to retrain the model each time with new datasets to keep it current. SFT also doesn't really do preference learning and aligning with human preferences very well.

And we hear these examples from our customers. For example, we have 100 examples, but we need production-quality models. Our model works great with some small set of examples in testing, but drifts in production. We want structured outputs with reasoning but can't afford to annotate thousands of high-quality labeled datasets. These are the actual challenges that we have heard from our customers who have been using traditional Supervised Fine-Tuning.

How Reinforcement Fine-Tuning Works: From Exploration to Deployment in Three Simple Steps

But what if your model could learn from a very small set of examples, explore thousands of solutions automatically and use the best solution to improve itself, and then continuously improve from production data? Welcome to Reinforcement Fine-Tuning on Bedrock. With RFT on Bedrock, we'll allow customers to easily create models that continuously learn from feedback and scoring, and as a result deliver high-quality, intent-aligned outputs for your specific business needs. With our launch, we're going to democratize RFT and bring it to all developers using Bedrock with minimal ML expertise.

Unlike traditional fine-tuning techniques, customers don't need a massive amount of high-quality labeled datasets for training, nor do they need to have deep machine learning expertise to use this feature.

Reinforcement Fine-Tuning explores multiple solutions, scores them, and then uses a feedback-driven learning loop where models improve iteratively over time based on a reward signal and reward function that you choose.

Let's take a look at this example customer query. A customer service agent is interacting with a customer who said, "I want to cancel my subscription." The model here explores multiple solution responses. Response one says "cancel." It's helpful, but it's abrupt. I wouldn't like that response from a customer service agent, whether it's an automated agent or a human agent. Response two says, "I understand, but may I please ask what prompted that decision?" As you can see, this response is empathetic and will score higher based on the reward function that I will create. Response three says, "OK, here we go, please fill out this form." As you can see in this use case, the model is exploring multiple solutions which will be scored based on your reward function and which will be used to improve the model weights.

What happens in Reinforcement Fine-Tuning is that the system actually explores multiple solutions that you just saw. It then collects the reward signal on them and exploits the best-scoring ones by shifting model weights towards those patterns. Exploration ensures that new solutions are tested and explored, and exploitation ensures that the model increasingly prefers those that consistently yield higher rewards, striking a balance between exploration and exploitation techniques.

Now let's see how this Reinforcement Fine-Tuning works in Bedrock. It's pretty easy, just three simple steps. Provide your data from multiple sources, and we'll see those sources. Number two, define what good looks like with our out-of-the-box templates. You can create your own custom reward function as well using a Lambda. Then start training, and once the training is finished, deploy the fine-tuned model with our on-demand inference, pay-as-you-go, with no machine learning expertise required.

So here we are. You can go into the Bedrock console on the left navigation. You will see "Tune," so you select custom models and then click "Create reinforcement fine-tuning job." Next, upload your data, and you can choose to upload data from your computer or your S3 file, or provide access to your model invocation logs, all in the JSONL format in the OpenAI chat completion format. Here you can see you're uploading the data from any of those sources.

Then next, let's define what good looks like using a reward function. We'll give you two options with some out-of-the-box and editable templates: model as a judge along with evaluation rubric, which you can edit, and a Lambda function for verifiable rewards, which you can also customize and edit. Here you can see model as a judge. We bring you some out-of-the-box templates for a variety of tasks like instruction following, summarization, chain-of-thought reasoning, RAG question-answering faithfulness. We also provide you an evaluation rubric. You can edit this evaluation rubric for the judge prompt. Option two provides you Reinforcement Fine-Tuning with verifiable rewards through a custom Lambda function, which you can edit.

Once you select these and start your training job, we also give you visibility into the training metrics, for example, training rewards, validation rewards, the episode length for training and validation. Once the training is complete, you can just deploy the fine-tuned model with Reinforcement Fine-Tuning with on-demand inference. We have token-based pricing for inference that charges based on the number of tokens processed. For Nova Lite 2, our pricing for Reinforcement Fine-Tuning fine-tuned model inference is the same as base model inference, and for training, we're charging one dollar per hour.

Live Demo: Building and Testing an RFT Model with FinQA and Multilingual Datasets

Now Shreyas will actually show you a working demo of this feature. Shreyas, thanks. So what's coming up next is we're going to show you how easy it is to get started with reinforcement fine-tuning, and like Shalendra said, it doesn't expect you to be an expert in reinforcement fine-tuning. You don't have to be a scientist. You do have to understand the basic three steps, right? One, have your data ready. Number two, create your reward function. And number three, start your training, right? So let's get started with the demo.

To get started, you first go to your console. You can go to your console today. You'll see a new option for when you want to create a custom model called reinforcement fine-tuning.

The first thing you want to select is what model you want to fine-tune. We're selecting Nova 2 Lite here, with more open source models coming soon. We're also going to select this JSONL file, which is our dataset, and you can also select an S3 location or invocation logs. But what does this JSONL format data look like?

We're going to go through two quick use cases. One is FinQA, which is a financial question and answer dataset. This is just one view of that dataset. There are multiple records in this dataset, and that's what you see inside the JSONL. This is a nicer view of it. Inside one of those records, you see a really complex prompt. You see some table in there. You have context about financial information and performance from 2016 and 2018. The question we want to ask and we want to train the model to do well at is what is the total operating expense in 2018 in millions. This involves table understanding. It involves long context. It involves mathematical calculations as well as matching up to an exact answer that you want your model to be trained on.

Here's a slightly different dataset. This is a multilingual dataset called CHABSA. It's a sentiment analysis dataset, something that Shalendra touched on earlier. For this, you want to be able to analyze multiple languages, think through your reasoning, and then understand what the final sentiment is going to be. Here as well, you can exactly match up the sentiment that you want your model to train over. You could have thousands of these records.

The second main step is creating that Lambda function for your rewards. One of the ways to do your Lambda function is to write custom code, and we have nice templates that you can use. But if you want to write your own custom Lambda function, you can do that as well. That's what we're scrolling through here. For verifiable rewards, you can also down-select some pre-templated Lambda functions. This one is for format and constraints checking, so maybe you need that for structured output for JSON outputs. You can also have one for math reasoning, which is a pretty common use case. When you create your job, this Lambda function is going to be created on your behalf. You can do it on your own as well, but in this case, this is the full Lambda function. You have full control over what happens inside the Lambda function, but all you have to do at the end of the Lambda function is to return the reward scores for each of those samples that comes into the Lambda. That's the only requirement. In this case, we're just highlighting that you can walk through your samples and return a numerical score for each one of the records.

We'll go back to the console for a bit to explore this option too, which is using a Lambda function that has AI feedback built in. Rather than writing your Lambda function entirely again, they have some nice templates. You first choose your model that you want to apply with this template. You can down-select on that dropdown four different reasoning or LLM as a judge kind of templates. That's instruction following, summarization, reasoning evaluation, and RAG faithfulness. Each one of those corresponds to a rubric that we have tested and benchmarked against a wide variety of problems. For example, what we're seeing here is a rubric, which is simple text that's just going to describe how you want to score each one of the inputs that are coming in. We'll take a look at another one, which is instruction following. Here we have a different set of a way to score the model, and we also want to cap the final score from zero to one, which is not necessary. You can have any score, but it's just good practice to do that as well.

Once again, when you create that Lambda function, it's going to entirely create your calls to Nova Premier in this case, which is the model that we selected as LLM as a judge. You can also select GPT-4o S 120B. There are a few other options that you can set up. Obviously, you need some place to put your output data. This is where your training logs and metrics can go to. There are important hyperparameters that you can set visually on the console. That's number of epochs, the learning rate. Here it's set to one times ten to the minus five. You can set the number of training samples per prompt and the evaluation interval of how often you want to actually evaluate your current model with the validation dataset. Finally, you can create a role and then hit create. Behind the scenes, this is creating your Lambda function for your reward, your role, as well as starting training.

Now, remember, your training can take some time. Training is going through all of your examples. It's calling your reward functions repeatedly when you have to train your model. A few hours later, you will see training metrics. In this case, we're seeing training rewards and validation rewards, and you can see that the line is trending up, which means the model is learning. Your reward score is going up both on the training side as well as the validation set. You also have other metrics that are useful, like train episode lengths, which in this case is the average number of prompt tokens output.

You also have gradient norm and critic advantages that are very useful for, for example, your data science team to debug what's happening behind the scenes.

Lastly, once you've finished your training, you want to see how it actually works. The way to do that is to set up inference. What is really exciting is that you can actually set up on-demand inference and you don't have to set up provisioned throughput. What that means is you only pay for the tokens in and tokens out. So in this case, we're going to give it a name, RFT model on-demand V1, and the model is already pre-selected. This is the model that got fine-tuned with RFT. You can also choose other models, but in this case, we're going to choose the model that's pre-selected and hit create. That's going to take a couple of minutes. So what the video is going to show is an example of a model where we already have this on-demand inference already set up. So that's the test FinQA example. That's the first dataset that we're looking at, which is the financial services Q&A example. You can hit test in playground and that opens up the familiar playground that you've seen in the Bedrock console. I'm going to paste in here an example financial Q&A dataset from the test set. You see a table in there, you see a question, and then you want to match up to an answer. So as you can see, the model's already started. This is not, we don't speed this video up. This is actually real time. In three seconds, you get back the full answer based on this really complex financial services question.

On the left sidebar, you can also configure some inference parameters that you're looking at right here. Another really interesting thing to do when you're comparing with your base model is to hit that compare mode toggle up there, and we're going to go back and select the Nova 2.0 Lite that was the base model to do a side-by-side comparison of what improvement was made due to RFT. So once again you see it's equally performant as the base model. You also only pay for the tokens in and tokens out at the base model rate. So super easy to get started, and with that I will hand over to Phil. Phil is from the Agentforce team at Salesforce and will talk about Agentforce as well as how they've been using RFT. Phil, thank you.

Salesforce's Agentforce 360: Achieving 97% Accuracy with TaxEval Using RFT

Hi, everybody. I want to start with a little bit of a historical moment here. About a year ago in December, there was a paper that came out. Some of you may have heard of it as the DeepSeek V3 paper, December last year. And then in January of earlier this year, another paper came out from the same group called the R1 paper. Behind the technology there that really, I think, took down Nvidia by 25% when the paper came out was the technology that we're going to talk about today. So in case you're wondering, historically, where does this come from?

I want to use this time to kind of describe why something so technical is impacting a company like Salesforce and the type of enterprise AI system that we are building. So in front of you is a slide that, for those who come to Dreamforce, is a slide that is probably shown about, I don't know, 20,000 times. And what this is about is Agentforce. What is it? Well, it is an enterprise AI system, an enterprise AI platform. It is not a new product. All that it is is a new veneer over all of the Salesforce capabilities and platform and acquisitions that we have made over the last 26 years, except we made it agentic. Agentic meaning all of those become tools, as well as able to take advantage of some of our recent investment into Data Cloud, all the acquisitions we have made, Tableau, Slack, and so on.

So at Dreamforce, about a month and a half ago, we announced the next version of Agentforce. This is the fourth version that we have called Agentforce 360. Who doesn't like 360? So at the heart of Agentforce 360, the next evolution of Agentforce, we are serving via this agentic platform both front-end customer-facing applications. This is the agent for service, sales, marketing, commerce, and so on, as well as the backend operations. And we announced very excitedly our brand new Agentforce agents for supply chain, agent for revenue management, agent for IT services, ITSM, and so on. And then in addition to providing automation via agents, we are providing support for many employees through agents that are now living natively in Slack or Slackbot, in case you haven't tried that, or agent for HR services. So that's what we mean when we talk about Agentforce.

And of course, behind the scenes, there are all kinds of technology. So we're going to look at all of these different layers of tech and see where the technology that AWS is announcing today is helping us. In particular, we're going to look at the layer that is called the trust layer. We use many models. As enterprises, they want to have the option to select any models that they want in a way that is trusted. So we are providing the wrapper to make sure that it works for all the agentic applications that you have in mind.

The use cases that came to our attention earlier this year after we read the famous Attention Is All You Need paper is, hey, the large frontier models that are out there, the GPT-5.1s, the Claude 4.5, those are really awesome. But they generally have long latency. So if you have latency-sensitive applications like voice applications, you probably want to use something that is a bit more efficient. Our criteria is very simple. We want high accuracy, but we want low latency, along with high explanatory power. And so the need drove us to look into creating our own fine-tuned small language model, that's what the industry usually calls it, so that we can get this balance of high accuracy, low latency, while providing low cost because we're able to host it and manage it ourselves on AWS.

So the particular model that we have been working on has now been released. It's called TaxEval. Internally, we call it TaxEval. So this is our in-house LLM as a judge model. There's a patent number behind that and so on. Currently, on a monthly basis, it serves roughly 130 million queries and it's expected to double and triple over the next few months. So the model is driving a number of applications: guardrail applications, instruction adherence, task resolution, citations, automatic inline generation of citations, as well as to provide response quality assessment in our testing center product, which is our observability and testing experimentation platform.

So the methodology that we went through is exactly like Shreyas was describing. There are a number of things that we were doing. This is before this amazing, easy-to-use product was available. This was quite a bit of work. The base model is a GPT-4o S20B, and it is being fine-tuned using a dataset that is both publicly available as well as synthetically generated SDG data. So the pipeline went through a combination. It's a mixed model training approach between Supervised Fine-Tuning with a reduced dataset, along with GRPO, which is this Reinforcement Fine-Tuning technique that Shreyas has talked about. And the result is TaxEval, literally within hours rather than previously would have taken weeks.

So here's the slide probably many of you are looking for, for those who are on the science side. On the very first row, the base model that we are comparing against is the GPT-4o 1120 version. Okay, so I want you to go all the way to the right, look into the F1 score. Think of it as an accuracy score, if you could, on two use cases. One is instruction adherence: how likely is the response to adhere to the instruction that you have given it. And GPT-4o F1 score is 88%. Completion is task completion score. So the GPT-4o frontier model last year was 83%. That's okay, not bad, but it costs a lot, for sure.

Now, the base model for the GPT-4o S20B, now remember, the GPT-4o is probably a one trillion parameter model. So a 20 billion parameter model, of course, is going to perform worse. And indeed, 64% for adherence, 79% for task completion. So standard technique number one that Shreyas and Shalendra introduced using Supervised Fine-Tuning was able to, using TaxEval, uplift it, doing pretty well, up to 93% F1 score. Now, if we apply this Reinforcement Fine-Tuning natively, the instruction adherence gets up to 95%, but for task completion, it's only 79%. So you're like, wait a second, what is going on?

So the secret sauce, if you will, turns out to be counterintuitive. If you bring in datasets that are beyond just instruction adherence, but that's two different tasks, instruction adherence and task completion, it actually improves both. The non-intuitive part is it also improved instruction adherence to now 97% and 95%, both of which are significantly better than the frontier model GPT-4o at a fraction of the cost, less than 10% or on AWS. That's pretty good.

So the next step, as we test this out, and this is now in production, of course we're going to standardize more and more of this. In fact, we're very much looking forward to beyond small task-specific models here. We're looking into our reasoning model. Perhaps there is now a better reason for us to move to something that is cheaper, faster, and perhaps even better.

So the one final example I'm going to give to you here is that reasoning is probably the final frontier when it comes to making agents able to adhere to the constraints that you want to give to it. As part of the Salesforce Agentforce 360 launch, we launched a brand new way of handling enterprise reasoning that we're calling the agent graph. The two-sentence summary is we externalize complex reasoning into a graph of specialized agents that are coordinated through the Agentforce orchestrator. The key trick here is that the state transition from one specialist agent to the other is controlled by a finite state machine, an FSM. For those who came from the CS world, this is about as deterministic as you're able to do it while retaining the natural sounding voice, if you will, of an LLM-based agent.

To do this, every one of the subspecialist agents here can make use of a reinforcement fine-tuning language model as a base, as its reasoner, and we'll be working through that over the coming year. Finally, just to let you take a look, we're looking at the trust layer here, but you could imagine over the next years, as many of you in the room will help revolutionize how enterprise agents will work, the impact of reinforcement fine-tuning could impact every single component on this slide. With that, I'd like to pass it back to Shalendra to conclude.

Summary and Amazon Bedrock's Continued Innovation in GenAI Platform Development

Thank you, Shreyas and Phil. So you just saw how our reinforcement fine-tuning on Bedrock works and how our customers like Salesforce are using it for their use cases. Today you will see reinforcement fine-tuning in Bedrock with Nova Lite 2 model, with support for additional models including open-weight models coming very soon. So in summary, three main reasons for you to consider and use RFT in Bedrock. Number one, ease of use. We have done all the end-to-end automation, so you don't have to worry about any infrastructure setup. You don't need any deep ML expertise.

Number two, better model performance, as you saw in Salesforce and in other experiments, with gains of up to 60 to 70% on average compared to base models with the use of RFT. And number three, visibility into training metrics so you understand what's going on. Then once the training is complete, once your RFT fine-tuned model is ready, you can deploy it with on-demand inference and only pay for tokens that you use.

Now before we wrap up, I also wanted to take a moment to talk to you about the innovation we have been doing on the Bedrock side overall. We continue to bring the latest and greatest foundation models on Bedrock, giving you enormous model choice, and we'll continue to invest in it more and more to serve you all better and better. Two, as you saw with RFT launch, we will continue to double down on model customization. Earlier this year, we launched the capability of model distillation, and we also launched on-demand inference for custom Nova and Llama models.

Number three, we have been investing heavily in Agentic AI, and we recently launched Agent Core. Hopefully you also saw additional announcements of Agent Core agent evaluation and policy features. And number four, we continue to launch new features in Bedrock Guardrails, and we recently launched the support for coding-related use cases as well in Guardrails. We will continue to make Bedrock the most innovative GenAI platform in the industry so you all can build the most powerful, useful, and efficient GenAI apps for your customers. Thank you.

; This article is entirely auto-generated using Amazon Bedrock.