Building event-driven workflows with DynamoDB Streams

Event-based architectures allow us to break down complex, hard-to-read code into more manageable components. Using DynamoDB streams and lightweight Lambda functions, we can create loosely coupled resources that automatically respond to events, streamlining workflows and improving system clarity.

1. The scenario

Bob is developing a gaming application where users earn scores for completing tasks. Each task is categorized by difficulty, allowing players to earn different scores based on the challenge level.

The backend logic adds these scores to a for the entire application.

Scores are submitted to an function processes the payload and saves it to the database. Alternatively, if the payload doesn’t require validation or restructuring, the API Gateway can connect directly to DynamoDB - offering a faster, but not always viable solution.

In both cases, the new score must be added to the user's total score.

2. Approaches

Bob has two main options to tackle this challenge.

2.1. Synchronous solution

In this approach, Bob adds a Lambda integration to the API Gateway endpoint. The function calculates the new score based on the task type provided in the request. It may also need to fetch the relevant task-score mapping from the database. Then, the function retrieves the user's existing total score from the database, adds the new score, and saves the updated total score back to the table.

This pattern is commonly used. One benefit is its simplicity: a single function handles all necessary operations, keeping the architecture straightforward. All that’s required is to write and deploy the code to Lambda.

The main drawback of the asynchronous approach is the added architectural complexity. Bob will need to set up and manage additional resources, which requires more time, design consideration, and maintenance.

Bob eventually decided to go with the asynchronous solution for this application.

3. The chosen asynchronous solution

Let’s explore how to set up an asynchronous workflow for score processing.

3.1. DynamoDB Streams

DynamoDB is straightforward:

const table = new dynamodb.Table(scope, 'SOME_ID', {

  // ... other properties

 stream: dynamodb.StreamViewType.NEW_IMAGE,

});

With this configuration, the Lambda function consuming the stream receives the full item as it appears after modification.

3.2. Avoiding recursion

However, there’s a challenge: since we’re using a single-table design, the new score item is saved in the same table as the total score.

When a new score is written, it’s added to the stream, triggering the Lambda function to process the score and update the total score in the same table. This update could trigger the function again, potentially causing a loop or function error.

To avoid this, we configure the .

Here’s an example of the processor function code in TypeScript:

export async function handler(event: DynamoDBStreamEvent, context: Context) {

  // Assuming that the batch size is set to 1 in the event source mapping, we can access the only record in the batch using its index.

  // If the batch size is set to a number greater than 1, you should iterate over the records to process them all.

  const score = event.Records[0].dynamodb?.NewImage;



  const [, uniqueId] = score.PK.S?.split('#') || [];

  const [, otherId] = score.SK.S?.split('#') || [];

  const scoreValue = score.Score.N;

  // other necessary attributes here



  // processing logic here

}

One important note: even if using the to avoid specifying data type descriptors (S, N, BOOL, etc.) in code, the stream record will still contain them. Be sure to include these descriptors in the access path when retrieving attribute values.

4. Summary

Using DynamoDB Streams and Lambda functions, we can build loosely coupled, event-driven workflows. This approach requires additional resources to be created and managed, but it leads to cleaner, more maintainable code in each function.

To prevent loops in the architecture, we can set up filters in the event source mapping. These filters ensure that the function only triggers when the stream record meets specific conditions, streamlining the event processing flow.

5. References, further reading

- How to create a Lambda function

Creating tables and loading data for code examples in DynamoDB - How to create a and populate DynamoDB tables