I would like to introduce middy-store, a new library I built over the last couple of months. I've been pondering this idea for a while, going back to this of 6MB for synchronous invocations and 256KB for asynchronous invocations. AWS Step Functions allows for a exception.

This becomes even more cumbersome when you only want to save part of the payload to S3 and leave the rest as is. For example, when working with Step Functions, the payload could contain control flow data for states like Choice or Map, which has to be accessed directly. This means the first Lambda saves a partial payload to S3, and the next Lambda has to load the partial payload from S3 and merge it with the rest of the payload. This requires ensuring that the types are consistent across multiple functions, which is, of course, very error-prone.

How it works

middy-store is a middleware for Middy. It's attached to a Lambda function and is called twice during a Lambda invocation: before and after the Lambda handler() runs. It receives the input before the handler runs and receives the output from the handler after it has finished.

middy-store accepts an Array<StoreInterface> in the options to provide one or more Stores. When middy-store runs before the handler and finds a reference in the input, it will iterate over the Stores and call canLoad() with the reference for each Store. The first Store that returns true will be used to load the payload with load().

On the other hand, when middy-store runs after the handler and the output is larger than the maximum allowed size, it will iterate over the Stores and call canStore() for each Store. The first Store that returns true will be used to store the payload with store().

Therefore, it is important to note that the order of the Stores in the array is important.

References

When a payload is stored in a Store, middy-store will return a reference to the stored payload. The reference is a unique identifier to find the stored payload in the Store. The value of the identifier depends on the Store and its configuration. For example, the Amazon S3 Store will use an S3 URI by default. However, it can also be configured to return other formats like an ARN arn:aws:s3:::<bucket>/<key>, an HTTP endpoint https://<bucket>.s3.us-west-1.amazonaws.com/<key>, or a structured object with the bucket and key.

The output from the handler after middy-store will contain the reference to the stored payload:

/* Output with reference */

{

  "@middy-store": "s3://bucket/key"

}

middy-store embeds the reference from the Store in the output as an object with a key "@middy-store". This allows middy-store to quickly find all references when the next Lambda function is called and load the payloads from the Store before the handler runs. In case you are wondering, middy-store recursively iterates through the input object and searches for the "@middy-store" key. That means the input can contain multiple references, even from different Stores, and middy-store will find and load them.

Selecting a Payload

By default, middy-store will store the entire output of the handler as a payload in the Store. However, you can also select only a part of the output to be stored. This is useful for workflows like AWS Step Functions, where you might need some of the data for control flow, e.g., a Choice state.

middy-store accepts a selector in its storingOptions config. The selector is a string path to the relevant value in the output that should be stored.

Here's an example:

const output = {

  a: {

    b: ['foo', 'bar', 'baz'],

  },

};



export const handler = middy()

  .use(

    middyStore({

      stores: [new S3Store({ /* S3 options */ })],

      storingOptions: {

        selector: '',          /* select the entire output as payload */

        // selector: 'a';      /* selects the payload at the path 'a' */

        // selector: 'a.b';    /* selects the payload at the path 'a.b' */

        // selector: 'a.b[0]'; /* selects the payload at the path 'a.b[0]' */

        // selector: 'a.b[*]'; /* selects the payloads at the paths 'a.b[0]', 'a.b[1]', 'a.b[2]', etc. */

      }

    })

  )

  .handler(async () => output);



await handler({});

The default selector is an empty string (or undefined), which selects the entire output as a payload. In this case, middy-store will return an object with only one property, which is the reference to the stored payload.

/* selector: '' */

{

  "@middy-store": "s3://bucket/key"

}

The selectors a, a.b, or a.b[0] select the value at the path and store only this part in the Store. The reference to the stored payload will be inserted at the path in the output, thereby replacing the original value.

/* selector: 'a' */

{

  a: {

    "@middy-store": "s3://bucket/key"

  }

}

/* selector: 'a.b' */

{

  a: {

    b: {

      "@middy-store": "s3://bucket/key"

    }

  }

}

/* selector: 'a.b[0]' */

{

  a: {

    b: [

      { "@middy-store": "s3://bucket/key" }, 

      'bar', 

      'baz'

    ]

  }

}

A selector ending with [*] like a.b[*] acts like an iterator. It will select the array at a.b and store each element in the array in the Store separately. Each element will be replaced with the reference to the stored payload.

/* selector: 'a.b[*]' */

{

  a: {

    b: [

      { "@middy-store": "s3://bucket/key" }, 

      { "@middy-store": "s3://bucket/key" }, 

      { "@middy-store": "s3://bucket/key" }

    ]

  }

}

Size Limit

middy-store will calculate the size of the entire output returned from the handler. The size is calculated by stringifying the output, if it's not already a string, and calculating the UTF-8 encoded size of the string in bytes. It will then compare this size to the configured minSize in the storingOptions config. If the output size is equal to or greater than the minSize, it will store the output or a part of it in the Store.

export const handler = middy()

  .use(

    middyStore({

      stores: [new S3Store({ /* S3 options */ })],

      storingOptions: {

        minSize: Sizes.STEP_FUNCTIONS,  /* 256KB */

        // minSize: Sizes.LAMBDA_SYNC,  /* 6MB */

        // minSize: Sizes.LAMBDA_ASYNC, /* 256KB */

        // minSize: 1024 * 1024,        /* 1MB */

        // minSize: Sizes.ZERO,         /* 0 */

        // minSize: Sizes.INFINITY,     /* Infinity */

        // minSize: Sizes.kb(512),      /* 512KB */

        // minSize: Sizes.mb(1),        /* 1MB */

      }

    })

  )

  .handler(async () => output);



await handler({});

middy-store provides a Sizes helper with some predefined limits for Lambda and Step Functions. If minSize is not specified, it will use Sizes.STEP_FUNCTIONS with 256KB as the default minimum size. The Sizes.ZERO (equal to the number 0) means that middy-store will always store the payload in a Store, ignoring the actual output size. On the other hand, Sizes.INFINITY (equal to Math.POSITIVE_INFINITY) means that it will never store the payload in a Store.

Stores

Currently, there is only one Store implementation for Amazon S3, but I'm planning to implement a Store backed by DynamoDB and DAX. DynamoDB, with its Time-To-Live (TTL) feature, provides a great option for short-term payloads that only need to exist during the execution of a workflow like Step Functions.

Amazon S3

The middy-store-s3 package provides a store implementation for Amazon S3. It uses the official @aws-sdk/client-s3 package to interact with S3.

import { middyStore } from 'middy-store';

import { S3Store } from 'middy-store-s3';



const handler = middy()

  .use(

    middyStore({

      stores: [

        new S3Store({

          config: { region: "us-east-1" },

          bucket: "bucket",

          key: () => randomUUID(),

          format: "arn",

        }),

      ],

    }),

  )

  .handler(async (input) => {

    return { /* ... */ };

  });

The S3Store only requires a bucket where the payloads are being stored. The key is optional and defaults to randomUUID(). The format configures the style of the reference that is returned after a payload is stored. The supported formats include arn, object, or one of the URL formats from the and is optional. If not set, the S3 client will resolve the config (credentials, region, etc.) from the environment or file system.

Custom Store

A new Store can be implemented as a class or a plain object, as long as it provides the required functions from the StoreInterface interface.

Here's an example of a Store to store and load payloads as base64 encoded and should be able to run it locally.

Contributions and Feedback

I've been tinkering with the API design for a while, and it's definitely not stable yet. I would love to get feedback on the current state as well as suggestions for changes or improvements. If you are eager to contribute to this project, please go ahead and submit feature requests or pull requests.