Lambda Managed Instances with Terraform: Multi-Concurrency, High Memory, and Compute Options

Lambda has always been one request at a time per execution environment. Your function starts, processes a single invocation, and sits idle until the next one arrives. If you need to handle a thousand concurrent requests, Lambda spins up a thousand execution environments - each with its own memory, its own cold start, and its own per-GB-second bill.

Lambda Managed Instances changes that model. Announced at re:Invent 2025 and expanded with

The AWS Compute Continuum

Before diving into the implementation, it helps to understand where Lambda Managed Instances fits in the AWS compute landscape. The options form a continuum from fully managed to fully self-managed:

covers Fargate and ECS Express Mode. The article covers the observability patterns used in this project.

Architecture

Each process receives a request, calls Bedrock to embed the query text, then fans out across categories using ThreadPoolExecutor. The catalog data (loaded from DynamoDB at process init) stays in memory across all requests handled by that process.

Why LMI Instead of Standard Lambda

This workload is a poor fit for standard Lambda and a strong fit for LMI. Here's why:

In-memory catalog at scale. Each process loads the product catalog with embedding vectors into memory at initialization. A 100K product catalog with 384-dimensional vectors is roughly 150 MB per process. With 10 concurrent processes, that's 1.5 GB for catalog data alone. Standard Lambda's maximum is 10 GB total, and you pay per-GB-second for every millisecond of that memory. LMI gives you up to 32 GB with configurable memory-to-vCPU ratios, and you pay EC2 instance pricing regardless of how much memory your function uses.

Multi-concurrency amortizes catalog loading. On standard Lambda, 10 concurrent requests means 10 independent execution environments, each cold-starting and loading the catalog into its own memory, each paying per-GB-second. On LMI, those 10 requests run as 10 processes on one EC2 instance. The catalog loads once per process at init time and stays warm for all subsequent requests routed to that process. At sustained throughput, this eliminates the repeated cold-start penalty.

Sustained throughput economics. A product recommendation API serving a storefront has predictable, sustained traffic - hundreds of requests per second during business hours. Each request involves a Bedrock API call for query embedding (I/O), cosine similarity across categories (CPU), and structured logging (I/O). At 10M+ invocations per month, EC2 pricing with Savings Plans is 60-72% cheaper than standard Lambda's per-GB-second model.

Configurable memory-to-vCPU ratio. This workload is memory-heavy (large catalog) with moderate CPU needs (vector math on 384 dimensions). The 4:1 memory-to-vCPU ratio gives 4 GB of memory per vCPU - enough for the catalog plus Bedrock client overhead. Standard Lambda locks you into a fixed ratio where more memory always means proportionally more CPU and higher cost.

Why Not Fargate?

This project could run on ECS Fargate. The handler logic would move into a FastAPI app, the catalog would load at container startup, and an ALB would handle routing. It would work fine. But the infrastructure footprint would be significantly larger:

For teams already comfortable with Lambda, LMI is the path of least resistance to get EC2 pricing and multi-concurrency without learning container orchestration. You keep the programming model you know and gain the hardware flexibility you need. The reverse is also true: for teams already invested in ECS, Fargate may remain the more operationally familiar choice - the muscle memory, dashboards, deployment pipelines, and on-call runbooks are already in place.

Where Fargate or EKS would be the better choice: custom native dependencies that exceed Lambda layer limits (PyTorch, large ML models), persistent connections (WebSocket, gRPC), specialized instance types not supported by LMI, or workloads that need the Kubernetes ecosystem. I covered Fargate patterns in my projects. My Security and Cost Optimization pillars. All resources use official HashiCorp providers (hashicorp/aws and hashicorp/archive where applicable) - no community modules or third-party providers.

For a fully production-hardened deployment, you'd also want to address the Reliability, Performance Efficiency, and Operational Excellence pillars more explicitly. The pattern - Logger, Tracer, and Metrics decorators in the correct order:

import json

import math

import os

from concurrent.futures import ThreadPoolExecutor, as_completed

import boto3

from aws_lambda_powertools import Logger, Metrics, Tracer

from aws_lambda_powertools.metrics import MetricUnit

from aws_lambda_powertools.utilities.typing import LambdaContext



logger = Logger()

tracer = Tracer()

metrics = Metrics()



# Module-level init runs ONCE PER PROCESS.

# With 10 concurrent processes, this runs 10 times.

# Each process loads its own catalog copy and boto3 clients.

PROCESS_ID = os.getpid()

AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")

EMBEDDING_MODEL_ID = os.environ.get(

    "EMBEDDING_MODEL_ID", "amazon.nova-2-multimodal-embeddings-v1:0"

)

EMBEDDING_DIMENSION = int(os.environ.get("EMBEDDING_DIMENSION", "384"))



dynamodb = boto3.resource("dynamodb", region_name=AWS_REGION)

table = dynamodb.Table(os.environ["DYNAMODB_TABLE"])

bedrock_runtime = boto3.client("bedrock-runtime", region_name=AWS_REGION)

_catalog: dict[str, list[dict]] = {}





@tracer.capture_method

def _load_catalog() -> None:

    """Load product catalog once per process. Uses Query (least privilege)."""

    if _catalog:  # already loaded in this process

        return

    # ... query DynamoDB by category and populate _catalog ...





@logger.inject_lambda_context(log_event=True)

@tracer.capture_lambda_handler

@metrics.log_metrics(capture_cold_start_metric=True)

def lambda_handler(event: dict, context: LambdaContext) -> dict:

    logger.append_keys(process_id=PROCESS_ID)

    _load_catalog()  # no-op after first call in this process



    # Extract params from event body or direct invocation

    body = event.get("body")

    params = json.loads(body) if isinstance(body, str) else (body or event)

    top_k = int(params.get("top_k", 5))



    # Step 1: Embed the query text via Bedrock (I/O-bound)

    query_embedding = _embed_query(params["query"])



    # Step 2: Search categories in parallel (CPU-bound)

    results: dict = {}

    categories = params.get("categories", list(_catalog.keys()))

    with ThreadPoolExecutor(max_workers=4) as executor:

        futures = {

            executor.submit(_search_category, cat, query_embedding, top_k): cat

            for cat in categories

        }

        for future in as_completed(futures):

            results[futures[future]] = future.result()



    metrics.add_metric(name="SearchRequests", unit=MetricUnit.Count, value=1)

    return {"statusCode": 200, "body": json.dumps({"results": results})}

Bedrock Embedding - Configurable Model

The query text is embedded via Amazon Bedrock before similarity search. The model is configurable via the EMBEDDING_MODEL_ID environment variable - Nova Multimodal Embeddings by default, with Titan Text Embeddings V2 as an alternative:

@tracer.capture_method

def _embed_query_nova(text: str) -> list[float]:

    """Nova Multimodal Embeddings request format."""

    request_body = {

        "taskType": "SINGLE_EMBEDDING",

        "singleEmbeddingParams": {

            "embeddingPurpose": "TEXT_RETRIEVAL",

            "embeddingDimension": EMBEDDING_DIMENSION,

            "text": {"truncationMode": "END", "value": text},

        },

    }

    response = bedrock_runtime.invoke_model(

        body=json.dumps(request_body),

        modelId=EMBEDDING_MODEL_ID,

        accept="application/json",

        contentType="application/json",

    )

    response_body = json.loads(response["body"].read())

    return response_body["embeddings"][0]["embedding"]

Product embeddings are generated at seed time using GENERIC_INDEX purpose and stored in DynamoDB alongside the product data. Query embeddings use TEXT_RETRIEVAL purpose at runtime. Nova supports 4 dimension sizes (256, 384, 1024, 3072) - trading off accuracy against memory and compute cost. The demo uses 384 dimensions as a practical balance.

Cosine Similarity - The CPU Bottleneck

The vector similarity computation is the compute-intensive core after the Bedrock call returns. For production, use NumPy - it's 10-50x faster than a pure Python loop and releases the GIL during C-level operations, which makes the ThreadPoolExecutor pattern actually parallel:

import numpy as np



def _cosine_similarity(query: np.ndarray, catalog: np.ndarray) -> np.ndarray:

    """Production version: batch operation across all products in a category."""

    # query shape: (D,), catalog shape: (N, D)

    norms = np.linalg.norm(catalog, axis=1) * np.linalg.norm(query)

    return np.dot(catalog, query) / np.where(norms == 0, 1, norms)

The pure Python version is included in the demo as an educational fallback (no NumPy dependency, easier to read):

def _cosine_similarity_pure(vec_a: list[float], vec_b: list[float]) -> float:

    """Educational version: shows the math, no dependencies."""

    dot = sum(a * b for a, b in zip(vec_a, vec_b))

    norm_a = math.sqrt(sum(a * a for a in vec_a))

    norm_b = math.sqrt(sum(b * b for b in vec_b))

    if norm_a == 0 or norm_b == 0:

        return 0.0

    return dot / (norm_a * norm_b)

The handler code, the capacity provider, the Terraform - none of it would need to change to run on an instance type with hardware-accelerated vector operations. The capacity provider's instance type selection is the only variable.

Process Memory Multiplication

This is the most important thing to understand about Python LMI. Each process loads its own copy of the catalog:

10 concurrent processes x 200 MB catalog = 2 GB just for catalog data

The MemoryUtilization CloudWatch metric tracks total memory consumption across all processes. If you're loading large datasets and running high concurrency, you'll hit memory limits. Tune with:

• Reduce PerExecutionEnvironmentMaxConcurrency (fewer processes, less memory)


• Increase memory_size (more memory per environment)


• Use 8:1 memory_gib_per_vcpu ratio (more memory, fewer vCPUs)


• Use shared /tmp as a cross-process cache (load once, read from all processes)

Observability

LMI publishes its own CloudWatch metrics in the AWS/Lambda namespace at 5-minute granularity. The capacity-provider-level metrics describe overall instance utilization; the execution-environment-level metrics describe per-function resource consumption.

Capacity provider metrics (dimensions: CapacityProviderName, InstanceType):

• 
  CPUUtilization - CPU usage across all instances in the capacity provider


• 
  MemoryUtilization - Memory usage across all instances


• 
  vCPUAllocated / vCPUAvailable - Used vs available vCPU count


• 
  MemoryAllocated / MemoryAvailable - Used vs available memory

Execution environment metrics (dimensions: FunctionName, CapacityProviderName, Resource):

• 
  ExecutionEnvironmentConcurrency - Active concurrent requests per environment


• 
  ExecutionEnvironmentConcurrencyLimit - Configured maximum concurrency per environment


• 
  ExecutionEnvironmentCPUUtilization - CPU usage of this function's environments


• 
  ExecutionEnvironmentMemoryUtilization - Memory usage of this function's environments

Alarms to Set First

If you only set three alarms when adopting LMI, set these:

1. 
   Capacity provider CPU utilization - Alarm when sustained CPU exceeds your scaling target (e.g., > 80% for 10 minutes if your target is 50%). This indicates the capacity provider is failing to scale out fast enough or has hit max_vcpu_count.


2. 
   Execution environment concurrency vs limit - Alarm when ExecutionEnvironmentConcurrency reaches ExecutionEnvironmentConcurrencyLimit for sustained periods. This means processes are saturated and incoming requests are being throttled or queued.


3. 
   Execution environment memory utilization - Alarm when memory exceeds 80%. With Python's per-process memory multiplication, hitting memory limits causes new process spawns to fail (InitResourceExhausted) rather than gradual degradation. Catch this before it happens.

These three cover the LMI-specific failure modes that standard Lambda alarms (Errors, Throttles, Duration) won't catch.

Deployment

Prerequisites

• AWS CLI configured with a profile (export AWS_PROFILE=your-profile)


• Terraform >= 1.11


• Python 3.14+ with boto3 (for the seed script)


• Amazon Nova Multimodal Embeddings model enabled in your AWS account (Bedrock console, Model Access)

Deploy

# Clone the repo

git clone https://github.com/RDarrylR/lambda-managed-instances-similarity-engine.git

cd lambda-managed-instances-similarity-engine



# Configure

cp infrastructure/terraform.tfvars.example infrastructure/terraform.tfvars

# Edit terraform.tfvars with your values



# Deploy infrastructure

make init

make apply



# Seed the product catalog

make seed



# Invoke

make invoke

Cost Analysis

Lambda Managed Instances pricing is fundamentally different from standard Lambda. Understanding when each model wins is the key decision.

Standard Lambda pricing (arm64/Graviton):

• $0.20 per million requests


• $0.0000133334 per GB-second (arm64)


• No minimum charge, no idle cost

Lambda Managed Instances pricing:

• $0.20 per million requests (same)


• EC2 on-demand instance pricing (varies by type)


• 15% management fee on the EC2 on-demand price


• No per-invocation duration charge

The critical difference: standard Lambda charges per GB-second of execution. LMI charges for EC2 time regardless of how many requests you serve. At low volume, you're paying for idle EC2 capacity. At high volume, that fixed EC2 cost is amortized across millions of requests.

Break-Even: Standard Lambda vs LMI

Consider this workload: 4 GB memory, 200ms average duration, sustained traffic.

Standard Lambda cost per request (arm64):

• Compute: 4 GB x 0.2s = 0.8 GB-seconds x $0.0000133334 = $0.00001067


• Request: $0.0000002


• Total: ~$0.0000109 per request

LMI on a c7g.medium (1 vCPU, 2 GB, ~$0.034/hr on-demand):

• EC2 + 15% fee: $0.034 x 1.15 = $0.0391/hr


• With 10 concurrent processes and 200ms per request, each process handles ~5 req/sec


• Instance throughput: ~50 req/sec = ~180,000 req/hr


• Cost per request: $0.0391 / 180,000 = ~$0.000000217

At this throughput, LMI is roughly 50x cheaper per request than standard Lambda. But the EC2 cost runs 24/7 whether you have traffic or not.

Monthly Cost Comparison

A single c7g.medium tops out around ~130M requests/month at 50 req/sec sustained. Beyond that, instance count scales roughly linearly with load - 500M req/month requires approximately 4 instances. The LMI columns reflect the actual instance count needed at each volume.

The break-even is around 2.5M requests/month at this memory and duration profile. Below that, standard Lambda wins because you pay nothing when idle. Above that, LMI wins and the advantage grows with volume.

Commitment Discounts Change the Math

LMI supports EC2 Savings Plans and Reserved Instances. Standard Lambda supports Compute Savings Plans (up to 17% discount on duration). The discount gap is significant:

For predictable production workloads with steady traffic, a 3-year commitment on LMI can reduce costs by 60% on the EC2 portion. Standard Lambda's maximum discount is 17%. This difference widens the gap at scale.

Hidden Costs

Don't forget the supporting infrastructure that LMI requires and standard Lambda doesn't:

• 
  NAT Gateway: ~$32/month + $0.045/GB data transfer (required for VPC telemetry)


• 
  VPC endpoints (if used instead of NAT): ~$7.20/month per endpoint per AZ


• 
  DynamoDB: On-demand reads for catalog loading (minimal for small catalogs, significant at scale)


• 
  Bedrock: Nova Multimodal Embeddings per-token pricing for each query embedding


• 
  CloudWatch: Log storage and metric costs increase with concurrency

For low-volume workloads, these fixed costs can exceed the compute savings. Factor them into your total cost of ownership.

When Each Pricing Model Wins

Standard Lambda wins when:

• Traffic is bursty or unpredictable (you pay nothing at zero traffic)


• Monthly volume is below the break-even threshold (~2-3M requests for this workload profile)


• You can't commit to 1-year or 3-year terms


• You don't need VPC connectivity (avoids NAT Gateway cost)

LMI wins when:

• Traffic is sustained and predictable (the EC2 cost is fully amortized)


• Monthly volume exceeds 5-10M requests


• You can commit to Savings Plans or Reserved Instances


• You need more than 10 GB memory or specific instance types


• You're already paying for VPC infrastructure

For this demo, expect to pay for:

• NAT Gateway (~$0.045/hour + data transfer)


• EC2 instances (varies by type, auto-selected by Lambda)


• DynamoDB on-demand reads (minimal for this catalog size)


• Bedrock embedding calls (per-token pricing for each query)

CLEANUP (IMPORTANT!!)

This infrastructure costs real money while running - approximately $2-4/day even with zero traffic (NAT Gateway + EC2 managed instances). Don't forget about it.

Make sure to destroy all resources when you're done:

make destroy

If the capacity provider fails to delete (it can take a few minutes to drain instances), wait and retry. Verify in the AWS console that no EC2 instances tagged with your project name are still running.

Networking: Three Supported Patterns

LMI requires VPC connectivity - the function execution environments need outbound network access for telemetry transmission and any AWS service calls. AWS documents three supported connectivity patterns:

1. 
   Public subnets with an internet gateway - simplest, suitable for dev/test only


2. 
   Private subnets with NAT Gateway - the pattern this demo uses


3. 
   Private subnets with VPC endpoints - the most AWS-aligned production pattern

NAT Gateway (used in this demo)

• Simple to set up - one resource, all outbound traffic routes through it


• ~$32/month base + $0.045/GB data transfer


• Traffic leaves your VPC, crosses the public internet (encrypted), then re-enters AWS


• Single point of failure unless you deploy one per AZ (~$64/month for 2-AZ HA)

VPC Endpoints (recommended for production)

For production, the most AWS-aligned pattern is one VPC endpoint per service per AZ. Traffic stays entirely on the AWS network and never touches the public internet. The endpoint set must cover every service the function calls - if you forget one, the function fails silently or hangs. For this workload, that means:

Critical security group detail: Interface endpoints have their own security groups. They must allow inbound HTTPS (port 443) from the function's security group. The function security group must allow outbound HTTPS to the endpoint security groups. If you skip this, DNS resolves but the connection is silently blocked.

Endpoints should be deployed in each AZ used by the capacity provider to avoid cross-AZ latency and data transfer costs. If your capacity provider has subnets in us-east-1a and us-east-1b, every interface endpoint also needs ENIs in both AZs. This is the same Cross-AZ Tax pattern from my - minimize the attack surface.

The Terraform for VPC endpoints is straightforward but verbose. I left it out of this demo to keep the focus on LMI itself. A follow-up project could add a networking_mode variable that switches between NAT Gateway and VPC endpoints.

A few things to watch for:

• 
  VPC connectivity isn't optional. Lambda Managed Instances requires a VPC. Without outbound connectivity (NAT Gateway or VPC endpoints), your function executes but logs and traces are silently lost. You'll debug a working function with no visible output. This is documented but easy to miss.


• 
  Scaling is asynchronous. LMI scales based on CPU utilization and execution-environment saturation, not per-invocation demand. Unlike standard Lambda, scaling isn't triggered by incoming requests - it's driven by resource consumption inside existing execution environments. Because scaling reacts to resource pressure instead of incoming traffic, inefficient code or high memory usage can delay scaling and increase throttling risk. The Scaler component decides when to add or remove instances, and instance launches aren't instant. Lambda maintains headroom so traffic can roughly double within minutes without immediate throttling, but if your traffic more than doubles within 5 minutes, you may see 429 throttles while capacity catches up. This is fundamentally different from standard Lambda's near-instant scaling. Plan for it with the target CPU utilization setting - lower values maintain more headroom.


• 
  Process memory multiplies. With Python, each concurrency slot is a separate process. Because Python uses process-based concurrency, memory usage scales linearly with concurrency - each worker process consumes its own memory. With Python, concurrency isn't "free" - each additional request increases memory consumption linearly. If your function uses 500 MB of memory and you set concurrency to 16, that's 8 GB of memory consumed per execution environment. Monitor the MemoryUtilization metric and tune accordingly.


• 
  publish = true is required. LMI runs on published function versions, not $LATEST. If you forget this, Terraform applies successfully but the function doesn't run on managed instances. Every code change needs a new published version.


• 
  Capacity providers are security boundaries, not isolation boundaries. Functions sharing a capacity provider run in containers on the same EC2 instances. This isn't Firecracker isolation. Separate untrusted workloads into separate capacity providers.


• 
  Powertools minimum version matters. Lambda Managed Instances requires Powertools for AWS Lambda (Python) version 3.23.0 or later. Pin the layer version in Terraform rather than using latest.


• 
  LMI doesn't scale to zero. Unlike standard Lambda where you pay nothing at zero traffic, LMI keeps a baseline of warm EC2 instances running for high availability. AWS launches a baseline of three managed instances for availability across AZs when you publish a function version with a capacity provider. In my testing with 2 AZs configured, 2 instances remained active overnight with zero traffic, but the documented baseline is three. There's no minimum instance setting, no Karpenter-style consolidation, and no way to force scale-to-zero short of deleting the function version or capacity provider. This is a meaningful cost difference for dev/test environments where you might leave infrastructure running between sessions. Run make destroy when you're not actively using the infrastructure, or design your dev environments to use standard Lambda where idle cost is zero.


• 
  Quotas to plan around. LMI has its own .
  
  
  
  
  
  

  
  
  
  Resources
  

  
  
  
  
  
  
  
  
  
  • 
    
  
  
  • 
    - ECS Fargate and Express Mode comparison
  
  
  • 
    , , , . Check out more of my projects at community.
    Vollständiger Original-Bericht

    Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

    ↗ Original-Artikel auf dev.to lesen