Amazon DevOps Guru for the Serverless applications - Part 13 Anomaly detection on Aurora Serverless v2 with Data API (kind of)

Introduction

In my article which I published as a part of the and use

The application creates products stored in the Aurora Serverless v2 PostgreSQL database and retrieves them by id using Data API. The relevant Lambda function which we'll use to retrieve product by its id is GetProductByIdViaAuroraServerlessV2DataApi and its handler implementation is to perform the stress test like this

hey -z 15m -c 300 -H "X-API-Key: XXXa6XXXX" https://XXX.execute-api.eu-central-1.amazonaws.com/prod/productsWithDataApi/1   

In this example we invoke the API Gateway endpoint with 300 concurrent containers for 15 minutes. Behind the prod/productsWithoutDataApi endpoint Lambda function GetProductByIdViaAuroraServerlessV2WithoutDataApi will be invoked wich will retrieve the product by id 1 from the Aurora Serverless v2 PostgreSQL database.

We configured in our [SAM template]((. So, with the increased number of invocations, we expect to reach the maximal number of the database connections available and high database (CPU) load soon, so that database won't be able to respond to the new Lambda function requests to retrieve product by id (Lambda will then also run into). With that we will provoke the anomaly and would like to figure out whether DevOps Guru will be able to detect it. And it was able, kind of.... The following insight was generated:

Comparing to the aggregated anomalous metrics identified in case of using JDBC instead of Data API described in my article .

So, what's the difference? Let's explore both incidents that we reproduced on Aurora Serverless v2 PostgreSQL cluster with JDBC(Non Data API) and Data API :

In terms of ACU utilization/scaling they both look the same:

• CPU Utilization looks the same for JDBC(Non-Data API) and Data API cases. But DevOps Guru seems to not to consider this metric, as we didn't see it even for JDBC experiment


• DBLoad(CPU) which is very low for Data API usage. It seems that for Dat API there is some Load Balancer in front of the Aurora Serverless v2 database which monitors the connection usage and protects the database for being overloaded.


• DatabaseConnection metric is not shown (or shown as 0) for Data API usage. The reason for that is that we don't manage database connection for Data API, it's done on the other side for us. Of course they still play an important role we learned in
  
  
  

  Now we see at least additional Aurora Serverless v2 database connection sum anomalous metric, but DBLoad(CPU) metrics are still missing.

  
  
  

  Graphed anomalies look like this:

  
  
  

  

  
  
  

  Of course, the experiment wasn't clean, as I did 2 load tests after each other and partially in parallel : the first one connecting to the database directly without API Gateway usage and the second by using Data API. This confirmed my initial assumption that database connection sum metrics is a very important criteria to generate DevOps Guru insight for Aurora Serverless v2 (and for RDS in general) and it's not expose in general in case of using Data API.
  
  I already contacted Devops Guru team and shared with them my insights with the expectations that they will improve the service. Or first of all exposing database connection as a CloudWatch Metric will be fixed for using Aurora Serverless v2 with Data API.

  
  
  

  
  
  
  Conclusion
  

  
  
  

  In this article learned that DevOps Guru could successfully detect anomalies with Aurora (Serverless v2) PostgreSQL database in case of Lambda function with Java 21 managed runtime connected to it via Data API but could only showed the anomalous metrics related to the Lambda function being timed out as the database didn't respond. The main reason for that seems to be that database connection as a CloudWatch Metric isn't exposed (or always displayed as 0) in case of using Aurora Serverless v2 with Data API. Aurora Serverless v2 database metrics (database connection sum) was only showed during the second artificial experiment.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen