Improving the performance of a node http server when inserting large amounts of data

This is a very specific case, and it helps reduce resource consumption of the microservice container and the database, and allows data insertion much faster. Of course, for those who have already been through this and have a lot of experience, it seems obvious, but of course it will not be obvious to all software development teams.

Let's go, I'm going to start from two assumptions: 1. Unlike Elixir, which seems like a great language to handle thousands of simultaneous HTTP requests or websockets, as long as there are CPU threads to process, NodeJs is single-threaded, and HTTP requests ARE EXPENSIVE for NodeJs to manage .

Tests

Two tests will be performed with the same endpoint, since it has already been configured to receive an array.

1. stress-test-one: There will be a maximum of 5000 VUs each passing an array of a single item;


2. stress-test-batch: There will be a maximum of 100 VUs each passing an array of a thousand items.

The duration is 10 seconds.

Execution

stress-test-one

The test went well, no HTTP requests gave an error, inserting a total of 20081 items, 1481 items per second, which is the same number of HTTP requests.

Duration.............: avg=2.02s    min=43.57ms  max=9.14s  p(95)=6.56s   

http_reqs............: 20081   1481.148722/s

inserted_itens.......: 20081   1481.148722/s  

successful_requests..: 100.00% 20081 out of 20081

However, one interesting thing is that this took an average of 2 seconds, with a maximum of 9 seconds, with a 95% percentile of 6.56 seconds.

Another thing was that the CPU usage of the node container was quite high:

Database behavior

Another interesting thing is the relationship between transactions and block/IO. While in the test sending one item per request the number of transactions was very high, with a low number of block/IO, sending data in batches the number of transactions was much lower, while the block/IO was higher.

I separated the first test from the second test with a black line:

Running the numbers

All code used in the tests is available in the repository: test-performance-inserting-strategies-nodejs, containing README files teaching how to upload docker containers and run load tests.