Analyzing and Troubleshooting Bottlenecks in FastAPI: Optimizing Auth Flow, Cryptography, and Data Serialization

This article is intended for those who, like me, are interested in profiling and performance optimization.

Key Takeaways (TL;DR):

• Offload CPU-bound crypto: Move synchronous operations like Argon2id into a ThreadPoolExecutor to stop them from blocking the asyncio event loop.


• Choose the right algorithm: Ed25519 is much faster for signing tokens than RSA-2048, but slower for verification. This trade-off can be mitigated by caching validated tokens.


• Use msgspec for serialization: Native msgspec serialization can be up to 5x faster than FastAPI's default with Pydantic, drastically reducing CPU load on data-heavy responses.

→ Jump straight to the Bottleneck Analysis

Preface

To begin with, I would like to touch upon the subject of development culture in high-level languages.

Python is built for maximum productivity: it hides everything "superfluous" from the developer — memory management, register operations, threads, and system calls. This often creates a false impression that the code runs "by itself," regardless of the environment.

Python's popularity is due to its low barrier to entry. It is often called simple, but that does not mean the language is easy to master: behind the external simplicity lies the colossally complex work of the interpreter. The language can be visualized as an onion, where layer by layer you delve into its inner workings. Initially, it was perceived as a simple scripting tool, but modern Python and the one from 15 years ago are different languages. Not in terms of syntax, but in terms of usage methodology. The shift from synchronous "scripting" to asynchronous services with deep static typing has radically changed the requirements for developers. If previously the "barrier to entry" was defined by basic syntax knowledge, today it includes an understanding of concurrency, system abstractions, and static analysis tools. The barrier to entry has significantly increased.

Python is a tool that allows for the creation of complex systems, and that is precisely what makes it treacherous for those who do not look deeper.

When we write in high-level languages, we are in a "cozy bubble" of abstractions. Profiling and benchmarking are tools that pierce this bubble.

Here is what they provide, at the very least, in this context:

• Awareness of the cost of abstractions
  
  
  We often do not pay attention to the fact that even behind a simple list.append() operation or a function call in Python lies an entire stack: memory allocation, type checking, GIL operation, and system calls. Through benchmarking, we learn to measure not the "beauty of the code," but the real cost of CPU time and memory bytes. We begin to see how non-obvious things (for example, the method of data serialization or unnecessary allocations in a loop) can slow down the system.



• 
  

  Development of systems thinking
  
  
  Most developers perceive code as an autonomous entity. Profiling forces one to see the "submerged part":

  
  
  
  
  
  • 
    OS Interaction: I/O profiling often shows that latency arises not because of the Python code, but because of how Linux manages context switching or how data is buffered.
  
  
  • 
    Database Connection: We begin to understand that "fast" Python code is just a transport for queries, and the "bottleneck" is often the execution plan of an SQL query or the absence of indexes.
  
  
  • 
    Network Stack: Load testing shows system behavior under the pressure of TCP connections — this is critical for operation, even though it lies outside the logic of the application itself.
  
  
  
  



• Correct application of optimizations
  
  
  Without using profiling tools, we can only guess which part of the codebase we think is slow and whether it is truly the "bottleneck." A profiler (e.g., py-spy) provides objective data (with the correct approach to profiling). This eliminates guesswork and allows for optimizations exactly where they are truly needed.

The most important thing benchmarks provide is curiosity. When you see that your code is running slowly, you go digging deeper: you start studying the layout of data structures in memory, looking into system calls (strace), and hardware architecture (CPU cache, memory management).

Let us touch slightly on solving the problem of slow systems through faster hardware.

When we encounter such problems, the obvious solution becomes purchasing more powerful equipment. In cloud computing, switching to a computer with more cores, disk space, or adding RAM can be done in minutes or seconds. Given that developer time is expensive, switching to more powerful equipment is often viewed as the simplest and fastest solution to the problem. However, in the long run, you risk ending up with a system that is slow and extremely expensive. Also, it should be taken into account that a performance problem cannot always be solved only by more powerful equipment; it all depends on where the bottleneck is located. For example, if you just need more RAM or your application's work can be executed in parallel, then switching to powerful equipment might improve the situation.

Using a hardware-first approach can entail long-term costs exceeding the price of the equipment itself. These include:

• 
  Culture of inefficiency. When developers have access to unlimited resources, they lose the motivation to write resource-efficient code.


• 
  Horizontal scaling. It is one thing if the code requires data processing a couple of times — spending an extra $5-10 is not a problem. But if this same task is executed, for example, 1000 times a month, a multiplicative effect occurs and expenses can reach significant sums.


• 
  Vertical scaling. When further scaling on a single machine becomes a problem, it is required to move to a distributed system, which often entails significant changes to the codebase and/or increased debugging complexity. Thus, reaching architectural turning points occurs at earlier stages.

From a business perspective, we face an inevitable trade-off: you can spend money on equipment or spend developer time on writing more efficient code. Both cost options are evaluated, and the one that is lower in the current situation is chosen. However, the problem can be looked at from another angle: all other things being equal, a more efficient program is better than a less efficient one. Faster hardware may be a suitable solution in many cases, but it is also worth considering how to make the application more efficient by default.

If we proceed only from the assumption that slow or inefficient software is inevitable and unavoidable, then we do not even think about how to improve the program's performance. The ability to write more resource-efficient code is not a constant, but a skill that can be developed. By applying profiling and developing skills, you will spend time programming and creating systems that are initially faster and consume fewer resources.

The reader might have a number of questions:

• is optimization at early stages premature?


• does writing efficient code require more effort and is it harder to maintain?

We all know Knuth's famous saying: "premature optimization is the root of all evil." Let us rephrase: "doing something at the wrong time is not the best option." The full quote is: "We should forget about small efficiencies, say about 97% of the time; premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." Premature optimization here is understood as the attempt to improve code where the impact on overall performance is negligible, while sacrificing implementation cleanliness. If you dig deeper, developing fast code from the very beginning can bring huge benefits.

I will give a simple example: choosing msgspec and msgspec.Struct models for serialization at the API output layer instead of standard serialization in a FastAPI application from the very beginning. This provides a significant performance boost under high load. We immediately lay the foundation for a high-throughput system capable of "digesting" a large number of requests (the article will break down the efficiency of native msgspec.json serialization vs the standard FastAPI framework solution). On the other hand, if you have a small application and/or a large volume of data is not expected, then such optimization is likely premature until you truly hit a limit.

Let us also consider a small example of how architectural performance influences the development cycle using integration testing as an example.

Suppose the application architecture (for example, due to blocking I/O, lack of indexing, or inefficient serialization patterns) makes running tests "heavy" — running the full test suite takes, for instance, 10 minutes. This creates so-called "infrastructure friction": the developer is forced to accumulate a "batch" of changes before each run to avoid wasting time waiting. In such a model, the cost of an error increases critically. If a bug pops up during the run, finding it in a "batch" of changes is much harder than in an isolated change. Conversely, if tests run quickly enough, the developer moves to a "short cycle." This forms a habit of taking small, safe steps: made a change — ran tests — got a result. This does not just speed up the work; it changes the approach to design, allowing more experiments to be made with less risk of breaking the system.

Writing faster code requires more time and is harder to maintain. Let us compare two examples:

import json



data = json.loads(request_body)



# Next, we need to manually check the keys, types, etc.

user_id = int(data.get("user_id"))

# ... validation code

import msgspec



# We define the structure once

class User(msgspec.Struct):

    user_id: int



user = msgspec.json.decode(request_body, type=User)

Writing a model with msgspec.Struct is comparable in the amount of code to manual dictionary checking. We described the data structure, obtained automatic validation, static typing, and a performance boost. The cost of maintaining such code is even lower because it becomes more transparent for the whole team: the data contract is described explicitly, not hidden in the form of string keys throughout the code.

The reader might object that these are just examples specially chosen to illustrate my point of view. And that is true. One can find many examples where writing faster code requires more effort, leads to less maintainable code, or both. Nevertheless, obviously, there are situations where the fast version of the code is just as easy to maintain as the slow one, and just as simple to write.

If we are not accustomed to paying attention to speed, there is a high probability that the code could have been written faster without additional effort. The problems we are solving are likely not unique. This means that someone has likely already written the corresponding library, tool, or documented an interaction pattern that allows for writing faster code.

This article will outline my personal experience within the framework of a training project. Also, I draw the reader's attention to the fact that this is not an attempt to imitate a production-level standard.

I intentionally chose the authentication/authorization module for several reasons:

• heavy mathematics is present (synchronous password hashing for users, token issuance using the RSA-2048 algorithm);


• it fits perfectly into the load testing methodology (hypothesis based on theoretical knowledge — measurements — confirmation/refutation of the hypothesis — optimization — re-measurements).

The test bench (AMD FX-8320/HDD) was chosen not only due to its accessibility but also based on the fact that "weak" hardware makes it easier to highlight problem areas that would have been nullified on more powerful and modern hardware.

All tests were conducted on a single host machine (one node, Loopback/Synthetic testing)

Configuration:

• 
  Server: Uvicorn 0.40.0
  
  
  
  
  • 
    Flags: --loop=uvloop --http=httptools --backlog=2048
    
  
  
  
  


• 
  OS Tuning: ulimit -n 65535, somaxconn=2048, overcommit_memory=1
  

Environment: ).

The test results will include Max Latency, P99, and StdDeviation values. However, when analyzing the results, we will primarily focus on **Mean Latency* and P90 for the following reasons:*

• 
  **On lightweight endpoints, system jitter manifests in a classic way: Max Latency and P99 disproportionately spike upwards relative to the smooth P90 plateau, artificially inflating StdDeviation.
  


• 
  **On heavy (CPU-bound) endpoints, the overhead from the OS scheduler is physically smoothed out against the background of the long execution of the business logic itself. In this case, the percentile distribution appears smooth, but Max Latency and StdDeviation still remain noisy due to hardware error.
  

Bottleneck Analysis

1. Hypothesis for /api/v1/access/signup (Registration)

• 
  Premise: Using synchronous password hashing at the ORM level (advanced-alchemy | pwdlib.hashers.argon2.Argon2Hasher) requires significant CPU computational power, and the subsequent data writing involves disk I/O operations.


• 
  Hypothesis Formulation: Due to the cooperative nature of asynchrony in Python, performing heavy synchronous hash calculations (Argon2id) in the main thread under load from wrk2 will completely block the Event Loop and the application server worker for the duration of the mathematical operation. This will stop the processing of incoming network events, causing other clients' requests to queue up at the system socket level. The bottleneck will not be the disk subsystem, but the monopolization of the single Event Loop thread by synchronous code (lack of offloading the task to run_in_executor).


• 
  Objective: Determine whether async offloading of CPU-bound tasks (delegating computations to a ThreadPoolExecutor) is required for user password hashing, or if the current synchronous execution is optimal.

2. Hypothesis for /api/v1/access/signin (Login)

• 
  Premise: User authentication involves synchronous password verification (see /api/v1/access/signup above), as well as the generation of two tokens (access/refresh) using RSA-2048 asymmetric cryptography, which requires significant CPU computational power. Searching for the user in the database (I/O) has low latency due to the use of an index (the email column).


• 
  Hypothesis Formulation: See point one. On this endpoint, the situation will worsen: the worker will sequentially hang first on password verification, and then be blocked twice by RSA-2048 asymmetric encryption when generating signatures for the access and refresh tokens. The total time a single request monopolizes the thread will increase several times over.


• 
  Objective: Verify the hypothesis that the sequential execution of CPU-bound operations (password verification + RSA signatures) is a critical bottleneck, and evaluate the feasibility of switching to signature algorithms with lower computational costs (Ed25519).

3. Hypothesis for /api/v1/access/me (Get User Profile)

• 
  Premise: The endpoint relies on the get_current_active_user dependency. On each request, the JWT token is first decoded (RSA-2048 asymmetric algorithm via decode_jwt). After successful validation, the cache is checked (decorator @cache from fastapi_cache.decorator). On a cache hit, the bytes are retrieved and deserialized (calling msgspec.msgpack.decode + UserAuth.model_validate). On a cache miss, a database query is executed, followed by converting the ORM object to UserAuth and saving it to the cache (serialized using msgspec.msgpack.encode + jsonable_encoder).


• 
  Hypothesis Formulation: With a cache, we reduce the load on the database but transfer part of the load to the CPU (deserialization). However, as the load on the endpoint increases, the cryptographic decoding and verification of the JWT (RSA-2048) will place a noticeable load on the CPU and reduce overall throughput. Deserialization will not have a significant impact.


• 
  Objective: Evaluate the computational costs of decoding and cryptographically verifying access tokens when using the RSA-2048 algorithm.

Note: Authentication/Authorization chain execution flow

[HTTP Request]

     │

     ▼

[Dependency Depends(access_token)]

     │

     ▼

[1. Decode token: get_payload_from_token()]

     │

     ├── ❌ decode_jwt() using PyJWT - CPU-bound (RSA-2048 modular arithmetic)

     └── Structure and exception check

     │

     ▼

[2. Get user: Authenticate.get_current_user()]

     │

     └── Pass token_payload["sub"] to _get_user_from_payload() method

     │

     ▼

[3. Check cache / Get data]

     │

     ├── ✔️ Cache hit: Deserialization (MsgPackCoderUserAuth) — CPU-bound

     │    └─ (msgspec.msgpack.decode + UserAuth.model_validate)

     │

     └── ❌ Cache miss: I/O-bound

          ├── DB query (users_service.get)

          └── Convert to UserAuth schema (users_service.to_schema) and write to cache (msgspec.msgpack.encode + jsonable_encoder)

     │

     ▼

[4. Additional checks (depending on the endpoint)]

     │

     ├── get_current_active_user — check is_active

     └── superuser_required / trainer_required — check is_superuser or role_slug

4. Hypothesis for MsgSpecJSONResponse (msgspec.json) (Serialization)

• 
  Premise: The project implements a custom MsgSpecJSONResponse, but for versatility and compatibility with Pydantic models, we are forced to use jsonable_encoder as an intermediate step before serialization.


• 
  Hypothesis Formulation: On high-load endpoints or when serializing large volumes of data, the overhead of jsonable_encoder will cause noticeable latency and increase the CPU load. It is expected that the overhead of jsonable_encoder will be partially compensated by the msgspec library.

Note 1: How jsonable_encoder works: jsonable_encoder traverses all objects, including lists and nested dictionaries, checks types, and converts complex structures (e.g., datetime, UUID, Pydantic models) into standard Python types. New intermediate objects are created for each such operation. This leads to excessive memory allocation and additional load on the garbage collector.

Note 2: Before the release of Pydantic v2, using jsonable_encoder + orjson did provide a performance boost because, despite the overhead of jsonable_encoder, the final byte assembly was faster than the standard mechanisms of the FastAPI framework.

Objective:

1. Evaluate the efficiency of serialization via MsgSpecJSONResponse compared to the standard "out-of-the-box" serialization of Pydantic models by the FastAPI framework.


2. Determine if it makes sense to introduce separate msgspec.Struct "output" schemas into the project just for serialization to achieve maximum performance.

Round 1: The First Wave of Optimizations

Note 1: Profiling with py-spy, baseline tests, and tests after the first optimization were carried out with the default Argon2id settings.

• 
  Note 2: The first optimization included:
  
  
  
  
  • Changing the PyJWT library to joserfc and the token signing algorithm from RSA-2048 to Ed25519.
  
  
  • The built-in synchronous password hashing at the ORM level of the advanced-alchemy library was moved to a ThreadPoolExecutor(max_workers=2). The value max_workers=2 was chosen strictly for the 2 physical cores of the test bench (running the Uvicorn server with core affinity to 0,1).
  
  
  
  

Preliminary Analysis of Event Loop Blocking:

Before proceeding to analyze the flame graphs from the py-spy profiler and load testing, we will switch asyncio to debug mode at the FastAPI application level to record event loop slowdowns at runtime:

@asynccontextmanager

async def lifespan(app: FastAPI):

    # Enable asyncio debug mode

    loop = asyncio.get_running_loop()

    loop.set_debug(True)

    # Set the threshold to 100 ms (0.1 sec)

    loop.slow_callback_duration = 0.1

    yield

Results of logging "heavy" callbacks:

• 
  /api/v1/access/signup: Thread blocking during synchronous password hashing was ~150 ms.


• 
  /api/v1/access/signin: The total blocking time for password verification and the sequential issuance of an access/refresh token pair reaches ~350–400 ms (of which: password verification — ~150 ms, token generation — ~250 ms).

Note: The figures provided are Wall-clock time (total astronomical delay time) recorded by asyncio debug. This is not pure cryptography mathematics in a vacuum, but the total time during which the main Python thread was monopolized by computations without returning control to the Event Loop. These measurements serve as a clear demonstration of how heavy CPU-bound code paralyzes the asynchronous runtime.

Flame Graph Analysis

• 
  

  Conditions:

  
  
  
  
  
  • 
    Application Server: Uvicorn, 1 worker, mapped to 1 physical module (core 0).
  
  
  • 
    py-spy (record mode, no core affinity) | wrk2 (mapped to 1 physical module (core 7)):
  
  
  • Endpoint /api/v1/access/signup: py-spy -> --rate 100; wrk2 -> RPS 4, 1 threads.
  
  
  • Endpoint /api/v1/access/signin: py-spy -> --rate 100 (increased to 150 after optimization); wrk2 -> RPS 2, 1 threads.
  
  
  • Endpoint /api/v1/access/me: py-spy -> --rate 150; wrk2 -> RPS 300, 1 threads.
  
  
  • 
    Docker containers: PostgreSQL, PGBouncer (cores 2,3) | Valkey (cores 4,5) were running. ulimit (nofile=65535) was set for all containers.
  
  
  • OS limits were raised (see OS Tuning)
  
  
  
  


• 
  

  Baseline Analysis:

  
  
  
  
  
  • Endpoint /api/v1/access/signup: The CPU time share for synchronous user password hashing was within ~22-25%. The flame graph clearly shows a deep call stack from the ORM integration level (advanced_alchemy/types/password...) down to the low-level argon2.low_level.hash_secret library, which monopolizes CPU time within the main thread.
  
  
  • Endpoint /api/v1/access/signin: The flame graph shows the cumulative effect of blocking the Event Loop with two heavy CPU-bound operations within a single request. The share of synchronous password verification via argon2.low_level.verify_secret takes about ~10–12% of the worker's CPU time. The main overhead in the execution profile is formed by the sequential issuance of a pair of JWT tokens (access/refresh) using the asymmetric RSA-2048 algorithm (jwt.algorithms.prepare_key / encode_jwt), taking up ~32–34% of CPU time for each token (a total of ~66% of the entire graph width).
  
  
  • Endpoint /api/v1/access/me: Decoding and verifying the signature of the incoming JWT token using the asymmetric RSA-2048 algorithm (decode_jwt -> pyjwt.verify) takes about ~10–12% of CPU time. At the same time, the layer for working with the Valkey cache backend and deserializing user data using msgspec take up a minimal share of the CPU.
  
  
  
  

Artifacts: | | ).