INTRODUCTION
While learning Python's asyncio, I understood coroutines and the event loop. Coroutines can run concurrently by pausing at await, allowing the event loop to switch between them.
But after understanding this, I had a few questions:
If I create 1000 coroutines, will all of them access the database simultaneously?
If two coroutines update the same variable, won't they overwrite each other's changes?
If producers generate work faster than consumers process it, where does that work wait?
These questions led me to Python's asynchronous synchronization primitives. They are not used to make coroutines asynchronous—they are used to coordinate asynchronous coroutines safely.
In this article, I'll explain why Semaphore, Lock, and Queue exist, the problems they solve, and how they are used in real-world backend applications.
What You Will Learn
Why synchronization primitives exist
Why the event loop alone is not enough
The problem solved by asyncio.Semaphore
The problem solved by asyncio.Lock
The problem solved by asyncio.Queue
Internal working of each concept
Real-world backend examples
How these concepts work together
Prerequisites
Before reading this article, you should understand:
Coroutines
asyncio
Event Loop
await
asyncio.create_task()
asyncio.gather()
The Problem
When I first learned asyncio, I thought:
Since Python uses only one thread with the event loop, why do we even need synchronization?
Initially this made sense because only one coroutine executes at a particular instant.
But later I realized something important.
A coroutine can pause whenever it reaches an await.
Coroutine A
↓
Reads shared data
↓
await
↓
Coroutine B starts executing
↓
Modifies same data
↓
Coroutine A resumes
Now both coroutines are working on the same resource.
Similarly,
1000 Coroutines
↓
All call the same API
↓
Server overloaded
Or,
1000 Jobs
↓
Only 5 workers
↓
Where should remaining jobs wait?
These problems cannot be solved by the event loop itself.
They require synchronization.
Why Async Synchronization Exists
The event loop schedules coroutines.
Synchronization primitives coordinate coroutines.
These are completely different responsibilities.
Event Loop
↓
Decides WHO executes
Synchronization
↓
Controls HOW they execute
Python provides three important synchronization primitives.
Semaphore
↓
Limit concurrent access
Lock
↓
Protect shared resources
Queue
↓
Store and distribute work
asyncio.Semaphore
The Problem
Suppose you create 1000 coroutines.
Each coroutine calls an external API.
Coroutine1
Coroutine2
Coroutine3
...
Coroutine1000
Without any limit,
all 1000 requests may start together.
This can:
overload the backend
exceed API rate limits
consume unnecessary memory
Why Semaphore Exists
A semaphore limits how many coroutines are allowed to execute a particular section simultaneously.
Semaphore(3)
↓
Permit
Permit
Permit
Only three coroutines may enter.
The remaining coroutines wait.
Internal Working
Task1 enters
↓
Permit Count = 2
Task2 enters
↓
Permit Count = 1
Task3 enters
↓
Permit Count = 0
Task4 arrives
↓
Wait
Task2 finishes
↓
Permit released
↓
Task4 enters
The semaphore is automatically released when execution leaves the async with block.
Example
import asyncio
semaphore = asyncio.Semaphore(3)
async def worker(task_id):
async with semaphore:
print(f"Task {task_id} started")
await asyncio.sleep(2)
print(f"Task {task_id} finished")
Real-world Example
Suppose your backend downloads images.
1000 Images
↓
Semaphore(20)
↓
Only 20 downloads happen simultaneously.
This prevents excessive resource usage.
What if Semaphore didn't exist?
1000 Tasks
↓
1000 API Calls
↓
Rate Limit
↓
Failures
asyncio.Lock
The Problem
Imagine two coroutines updating the same bank balance.
Balance = ₹1000
Coroutine A
Read Balance
↓
Add ₹500
↓
Write Balance
Coroutine B
Read Balance
↓
Subtract ₹200
↓
Write Balance
Both read the same value before either writes it.
One update overwrites the other.
This is called a race condition.
Critical Section
A critical section is a block of code that accesses or modifies shared resources and therefore should only be executed by one coroutine at a time.
Why Lock Exists
A lock ensures only one coroutine executes the critical section at a time.
Other coroutines wait until the lock is released.
Internal Working
Coroutine1 acquires lock
↓
Critical Section
↓
Coroutine2 waits
↓
Coroutine1 finishes
↓
Lock Released
↓
Coroutine2 enters
Example
lock = asyncio.Lock()
async def update_balance():
async with lock:
balance = await get_balance()
balance += 500
await save_balance(balance)
Real-world Example
Inventory Management
Product Quantity = 1
Two users purchase simultaneously.
Without Lock,
both may successfully purchase the same product.
With Lock,
only one coroutine updates the inventory at a time.
What if Lock didn't exist?
Coroutine A
↓
Read
↓
await
↓
Coroutine B
↓
Modify
↓
Coroutine A resumes
↓
Incorrect Data
asyncio.Queue
The Problem
Imagine customers placing orders faster than chefs can prepare them.
Orders
↓
1
2
3
4
5
6
Only two chefs are available.
Where should the remaining orders wait?
Why Queue Exists
A queue temporarily stores work until workers become available.
It follows the FIFO (First In, First Out) principle.
Internal Working
Producer
↓
queue.put()
↓
Queue
↓
queue.get()
↓
Consumer
↓
queue.task_done()
Example
queue = asyncio.Queue()
await queue.put("Order1")
job = await queue.get()
queue.task_done()
Real-world Example
Image Processing
User uploads image
↓
Queue
↓
Background Worker
↓
Compress Image
↓
Generate Thumbnail
↓
Upload
The user gets an immediate response while background workers process the image.
What if Queue didn't exist?
1000 Jobs
↓
Workers Busy
↓
Jobs Lost
or
The producer must continuously wait for a worker to become free.
How These Three Work Together
In a real backend system, these primitives are often used together.
User Request
↓
Queue
↓
Worker
↓
Semaphore
↓
Lock
↓
Database
Each primitive solves a different problem.
Queue stores incoming work.
Semaphore limits concurrent processing.
Lock protects shared data.
Advantages
Semaphore
Prevents resource exhaustion
Controls concurrency
Helps respect API rate limits
Lock
Prevents race conditions
Protects shared resources
Ensures data consistency
Queue
Buffers incoming work
Implements producer-consumer architecture
Decouples producers from workers
Conclusion
Initially, I thought the event loop alone was enough because only one coroutine executes at a time. But while learning more, I realized that asynchronous applications face a different set of problems:
Too many coroutines may compete for limited resources.
Multiple coroutines may modify shared data.
Producers and consumers may run at different speeds.
These problems are solved by asyncio.Semaphore, asyncio.Lock, and asyncio.Queue.
Understanding why these synchronization primitives exist is much more valuable than simply memorizing their syntax, because once the problem is clear, the solution becomes intuitive.
SOCIAL SHARE CARD GENERATOR