You've been coding for hours.
The feature is elegant.
Tests pass.
Code review looks great.
You hit merge.
Forty-five minutes later, your Slack lights up. Not a polite ping—an explosion.
🚨 Red alerts.
😤 Customer complaints.
😰 Your CEO asking, "What's happening?"
Your AI support agents—the ones customers love—are frozen.
Conversations stuck.
ChromaDB timing out.
Database maxed out.
Everything you built is collapsing in real time.
By the end, you'll trace it to three innocent lines of code you wrote on a quiet Tuesday morning.
TL;DR — The One-Line Disaster
Calling .save() inside a Django signal caused:
- Recursive writes
- Triggered related signals
- Overwhelmed the database
- Blocked Celery workers
- Full system meltdown under load
The fix:
Signals should never modify models. Use them only to queue async work after commit.
The Setup: Everything Looked Perfect
We built a clean pipeline:
- Users send support questions
- LangGraph agents (OpenAI-powered) respond
- Messages get stored
- Embeddings added to ChromaDB for semantic search
Our models were straightforward:
class SupportConversation(models.Model):
customer_id = models.CharField(max_length=100)
agent_name = models.CharField(max_length=50)
status = models.CharField(max_length=20, default='active')
updated_at = models.DateTimeField(auto_now=True)
class ConversationMessage(models.Model):
conversation = models.ForeignKey(SupportConversation, on_delete=models.CASCADE)
role = models.CharField(max_length=20) # 'user' or 'assistant'
content = models.TextField()
embedding_id = models.CharField(max_length=255, null=True)
Then one Tuesday morning, a developer added this:
@receiver(post_save, sender=ConversationMessage)
def handle_message_saved(sender, instance, created, **kwargs):
if created and instance.role == 'assistant':
generate_embeddings_task.delay(instance.id)
instance.conversation.save() # ← The ticking time bomb
It seemed harmless.
Tests passed.
Staging worked.
10 users? No problem.
And then Friday happened.
The Meltdown: How It All Fell Apart
11:35 AM — Marketing launches a flash promo.
"Try our AI support agents—free for 24 hours."
Within 60 seconds, 500 fresh chat sessions flood in.
What happens next:
A user sends a message →
AI responds →
ConversationMessage saves →
Signal fires →
Celery task queued →
conversation.save() triggers →
Another signal fires →
Database write →
Repeat… and repeat… and repeat.
Each conversation generates multiple unexpected writes.
The chain reaction unfolds:
11:36:00 — DB connection pool: 48/50 in use
11:36:10 — Celery workers blocking, waiting for DB
11:36:20 — ChromaDB calls backing up behind DB queue
11:36:30 — 5,000 tasks in queue
11:36:50 — 25,000 tasks queued
11:37:00 — Everything is stuck
Messages stop saving.
Requests time out.
ChromaDB errors.
And then Slack explodes:
@oncall: "AI support is frozen."
@product: "Customers can't get replies."
@infra: "Why are Celery workers dead?"
@devops: "DB CPU at 100%, restarting now..."
The platform collapses.
Recovery time: 4+ hours.
The root cause? Nested signals triggering recursive .save() calls.
Root Cause: The Recursive Death Spiral
ConversationMessage.save()
↓
post_save signal fires
↓
instance.conversation.save() ← nested save
↓
post_save (SupportConversation) fires
↓
Model writes again
↓
Triggers more signals
↓
More writes under load = exponential explosion
Under high concurrency, this becomes an accidental DDoS against your own database.
The Four Fixes That Saved Us
1. Always Guard with created
Ensure the handler fires only once:
from django.db.models.signals import post_save
from django.dispatch import receiver
from django.db import transaction
@receiver(post_save, sender=ConversationMessage)
def queue_message_processing(sender, instance, created, **kwargs):
# ✅ Only run on NEW messages
if not created:
return
if instance.role == 'assistant':
# ✅ Queue AFTER transaction commits (no nested saves)
transaction.on_commit(
lambda: generate_embeddings_task.delay(instance.id)
)
Why it works:
- No repeated firing
- No nested writes
- The async task runs only after the DB transaction commits
- Clean separation of concerns
2. Let Celery Update Models Safely
Use update_fields to skip signal handlers:
from celery import shared_task
@shared_task
def generate_embeddings_task(message_id):
message = ConversationMessage.objects.get(id=message_id)
# Call OpenAI API
embedding = openai.embeddings.create(
model="text-embedding-3-small",
input=message.content
).data[0].embedding
# Store in ChromaDB
chroma_client.add(
ids=[f"msg_{message_id}"],
embeddings=[embedding]
)
# ✅ CRITICAL: Use update_fields to skip the signal handler
ConversationMessage.objects.filter(id=message_id).update(
embedding_id=f"msg_{message_id}"
)
Update your handler to recognize this:
@receiver(post_save, sender=ConversationMessage)
def queue_message_processing(sender, instance, created, update_fields=None, **kwargs):
if not created:
# ✅ If only embedding_id changed, skip entirely
if update_fields and update_fields == {'embedding_id'}:
return
return
# ... rest of handler
Why it works:
- Celery can safely modify models
- Queryset updates with
update_fieldsbypass post_save signals - No recursive loops even under heavy load
3. Use QuerySet Updates for Parent Stats
❌ DON'T do this:
conversation.message_count = 5
conversation.save() # ❌ triggers another signal!
✅ DO this instead:
SupportConversation.objects.filter(id=conversation.id).update(
message_count=conversation.conversationmessage_set.count()
)
Why:
- QuerySet operations bypass signals entirely
- No cascade
- Atomic database operation
- 10x faster
4. Batch Updates Instead of Per-Message
Instead of firing stats updates on every message, batch them:
from celery import shared_task
from django.db.models import Count
@shared_task
def batch_update_conversation_stats():
"""
Run every 30 seconds via Celery Beat.
Update all active conversations at once.
"""
conversations = SupportConversation.objects.filter(
status='active'
).annotate(msg_count=Count('conversationmessage'))
for conv in conversations:
SupportConversation.objects.filter(id=conv.id).update(
message_count=conv.msg_count
)
# In celery.py, schedule with:
# 'batch-update-stats': {
# 'task': 'myapp.tasks.batch_update_conversation_stats',
# 'schedule': crontab(minute='*/1'), # Every minute
# }
The math:
- 500 concurrent messages = 500 individual updates (❌ cascades)
- 500 concurrent messages = 1 batch update every 30 seconds (✅ efficient)
Testing for Signal Cascades (Critical!)
You must test this:
from django.test import TestCase
from django.test.utils import CaptureQueriesContext
from django.db import connection
class ConversationSignalTests(TestCase):
def test_message_creation_is_silent(self):
"""
Verify: Creating ONE message = ONE database write.
If this fails, you have a signal loop.
"""
conv = SupportConversation.objects.create(customer_id="test_user")
with CaptureQueriesContext(connection) as ctx:
ConversationMessage.objects.create(
conversation=conv,
role='user',
content="Help!",
tokens_used=1,
response_time_ms=10
)
writes = [q for q in ctx.captured_queries
if q['sql'].startswith(('INSERT', 'UPDATE'))]
self.assertEqual(len(writes), 1,
f"Expected 1 write, got {len(writes)}. Signal loop detected!")
def test_load_no_cascade(self):
"""
Simulate production load: 100 concurrent messages.
Verify no cascade explosion.
"""
convs = [
SupportConversation.objects.create(customer_id=f"user_{i}")
for i in range(100)
]
with CaptureQueriesContext(connection) as ctx:
for conv in convs:
ConversationMessage.objects.create(
conversation=conv,
role='user',
content="Question",
tokens_used=3,
response_time_ms=15
)
writes = [q for q in ctx.captured_queries
if q['sql'].startswith(('INSERT', 'UPDATE'))]
self.assertLess(len(writes), 150,
f"Too many writes: {len(writes)}. Cascade detected!")
Best Practices Checklist
✔ Always Do:
- ✅ Use the
createdflag guard - ✅ Use
transaction.on_commit()for async work - ✅ Keep signals pure—never modify models
- ✅ Use
queryset.update(), never.save()in handlers - ✅ Use
update_fieldswhen Celery updates models - ✅ Batch expensive tasks (stats, aggregations)
- ✅ Stress test with concurrent load
- ✅ Monitor signal depth in production
✘ Never Do:
- ❌ Call
.save()inside a signal handler - ❌ Trigger parent model saves from child signals
- ❌ Assume "it works locally" means "it scales"
- ❌ Trust signals under load without testing
- ❌ Create circular signal dependencies
Final Lesson: The Signal Rule
Django signals are powerful.
They're also silent footguns.
One innocent .save() buried in a signal handler can:
- 💥 Hammer your database
- 🧊 Freeze your async workers
- 📉 Cascade into platform-wide downtime
- 😞 Ruin your Friday
The rule is simple:
Signals should queue async work—not change database state.
Celery should handle all updates.
Follow this, and you'll never meet the monster we met at 11:36 AM on a Friday.
Signal fires
↓
Queue Celery task
↓
Commit transaction
↓
Celery task runs safely
↓
Updates database (signal-free)
↓
✅ No cascade
Resources
- Django Signals Documentation
- Django Transactions & on_commit
- Celery Best Practices
- ChromaDB Python Client
Last updated: December 1, 2025
Have your own signal horror story? Drop it in the comments—you're not alone. 🚀