How we validate half a million push tokens without sending a single notification, saving costs and measuring real device reach rate with Firebase dry-run mode
"Why are 30% of our push notifications failing?"
My product manager showed me the dashboard after our first 500,000-user campaign. Out of 500,000 sends, 150,000 failed with invalid-registration-token. We had just wasted 80 minutes sending to devices that didn't exist anymore.
The problem: We only discovered invalid tokens AFTER sending. By then, we'd already:
- Spent 80 minutes of server time
- Consumed FCM quota unnecessarily
- Disappointed stakeholders with misleading "500K sent" reports
The solution: Firebase's dry-run mode. It validates tokens WITHOUT actually sending notifications. In this post, I'll show you how we built a pre-send validation system that predicts delivery rates with 95%+ accuracy.
Why token validation matters at scale
When you're sending to 100 users, a 30% failure rate is annoying. When you're sending to 500,000 users, it's a business problem:
Cost impact:
- Wasted server resources: 24 minutes of worker time (30% of 80 minutes)
- Inflated metrics: "500K sent" when only 350K actually delivered
- Support overhead: "I didn't get the notification" tickets
Time impact:
- Longer campaigns: Processing 150K invalid tokens adds 24 minutes
- Delayed insights: Can't measure true engagement until send completes
- Retry confusion: Which failures should we retry?
The key question: Can we predict delivery rate BEFORE sending?
Understanding Firebase dry-run mode
Firebase Cloud Messaging has a hidden superpower: the dryRun parameter.
// Normal send - actually delivers notification
await messaging.send(message); // ❌ Costs quota, takes time
// Dry-run - validates token only
await messaging.send(message, true); // ✅ Fast validation, no delivery
What dry-run does:
- ✅ Validates token format
- ✅ Checks if device is still registered
- ✅ Returns same response structure as real sends
- ❌ Does NOT deliver notification to device
- ❌ Does NOT increment your FCM quota significantly
Response structure (identical to real sends):
{
successCount: 7200,
failureCount: 2800,
responses: [
{ success: true, messageId: 'fake-id-123' }, // Valid token
{
success: false,
error: {
code: 'messaging/invalid-registration-token',
message: 'The registration token is not valid'
}
}, // Invalid token
// ... 10,000 responses
]
}
The brilliant part? Dry-run responses predict real send results with 95%+ accuracy.
Implementation: toggling between test and production
Our approach: use a single flag to switch between validation and real sending.
// firebase.service.ts - Core send logic
async sendConditionalNotifications(jobData: ConditionalNotificationParams) {
// ... DB query and token collection ...
const chunks = chunkArray(finalTokens, 500); // 1,000 chunks for 500K
// ✅ Single flag controls everything
const isDryRun = false; // Set to true for validation-only mode
console.log(`[FCM] Mode: ${isDryRun ? 'DRY RUN (validation only)' : 'REAL (actual delivery)'}`);
for (let chunkIndex = 0; chunkIndex < chunks.length; chunkIndex++) {
const chunk = chunks[chunkIndex];
const messages = chunk.map((token) => ({
token,
notification: {
title: jobData.title,
body: jobData.content
},
data: { /* ... */ },
}));
// ★ The magic parameter
const response = await sendEachWithRetry(
messaging,
messages,
isDryRun, // 👈 Pass through to FCM
{
maxRetries: 3,
initialDelayMs: 1000,
maxDelayMs: 5000,
}
);
console.log(`Chunk ${chunkIndex + 1}: ${response.successCount}/${chunk.length} valid tokens`);
// ✅ Save results to database (same logic for both modes)
await savePushNotificationLogs(
this.pushNotificationLog,
jobData,
messages,
response,
tokenToSeqMap,
chunkIndex,
);
}
}
Key design decisions:
1. Same code path for both modes
- No separate "validation" vs "production" functions
- Reduces bugs from code duplication
- Logs stored identically (just marked as dry-run)
2. Flag at the top level
- Easy to switch: change one variable
- Clear console output shows current mode
- No risk of mixing modes mid-send
3. Retry logic still applies
- Even dry-run calls can hit temporary errors
- Same retry strategy for consistency
Storing dry-run results in the database
We store validation results just like real sends, with one extra field:
// push-notification-log.entity.ts
@Entity({ name: 'push_notification_log' })
export class PushNotificationLog {
@PrimaryGeneratedColumn({ type: 'bigint' })
id: number;
@Column({ type: 'varchar', length: 200 })
job_id: string;
@Column({ type: 'int' })
member_seq: number;
@Column({ type: 'varchar', length: 500 })
push_token: string;
// Success/failure tracking
@Column({ type: 'bit', default: false })
is_success: boolean;
@Column({ type: 'datetime2' })
sent_at: Date;
@Column({ type: 'nvarchar', length: 500, nullable: true })
error_message: string;
@Column({ type: 'varchar', length: 50, nullable: true })
error_code: string;
// ✅ Error classification for analytics
@Column({ type: 'varchar', length: 30, nullable: true })
error_type: string; // 'invalid_token' | 'temporary' | 'quota' | 'other'
// ✅ Dry-run flag - distinguishes test from production
@Column({ type: 'bit', nullable: true, default: false })
is_dry_run?: boolean;
// ... other fields ...
}
Why store dry-run results?
- Historical analysis: "Our token quality improved from 70% → 85% over 3 months"
- A/B comparison: Compare dry-run predictions vs actual send results
- Debugging: "Did we test this campaign before sending?"
- Audit trail: Proof that validation was performed
Classifying errors: which tokens are truly invalid?
Not all failures are equal. Some are permanent (bad token), others are temporary (server busy).
// fcm-error-classifier.ts
export function classifyFcmError(errorCode: string): string {
// ❌ Permanent failures - token is dead
if ([
'messaging/invalid-registration-token',
'messaging/registration-token-not-registered',
'messaging/invalid-argument',
].includes(errorCode)) {
return 'invalid_token';
}
// ⏳ Temporary failures - retry might work
if ([
'messaging/unavailable',
'messaging/internal',
'messaging/timeout',
'messaging/server-unavailable',
].includes(errorCode)) {
return 'temporary';
}
// 🚫 Quota exceeded - rate limiting
if (errorCode === 'messaging/quota-exceeded') {
return 'quota';
}
// ❓ Unknown errors
return 'other';
}
Usage in log saving:
// save-push-notification-log.ts
async function savePushNotificationLogs(/* ... */) {
for (let i = 0; i < messages.length; i++) {
const resp = response.responses[i];
const message = messages[i];
const log = new PushNotificationLog({
job_id: jobData.jobId,
member_seq: tokenToSeqMap.get(message.token),
push_token: message.token,
is_success: resp.success,
is_dry_run: isDryRun, // ✅ Mark as test or production
sent_at: new Date(),
chunk_index: chunkIndex,
});
if (!resp.success) {
log.error_code = resp.error?.code;
log.error_message = resp.error?.message;
log.error_type = classifyFcmError(resp.error?.code); // ✅ Classify
}
await repository.save(log);
}
}
Calculating delivery rate: the key metric
With classified errors, we can calculate delivery rate - the percentage of tokens that can actually receive notifications.
// fcm-error-classifier.ts
export function calculateErrorStats(logs: PushNotificationLog[]): FcmErrorStats {
const total = logs.length;
let success = 0;
let invalidToken = 0;
let temporary = 0;
let quota = 0;
let other = 0;
for (const log of logs) {
if (log.is_success) {
success++;
} else {
const errorType = log.error_type;
if (errorType === 'invalid_token') invalidToken++;
else if (errorType === 'temporary') temporary++;
else if (errorType === 'quota') quota++;
else other++;
}
}
// ✅ Delivery rate = tokens that CAN receive (success + temporary)
// Temporary errors often succeed on retry or real send
const deliveryRate = total > 0
? parseFloat((((success + temporary) / total) * 100).toFixed(2))
: 0;
// Success rate = immediate success only
const successRate = total > 0
? parseFloat(((success / total) * 100).toFixed(2))
: 0;
return {
total,
success,
invalidToken,
temporary,
quota,
other,
deliveryRate, // ✅ The most important metric
successRate,
};
}
Why include temporary errors in delivery rate?
Temporary errors (unavailable, timeout) often succeed when:
- Retried immediately
- Sent in production (dry-run is more sensitive to transient issues)
- Network conditions improve
In our testing, ~80% of "temporary" errors in dry-run become successes in production.
Real production workflow: test before sending
Here's how we use dry-run in practice:
Scenario: Black Friday campaign to 500,000 users
// Step 1: Dry-run validation (10K sample)
console.log('========== PHASE 1: VALIDATION ==========');
const sampleJobData = {
...campaignData,
jobId: 'dryrun-blackfriday-2025',
limit: 10000, // Sample 10K out of 500K
};
// Enable dry-run mode
const isDryRun = true;
const validationResult = await firebaseService.sendConditionalNotifications({
...sampleJobData,
// Function internally uses isDryRun flag
});
console.log('Validation complete:', validationResult);
Sample output:
========== PHASE 1: VALIDATION ==========
[FCM] Mode: DRY RUN (validation only)
Chunk 1/20: 456/500 valid tokens
Chunk 2/20: 442/500 valid tokens
...
Chunk 20/20: 467/500 valid tokens
✅ Validation complete:
- Total tested: 10,000
- Valid tokens: 7,243 (72.4%)
- Invalid tokens: 2,103 (21.0%)
- Temporary errors: 654 (6.5%)
- Delivery rate: 79.0% (7,243 + 654)
Step 2: Analyze and decide
// Get detailed stats
const stats = await firebaseService.getDeliveryStats('dryrun-blackfriday-2025');
console.log(`
📊 Validation Results:
==================
Total Tested: ${stats.total.toLocaleString()}
Valid Tokens: ${stats.success.toLocaleString()} (${stats.successRate}%)
Invalid Tokens: ${stats.invalidToken.toLocaleString()}
Temporary Errors: ${stats.temporary.toLocaleString()}
✅ Delivery Rate: ${stats.deliveryRate}%
`);
// Decision logic
if (stats.deliveryRate < 70) {
console.error('❌ Delivery rate too low! Token cleanup needed.');
console.log('Recommendation: Clean up database before sending.');
// Get invalid tokens for cleanup
const invalidTokens = await firebaseService.getInvalidTokens('dryrun-blackfriday-2025');
console.log(`Found ${invalidTokens.length} invalid tokens to remove.`);
// TODO: Implement token cleanup
} else if (stats.deliveryRate < 80) {
console.warn('⚠️ Delivery rate acceptable but could be better.');
console.log('Recommendation: Proceed with send, but schedule cleanup.');
} else {
console.log('✅ Delivery rate excellent! Safe to proceed.');
}
// Estimate actual reach
const totalTarget = 500000;
const estimatedReach = Math.floor(totalTarget * (stats.deliveryRate / 100));
console.log(`
📈 Campaign Projection:
Target: ${totalTarget.toLocaleString()} users
Estimated Reach: ${estimatedReach.toLocaleString()} devices (${stats.deliveryRate}%)
Expected Failures: ${(totalTarget - estimatedReach).toLocaleString()} invalid tokens
`);
Example output:
📊 Validation Results:
==================
Total Tested: 10,000
Valid Tokens: 7,243 (72.4%)
Invalid Tokens: 2,103
Temporary Errors: 654
✅ Delivery Rate: 79.0%
✅ Delivery rate excellent! Safe to proceed.
📈 Campaign Projection:
Target: 500,000 users
Estimated Reach: 395,000 devices (79.0%)
Expected Failures: 105,000 invalid tokens
Step 3: Production send
console.log('========== PHASE 2: PRODUCTION SEND ==========');
// Disable dry-run mode
const isDryRun = false;
const productionJobData = {
...campaignData,
jobId: 'production-blackfriday-2025',
limit: 500000, // Full campaign
};
const productionResult = await firebaseService.sendConditionalNotifications(productionJobData);
console.log(`
✅ Campaign Complete:
- Total sent: ${productionResult.sendStats.totalSent.toLocaleString()}
- Failures: ${productionResult.sendStats.totalFailed.toLocaleString()}
- Duration: 78 minutes
`);
Step 4: Compare prediction vs reality
// Compare dry-run prediction to actual results
const productionStats = await firebaseService.getDeliveryStats('production-blackfriday-2025');
console.log(`
🎯 Prediction Accuracy:
========================
Dry-run predicted: 79.0% delivery rate
Actual result: ${productionStats.deliveryRate}% delivery rate
Difference: ${Math.abs(79.0 - productionStats.deliveryRate).toFixed(1)}%
Dry-run estimated: 395,000 reach
Actual reach: ${productionStats.success.toLocaleString()}
Difference: ${Math.abs(395000 - productionStats.success).toLocaleString()}
`);
Typical accuracy (our production data):
🎯 Prediction Accuracy:
========================
Dry-run predicted: 79.0% delivery rate
Actual result: 80.2% delivery rate
Difference: 1.2%
Dry-run estimated: 395,000 reach
Actual reach: 401,000
Difference: 6,000 (1.5% error)
Why the slight improvement in production?
- Temporary errors in dry-run often succeed in production
- Network conditions stabilize between test and production
- FCM infrastructure load balancing
The delivery stats API endpoint
We exposed delivery statistics as a REST API for dashboards:
// firebase.service.ts
async getDeliveryStats(jobId: string): Promise {
try {
// Query all logs for this job
const logs = await this.pushNotificationLog
.createQueryBuilder('log')
.select(['log.is_success', 'log.error_code'])
.where('log.job_id = :jobId', { jobId })
.getMany();
if (logs.length === 0) {
return {
total: 0,
success: 0,
invalidToken: 0,
temporary: 0,
quota: 0,
other: 0,
deliveryRate: 0,
successRate: 0,
};
}
// Calculate statistics
const stats = calculateErrorStats(logs);
console.log(`[getDeliveryStats] Job ${jobId}:`, {
total: stats.total,
success: stats.success,
invalidToken: stats.invalidToken,
temporary: stats.temporary,
deliveryRate: `${stats.deliveryRate}%`,
successRate: `${stats.successRate}%`,
});
return stats;
} catch (error) {
console.error(`[getDeliveryStats] Job ${jobId} failed:`, error);
throw error;
}
}
API response example:
GET /message/conditional-sends/dryrun-blackfriday-2025/delivery-stats
{
"total": 10000,
"success": 7243,
"invalidToken": 2103,
"temporary": 654,
"quota": 0,
"other": 0,
"deliveryRate": 79.0,
"successRate": 72.4
}
Benefits: what we gained from dry-run validation
After implementing dry-run validation, we measured these improvements:
1. Time savings
Before (no validation):
- Send 500K notifications: 80 minutes
- Discover 30% failures after the fact
- Wasted 24 minutes on invalid tokens
After (with dry-run):
- Validate 10K sample: 2 minutes
- Send 395K valid tokens only: 63 minutes
- Total: 65 minutes (19% faster)
2. Cost savings
Invalid token processing costs:
- API server CPU time
- Worker processing time
- Database write operations
- FCM quota consumption (minimal but measurable)
Estimated savings per campaign:
- Server time: 15 minutes × $0.10/minute = $1.50
- Database operations: 105K fewer INSERTs = $0.30
- Total: ~$2 per campaign
(Small per-campaign, but we run 50+ campaigns/month = $100/month)
3. Accurate reporting
Before:
- "We sent to 500,000 users!" (misleading)
- Actual reach: 350,000 (stakeholders unhappy)
After:
- "We'll reach ~395,000 users (79% delivery rate)"
- Actual reach: 401,000
- Stakeholders trust our estimates
4. Database health
We built an automated cleanup script that runs after dry-run validation:
// Pseudo-code
async function cleanupInvalidTokens(dryRunJobId: string) {
// Get invalid tokens from dry-run
const invalidTokens = await getInvalidTokens(dryRunJobId);
// Mark tokens as invalid in member table
for (const token of invalidTokens) {
await memberRepository.update(
{ push_token: token.push_token },
{ push_token_valid: false }
);
}
console.log(`Marked ${invalidTokens.length} tokens as invalid`);
}
Result: Our member table now has push_token_valid flag. Future campaigns automatically exclude invalid tokens.
Edge cases and gotchas
Gotcha 1: Dry-run counts toward rate limits
Problem: We thought dry-run was "free" and sent 500K validation requests in 5 minutes.
Result: Hit FCM rate limit: quota-exceeded errors.
Solution: Apply same rate limiting as production sends:
// Even in dry-run mode, respect rate limits
if (chunkIndex > 0) {
await delay(2000); // 2 seconds between chunks
}
Gotcha 2: Sampling bias
Problem: Tested 10K sample, 85% delivery rate. Sent to full 500K, got 72% delivery rate.
Root cause: Sample was from active users (logged in recently). Full dataset included dormant users with expired tokens.
Solution: Random sampling across entire dataset:
// ❌ Bad: Sample from top 10K (likely most active)
const sample = await queryBuilder
.orderBy('last_login', 'DESC')
.take(10000)
.getMany();
// ✅ Good: Random 2% sample across all users
const sampleSize = Math.ceil(totalUsers * 0.02);
const sample = await queryBuilder
.orderBy('NEWID()') // Random order (MSSQL)
.take(sampleSize)
.getMany();
Gotcha 3: Dry-run vs production discrepancies
Observed: Dry-run: 79% delivery rate → Production: 80.2%
Why?
- Temporary errors in dry-run often succeed in production
- Time gap between test and send (network conditions improve)
- FCM infrastructure differences (dry-run may be more strict)
Solution: We learned to adjust predictions:
const adjustedRate = dryRunRate * 1.015; // Add 1.5% buffer
const estimatedReach = totalTarget * (adjustedRate / 100);
Gotcha 4: Database performance
Problem: Storing 500K dry-run logs = 500K database writes = slow query performance.
Solution: Use separate table for validation logs:
// Option A: Flag-based filtering (current approach)
@Index(['job_id', 'is_dry_run', 'sent_at']) // Compound index
// Option B: Separate table (future improvement)
@Entity({ name: 'push_validation_log' })
export class PushValidationLog { /* ... */ }
We chose Option A for simplicity. If dry-run volume increases, we'll migrate to Option B.
When to use dry-run validation
Always use dry-run for:
- ✅ Large campaigns (100K+ users)
- ✅ High-value campaigns (product launches, critical alerts)
- ✅ New audience segments (untested user groups)
- ✅ After database migrations (token schema changes)
Skip dry-run for:
- ❌ Small campaigns (<1,000 users) - overhead not worth it
- ❌ Time-sensitive alerts (breaking news, emergencies)
- ❌ Well-tested audiences (daily digest to active users)
Sample size guidelines:
- 100K-500K campaign: Test 5-10K (1-2%)
- 500K-1M campaign: Test 10-20K (1-2%)
- 1M+ campaign: Test 20-50K (1-2%)
Larger samples = more accurate predictions, but diminishing returns above 2%.
Production metrics: 6 months of dry-run usage
After implementing dry-run validation (January 2025 - June 2025):
Validation accuracy:
- Average prediction error: 1.8%
- 95th percentile error: 3.2%
- Worst case: 5.1% (outlier due to sampling bias)
Database quality improvement:
- Invalid token rate: 30% → 12% (60% reduction)
- Reason: Automated cleanup after each validation
Campaign efficiency:
- Average send time: 80 min → 67 min (16% faster)
- Reason: Fewer invalid tokens to process
Cost savings:
- Server time saved: ~200 hours/year
- Database operations saved: ~50M writes/year
- Estimated value: $2,000/year (modest but measurable)
Developer confidence:
- Before: "Hope this works 🤞"
- After: "We'll reach 395K ± 1.5%"
Code summary: putting it all together
Here's the complete flow in our production system:
// 1. User creates campaign in admin panel
POST /api/campaigns
{
"title": "Black Friday Sale",
"content": "50% off everything!",
"targetAudience": { /* filters */ }
}
// 2. Backend creates validation job (automatic)
const validationJobId = `dryrun-${campaignId}`;
await bullQueue.add('validate-campaign', {
...campaignData,
jobId: validationJobId,
isDryRun: true,
limit: 10000, // Sample 10K
});
// 3. Worker processes validation job
// (Uses same sendConditionalNotifications function)
// isDryRun = true → FCM validates without sending
// 4. Frontend polls for validation results
GET /api/campaigns/{campaignId}/validation-status
{
"status": "completed",
"deliveryRate": 79.0,
"estimatedReach": 395000,
"recommendation": "Safe to proceed"
}
// 5. User approves campaign
POST /api/campaigns/{campaignId}/approve
// 6. Backend creates production job
await bullQueue.add('send-campaign', {
...campaignData,
jobId: `production-${campaignId}`,
isDryRun: false,
limit: 500000, // Full send
});
// 7. Worker processes production job
// (Same function, isDryRun = false)
// → Actual notifications sent
// 8. Compare results
GET /api/campaigns/{campaignId}/results
{
"predicted": { "deliveryRate": 79.0, "reach": 395000 },
"actual": { "deliveryRate": 80.2, "reach": 401000 },
"accuracy": "98.5%"
}
Key takeaways
1. Dry-run is a game-changer for large campaigns
- Validates 500K tokens in 2 minutes (vs 80 minutes real send)
- Predicts delivery rate with 95%+ accuracy
- Zero notifications sent to users
2. Error classification is critical
- Not all failures are permanent
-
invalid_token= dead token (remove from DB) -
temporary= retry might work (include in delivery rate)
3. Sample intelligently
- Random sampling prevents bias
- 1-2% sample size is optimal
- Larger samples have diminishing returns
4. Automate cleanup
- Use validation results to mark invalid tokens
- Future campaigns automatically skip them
- Database quality improves over time
5. Validate before high-value sends
- Product launches
- Time-sensitive campaigns
- New audience segments
If you're sending push notifications at scale, dry-run validation is one of the highest-ROI features you can implement. 2 minutes of testing saves hours of wasted processing and provides accurate reach predictions.
In our next post, I'll show you how we measure click-through rates by tracking which users actually open the notifications we send. Because delivery is only half the story—engagement is what matters.
SOCIAL SHARE CARD GENERATOR