Design a Notification System - Push, Email, and SMS at Billion Scale

Modern applications send billions of notifications daily across multiple channels. Designing this system teaches you about message queues, third-party integrations, and handling failures gracefully at scale.

Step 1 - Requirements Gathering

Functional Requirements

• Support multiple channels: push notifications, email, SMS, and in-app
• Users can configure notification preferences and opt out per channel
• Templated messages with personalisation (e.g., "Hi , your order shipped")
• Support for scheduled and immediate notifications

Non-Functional Requirements

• Scalability - handle 10 million+ notifications per day
• Reliability - no lost notifications; at-least-once delivery
• Low latency - urgent notifications delivered within seconds
• Extensible - easy to add new channels (WhatsApp, Slack)

Step 2 - High-Level Architecture

Caller Services → Notification Service → Validation + Preference Check
                                              ↓
                                    Message Queue (Kafka)
                                    ↓       ↓       ↓       ↓
                                 Push    Email    SMS    In-App
                                Worker   Worker  Worker  Worker
                                  ↓        ↓       ↓       ↓
                                 FCM   SendGrid  Twilio  WebSocket

Core Components

1. Notification Service - validates requests, checks user preferences, renders templates, and enqueues messages
2. Message Queues - separate queues per channel for independent scaling and isolation
3. Delivery Workers - channel-specific workers that handle third-party API calls
4. Template Service - stores and renders notification templates with variable substitution

Step 3 - Deep Dive: Key Components

User Preferences and Opt-Out

Store preferences per user and per notification type:

user_preferences:
  user_id: "u123"
  channels:
    email: { enabled: true, quiet_hours: "22:00-08:00" }
    push: { enabled: true }
    sms: { enabled: false }
  categories:
    marketing: { enabled: false }
    security: { enabled: true, override_quiet_hours: true }

The notification service checks preferences before enqueuing - never send what the user doesn't want.

Priority Queues

Not all notifications are equal. Use priority levels:

• Critical - security alerts, OTP codes → processed immediately
• High - order confirmations, payment receipts → seconds
• Medium - social interactions, comments → minutes acceptable
• Low - marketing, recommendations → can be batched

Step 4 - Reliability and Failure Handling

Deduplication

Prevent duplicate notifications using an idempotency key. Before processing, check if the notification ID exists in a deduplication store (Redis with TTL):

Key: "dedup:{notification_id}" → TTL: 24 hours
If key exists → skip (already sent)
If not → process and set key

Retry with Exponential Backoff

Third-party services fail. Implement retries with increasing delays:

Attempt 1: immediate
Attempt 2: wait 1 second
Attempt 3: wait 4 seconds
Attempt 4: wait 16 seconds
Attempt 5: move to dead-letter queue

Rate Limiting Per User

Protect users from notification fatigue:

• Maximum 3 push notifications per hour per user
• Maximum 1 marketing email per day
• Security notifications bypass rate limits

Step 5 - Third-Party Integration

| Channel | Provider | Protocol | Challenges | |---------|----------|----------|------------| | Push (iOS) | APNs | HTTP/2 | Token management, silent push limits | | Push (Android) | FCM | HTTP | Topic-based broadcasting | | Email | SendGrid/SES | SMTP/API | Deliverability, spam filters | | SMS | Twilio | REST API | Cost per message, regional compliance | | In-App | WebSocket | WS | Connection management, offline buffering |

Analytics Pipeline

Track the notification lifecycle for every message:

• Created → Queued → Sent → Delivered → Opened → Clicked

Store events in a time-series database or event stream for dashboard reporting: delivery rates, open rates, and click-through rates per channel and notification type.

Step 6 - Bottlenecks and Optimisations

| Bottleneck | Solution | |-----------|----------| | Third-party rate limits | Queue with controlled throughput per provider | | Template rendering at scale | Pre-render and cache common templates | | Preference lookups per message | Cache user preferences in Redis | | Peak traffic spikes | Auto-scale workers based on queue depth | | Cross-region delivery | Deploy workers close to provider endpoints |

📚 Further Reading

• Notification System - System Design Interview Vol. 1, Ch. 10 - Alex Xu's comprehensive notification system design
• How Airbnb Built a Notification Platform - real-world multi-channel notification architecture
• Firebase Cloud Messaging Docs - understanding push notification delivery mechanics