Push Notification System

What is a Notification System?

A notification system routes and delivers messages to users across multiple channels — push (iOS/Android), email, and SMS. It must handle high fan-out (one event triggers notifications to millions), user preferences (opt-out, quiet hours), and reliable delivery.

Key operations:

• Send: Accept notification request, route to correct channel(s)
• Fan-out: One event → notifications to millions of users
• Deliver: Call third-party providers (FCM, APNs, SendGrid, Twilio)
• Track: Delivery status, read receipts

Functional

• Send push, email, and SMS notifications
• Respect user preferences (opt-outs, quiet hours, frequency caps)
• Support immediate and scheduled notifications
• Track delivery status

Non-Functional

• Scale: 10M notifications/day = ~120/sec average, 10K/sec peak (flash sales)
• Latency: push < 1s, email < 30s, SMS < 10s
• At-least-once delivery
• 99.9% availability

Capacity

• 10M/day × avg 1KB payload = 10GB/day through the system
• Device token DB: 1B users × 2 devices × 100B = ~200GB

The Fan-out Problem

A single event (e.g. a flash sale) can trigger notifications to 50M users in seconds. You can't do this synchronously in the API call — it would timeout.

Key insight: Decouple with Kafka. Accept the request fast, fan out asynchronously.

Channel Isolation

Each channel (push, email, SMS) has different:

• Throughput limits (FCM: fast, SMS: slow + expensive)
• Failure modes (APNs rejects invalid tokens)
• Retry semantics

Kafka topics per channel let workers scale independently.

User Preferences are Critical

Always check opt-outs and quiet hours before sending. Nothing damages trust more than spamming a user who opted out.

Flow

1. Caller sends POST /notify with (user_id[], template, channel, data)
2. Notification Service:
   • Fetches user preferences + device tokens from User Prefs DB
   • Filters out opted-out users
   • Publishes one message per channel per user to Kafka
3. Channel workers consume from Kafka:
   • Push Worker: calls FCM (Android) or APNs (iOS)
   • Email Worker: calls SendGrid / SES
   • SMS Worker: calls Twilio
4. On failure: Kafka consumer retries with exponential backoff

Device Token Management

• Register device token on app install
• FCM/APNs return invalid token errors → clean up DB
• One user can have multiple devices

Priority Queues

Not all notifications are equal. Critical alerts (OTP, fraud) should skip the queue:

• Kafka topic: notifications.high-priority → dedicated fast worker
• Kafka topic: notifications.low-priority → batched worker

Rate Limiting

FCM has per-project send limits. SMS is expensive. Add a per-user frequency cap:

• Max 3 push/hour, 1 email/day, 1 SMS/week for marketing
• No cap for transactional (OTP, order confirmation)

Idempotency

Event bus retries can cause duplicate sends. Use a notification_id as idempotency key:

1. Check Redis set: has notification_id been processed?
2. If yes → skip
3. If no → send → add to Redis set with TTL

What Interviewers Look For

1. Fan-out via Kafka — not synchronous, not a single thread
2. Channel separation — push/email/SMS have different workers
3. User preferences — opt-out and quiet hours checked before send
4. Idempotency — dedup on notification_id prevents double sends
5. Priority queues — OTP/fraud alerts skip marketing queue
6. Device token cleanup — remove stale tokens on send failure

Common Mistakes

• Synchronous delivery in the API handler
• Single queue for all channels
• Not filtering opt-outs before fan-out
• Ignoring idempotency on retries