Building Resilient Real-Time Systems: WebSockets, Redis, and Strategies for High Availability

Originally published on tamiz.pro.

Real-time systems are at the heart of modern interactive applications, from chat platforms to collaborative editing tools and financial dashboards. Delivering a seamless, low-latency experience while ensuring high availability and fault tolerance presents significant architectural challenges. This deep-dive explores how WebSockets, for persistent client-server communication, and Redis, for state management and pub/sub, can be combined with strategic design patterns to build resilient real-time systems.

The Core Challenge of Real-Time Resilience

The primary challenge in real-time systems is maintaining continuous connectivity and data flow despite inevitable network issues, server failures, or application restarts. A single point of failure can disrupt numerous active user sessions, leading to a poor user experience. Resilience, in this context, means the system's ability to recover gracefully from failures and continue operating, even if in a degraded state.

WebSockets: The Foundation for Real-Time Communication

WebSockets provide a full-duplex communication channel over a single TCP connection, enabling persistent, low-latency message exchange between client and server. Unlike traditional HTTP requests, WebSockets keep the connection open, eliminating the overhead of connection setup for each message. However, managing a large number of concurrent WebSocket connections and ensuring their availability across a distributed system requires careful design.

WebSocket Server Scalability and Load Balancing

Directly load balancing WebSocket connections using standard HTTP load balancers can be tricky because WebSockets are long-lived. Sticky sessions are often employed to ensure a client consistently connects to the same backend server. While this works, it can lead to uneven server load and complicates server replacement during failures.

A more robust approach involves a dedicated WebSocket gateway layer that can manage connections and route messages to appropriate backend services. This gateway can be stateless itself, relying on a shared state layer (like Redis) to manage session information.

Redis: The Backbone for State, Pub/Sub, and Persistence

Redis is an in-memory data store known for its speed and versatile data structures. It becomes indispensable in real-time architectures for several reasons:

1. Pub/Sub Messaging: Redis's Publish/Subscribe (Pub/Sub) mechanism allows multiple WebSocket servers to communicate with each other and broadcast messages to relevant clients without direct server-to-server connections. A message published to a Redis channel is received by all subscribers to that channel.


2. Shared State Management: When a client connects to a WebSocket server, its session data (e.g., user ID, active channels) can be stored in Redis. This allows any WebSocket server in the cluster to handle messages for that client, improving fault tolerance and enabling horizontal scaling.


3. Presence Management: Redis can track which users are currently online and connected to which WebSocket server, crucial for features like online user lists or targeted notifications.


4. Message Queues: For scenarios requiring message persistence or processing guarantees, Redis can act as a lightweight message queue (e.g., using LPUSH/RPOP on lists or Streams).

Architectural Patterns for Resilience

Combining WebSockets and Redis, several architectural patterns emerge for building resilient real-time systems.

1. The Pub/Sub Broker Pattern

This is a fundamental pattern for distributing messages across multiple WebSocket servers.

• Client-to-Server: A client connects to a WebSocket server (WS1).


• Server-to-Redis: When WS1 receives a message that needs to be broadcast or sent to another user, it publishes the message to a specific Redis channel (e.g., chat:room:general, user:123:inbox).


• Redis-to-Servers: All WebSocket servers subscribe to relevant Redis channels. WS1 and WS2 both receive the message from Redis.


• Server-to-Clients: WS1 and WS2 (and any other servers) then check if they have active connections for the target clients of that message. If WS2 has a connection for the target client, it forwards the message over the WebSocket.
  

graph TD

    C1[Client 1] --> WS1[WebSocket Server 1]

    C2[Client 2] --> WS2[WebSocket Server 2]

    WS1 -- Publishes Message --> Redis[Redis Pub/Sub]

    WS2 -- Publishes Message --> Redis

    Redis -- Subscribes to Channel --> WS1

    Redis -- Subscribes to Channel --> WS2

    WS1 -- Forwards to C1 --> C1

    WS2 -- Forwards to C2 --> C2

Benefits: Decouples WebSocket servers, allowing independent scaling and fault tolerance. If WS1 fails, clients connected to WS2 are unaffected, and new clients can connect to any available server.

2. Session Management with Redis

To make the WebSocket servers truly stateless and interchangeable, session information must be externalized.

• When a client connects to a WebSocket server, the server stores the mapping of sessionId -> {userId, wsServerId} in Redis.


• The wsServerId is crucial for targeted messaging. Each WebSocket server needs a unique identifier.


• When a message needs to be sent to a specific userId, the system looks up their sessionId in Redis. If found, it retrieves the wsServerId and publishes the message to a Redis channel specific to that server (e.g., server:ws_server_id_X:inbox).


• The target WebSocket server (WS_server_id_X) receives the message from its dedicated channel and forwards it to the client.
  

// Example: Storing session info in Redis on connection

const userId = 'user_abc';

const sessionId = 'some_unique_session_id';

const wsServerId = 'server_instance_1'; // Unique ID for this WS server instance



redisClient.hset(`ws:sessions:${userId}`, sessionId, wsServerId, (err, reply) => {

  if (err) console.error('Failed to store session:', err);

  console.log('Session stored:', reply);

});



// Example: Retrieving session for targeted message

const targetUserId = 'user_abc';

redisClient.hgetall(`ws:sessions:${targetUserId}`, (err, sessions) => {

  if (err) {

    console.error('Failed to retrieve sessions:', err);

    return;

  }

  for (const sessionId in sessions) {

    const targetWsServerId = sessions[sessionId];

    // Publish message to the specific server's inbox channel

    redisClient.publish(`ws:server:${targetWsServerId}:inbox`, JSON.stringify({

      type: 'private_message',

      sessionId: sessionId,

      payload: 'Hello from another user!'

    }));

  }

});

Benefits: Allows any WebSocket server to handle a client's initial connection or subsequent reconnections. If a server fails, clients can reconnect to any other available server, and their session state can be retrieved from Redis.

3. Connection Heartbeats and Reconnection Strategies

Even with robust server-side architecture, network instability can cause client-side WebSocket connections to drop. Implementing heartbeats and intelligent reconnection logic is critical.

• Client-side Heartbeats: Clients periodically send a small 'ping' message to the server. If a 'pong' isn't received within a timeout, the client assumes the connection is dead and attempts to reconnect.


• Server-side Heartbeats: Servers can also send pings. If a client doesn't respond to a ping, the server can gracefully close the connection and remove its session from Redis.


• Exponential Backoff: When a client reconnects, it should use an exponential backoff strategy to avoid overwhelming the server with repeated connection attempts during an outage.


• Last Will and Testament (LWT): While not standard WebSocket, protocols like MQTT over WebSockets offer LWT features, where the server can broadcast a message if a client disconnects unexpectedly, useful for presence.

4. Redis High Availability (HA)

Redis itself can be a single point of failure. To achieve true resilience, Redis needs to be highly available.

• Redis Sentinel: Provides automatic failover for Redis instances. Sentinel monitors Redis master and replica instances, detects failures, and promotes a replica to master if the current master becomes unavailable. Clients can connect to Sentinels to discover the current master's address.


• Redis Cluster: For larger-scale deployments, Redis Cluster shards data across multiple master nodes, providing both high availability and horizontal scaling of data. Each master node has its own replicas for failover.
  

# Example Redis Sentinel configuration (sentinel.conf)

port 26379



sentinel monitor mymaster 127.0.0.1 6379 2

sentinel down-after-milliseconds mymaster 5000

sentinel failover-timeout mymaster 60000

sentinel parallel-syncs mymaster 1

Benefits: Ensures that the critical shared state and Pub/Sub mechanism of Redis remain operational even if individual Redis instances fail.

Considerations for Message Delivery Guarantees

While WebSockets and Redis Pub/Sub offer speed, they typically provide