When was the last time you used a chat service and noticed messages arriving out of order? It’s likely that you can’t think of one. Getting messages to arrive in the right order is more or less job number one of a chat service.
Sounds simple. But scale this to millions of users across a global network and it becomes a complex distributed computing problem. Even at smaller scales, message ordering isn’t guaranteed once you go beyond a simple, direct connection between two users. Once you have more than a handful of clients and servers, preserving message order becomes a complex engineering problem.
In fact, it’s such a tough challenge that most chat services and realtime platforms simply hand the problem over to you, as the application developer.
In this article, we’re going to look at the reasons why message ordering is harder than it seems, the impact of scale, and some practical advice on how to address the challenge if you’re building your own messaging platform.
Why message ordering is a challenge
Why is it so hard to keep chat messages in order? There are two main issues:
Coordinating multiple chat users: Direct chats between two people are relatively easy because there’s not much to go wrong. But if you’re connecting multiple people through a backend then it becomes a lot more challenging to keep messages in the right order.
Scale and complexity: And, similarly, the more users, traffic, and backend servers you have, the more opportunities there are for messages to arrive out of sync.
So, let’s start by looking at three basic chat architectures and then what happens when your chat application scales.
Direct chat between two people
Think of a simple chat box on a website. Both sides of the conversation connect directly to one another, perhaps using a WebSocket connection.
So, what issues might arise?
Multi-hop network delays: Messages now travel two hops—from the sender to the server, and then from the server to the recipient. This additional step introduces latency on its own, while also increasing the chance of network interruptions, such as temporary congestion on the open internet.
Processing lag on the server: At times of high load, the server might receive messages faster than it can process them. This can happen in bursty traffic scenarios or when other services are also competing for resources. As a result, a message sent later might be processed and sent out before an earlier one, purely because it reaches an idle processor first.
Asynchronous handling: One way to handle load more efficiently is to introduce multiple parallel systems and have them handle messages asynchronously. In this model, messages are queued and handled by multiple parallel processes, which can lead to out-of-order delivery if processes complete at different speeds.
Even with just two people chatting, introducing an intermediary backend increases the likelihood of messages arriving out of order. However, the flexibility and control it provides often outweigh the added complexity. For multi-user chats, though, there’s really no practical alternative to a backend; a backend is essential to manage and coordinate messages reliably across participants.
Multi-user conversations
Most chat applications enable multiple people to converse either in one-on-one chats or larger group (multi-user) conversations. In these setups, ensuring that messages arrive in the right order becomes a major challenge. The system isn’t just managing the order between two people; it now has to coordinate messages across multiple participants, each with unique network conditions and potential delays.
.
In essence, scaling-up is a continuation of what you’ve almost certainly already been doing. That is, adding more chat processes on a single server to create multiple, parallel message queues. That in itself opens the chance of messages arriving in the wrong order because one queue might process faster than the other. But the real complexity starts when you message queues running across multiple backend servers.
There are a couple of ways to do this. One is to have dedicated servers for ingestion, processing, and delivery. However, a more common horizontal scaling model would be to have multiple servers with identical capabilities. This way, you can spin up new instances of the chat backend when demand increases. And when traffic slows, removing instances will save you from paying for capacity you don’t need at that moment.
The trade-off is that this greatly increases the chance of messages arriving out of order.
Imagine you have five people in a group chat for work. Bill, Maryam, and Paula are working in an office on the US east coast. Their chat clients connect to a backend server in New York. Don and Shirley are working from home in Europe and their chat clients connect to a server in London.
Coordinating messages across those servers might look something like this:
- Bill’s chat client is connected to the New York server. He sends a message with a large image attached. The New York server processes the message.
- Maryam’s chat client is also connected to the New York server. She sends a message to the group chat and the New York server adds it to the queue behind Bill’s large message.
- Don is connected to the London server. He sends a message at the same time as Maryam. But because Maryam’s message is stuck behind Bill’s on the New York server, and Bill’s message is still processing, Don’s message arrives with the other chat participants before everyone else’s.
and , the protocol is effectively unavailable in web browsers for most practical applications. And .
What does that mean?
WebSocket guarantees ordered delivery only for messages sent and received over the same, uninterrupted connection. If you have multiple connections (due to scale, reconnections, or multi-device usage), you need to ensure global message order at the application level. The alternative is to choose a realtime or chat platform that handles message ordering on your behalf.
Adding unique identifiers and sequencing
Unique identifiers, like sequence numbers or timestamps, are critical for keeping messages in order, but they’re tricky to manage across distributed servers at scale. A centralized sequence generator can become a bottleneck, adding latency as nodes wait for the latest sequence number.
In a fully distributed setup, logical clocks or decentralized sequencing bring their own risks—like clock drift and inconsistency—especially during network partitions. To maintain order, you’ll often need consensus mechanisms, such as Raft or Paxos, to keep all nodes aligned, but this adds complexity and coordination overhead.
To address these challenges, there are a few practical approaches for managing unique identifiers at scale. Each has its trade-offs in terms of latency, complexity, and consistency, but they can help ensure ordered message delivery across distributed systems:
System-wide sequence numbers: Using a global sequence number enforces order but can introduce latency, as servers must sync. Logical clocks or a centralized sequence generator (e.g., with Raft for leader election) can mitigate risks but add complexity.
Timestamps with client-side IDs: Timestamps, combined with a unique sender ID, provide order but need buffer adjustments to account for clock drift. Synchronizing clocks using NTP or setting buffers can help avoid minor inconsistencies.
Combined sequencing: Using both sequence numbers and timestamps provides a more precise order, though it requires complex coordination, especially under high traffic.
Buffering, windowing, and watermarking
To keep messages ordered without adding lag, large-scale chat systems often borrow techniques from streaming data: buffering, windowing, and watermarking. Here’s how each approach helps manage the challenges of maintaining order as the system scales:
Buffering and reordering: Incoming messages are temporarily stored on the client or server until they can be shown in sequence. Sequence numbers help detect gaps. If a message arrives out of order, later messages are buffered until the gap is filled. This preserves order but can cause delays if not tuned carefully for high traffic, where buffering needs to scale dynamically with load.
Windowing: Windowing holds messages within defined time intervals, reordering them before display. Once the window closes, messages are released together, which reduces the risk of out-of-order display. At scale, adjusting window size based on traffic conditions is essential to keep the right balance between responsiveness and message order.
Watermarking: Watermarking sets a time limit for holding messages in the buffer. Once that limit is reached, all buffered messages are displayed—even if slightly out of order—keeping chat responsive and preventing long delays that result from waiting on older messages.
Together, these techniques help manage reordering and timing constraints, balancing speed and accuracy in message delivery as the system scales. By fine-tuning these approaches, you can maintain a responsive, realtime experience for users, even under high traffic and fluctuating loads.
Concurrency and load distribution
Concurrency is both an asset and a challenge in a large-scale chat system. While it boosts capacity and resilience, it also increases the risk of messages arriving out of order. To tackle this, there are a couple of core techniques to consider:
Consistent routing: Direct all messages from a user or group to a specific processing node or queue, minimizing the chance of out-of-sequence processing even during peak loads. As the system scales, consistent routing keeps ordering predictable, though it requires careful management to prevent bottlenecks at individual nodes.
Partitioning strategies: Techniques like consistent hashing distribute load across nodes while keeping message order intact. By routing messages from the same user or group to the same partition, consistent hashing helps prevent overloading any one node, maintaining order across distributed systems. At scale, though, partitions need tuning to adapt to shifting loads without sacrificing order.
These methods draw on advanced distributed computing techniques. As concurrency grows, maintaining message order requires effective load distribution, consistent state management, and fault tolerance across nodes. While consistent routing and partitioning help streamline ordering, scaling up demands careful coordination to avoid reordering issues.
Handling retries and deduplication
While ordering is crucial, handling duplicate messages is equally important especially where you have multiple concurrent processes. Deduplication logic ensures that each message is processed only once, even if it’s received multiple times due to retries.
Deduplication logic: Using unique identifiers for each message helps prevent duplicates from being processed out of sequence. By tracking these IDs across distributed nodes, the system can detect and ignore duplicates, ensuring retried messages don’t break the intended order or result in multiple displays of the same message.
Idempotent message processing: Idempotency ensures that processing the same message multiple times has the same effect as processing it once. For example, if a message updates a chat or changes a user’s status, the logic checks whether the update has already been applied before proceeding. This approach prevents duplicate messages from causing unintended changes or misordering.
Together, deduplication and idempotent processing ensure that messages are handled reliably and in order, even under high traffic and frequent retries. These techniques help maintain consistency across the system, keeping the chat experience smooth and predictable as demand scales.
Other architectural considerations
Beyond ordering and concurrency, your chat system’s architecture will also need to handle:
Delivery semantics at scale: Deciding between at-most-once, at-least-once, and exactly-once delivery becomes more complex as you handle more users and messages. Exactly-once delivery ensures each message arrives only once and in the correct order, even during retries. Achieving this at scale requires idempotent processing and stateful components, adding complexity and resource overhead. As your system grows, you need to balance avoiding duplicates against increased latency and processing costs.
Message durability under heavy load: Durability in case of failures is straightforward with low traffic. Scale-up and you could find persistent storage becomes a bottleneck. Your system must efficiently write to durable storage and recover messages after failures, all while maintaining realtime performance for millions of users.
Creating a resilient, responsive chat architecture that performs well at scale involves significant architectural and engineering effort. The challenges evolve and multiply as your traffic and user base grow, raising the question: Is building and maintaining such a complex system in-house the best use of your resources?
Should you build or buy to guarantee message ordering?
Building a chat system that reliably maintains message order at scale is no small feat. And, like other parts of your application infrastructure, it doesn’t have to be built from scratch.
Instead, you essentially have three main options. Each comes with different characteristics in terms of time, resources, and control. Let’s explore them so you can evaluate which approach aligns best with your goals and capabilities.
Build in-house: Building a chat system in-house requires a significant, ongoing commitment. It’s resource-intensive and demands specialized expertise for design, implementation, and maintenance. While building in-house offers maximum control and customization, it often diverts resources from your core application needs. Even if chat is central to your application, consider whether you can deliver features beyond what existing platforms offer. This option can make sense if you need highly specialized features or tight control, but be prepared for the dedicated team and budget required to support it long-term.
Use a chat service: Using a dedicated chat service can save your team’s time and resources, letting you focus on core functionality rather than chat-specific challenges. But the flipside is that an out-of-the-box solution might require that you modify your requirements to meet the service’s approach. And you could find that customization or advanced features are harder to achieve. Crucially, though, the biggest downside of most chat services is that they don’t guarantee message ordering. As a result, you'll need a plan to handle out-of-order messages when they arrive on the client.
Build with a realtime platform: Working with a realtime platform-as-a-service (PaaS), like Ably, lets you offload the complexities of scaling, message ordering, and maintaining low latency while retaining the flexibility to build exactly the solution you need. This reduces the long-term load both on your team and infrastructure, helping you get features to users faster. Beware that, unlike Ably, not all realtime PaaS providers guarantee message ordering. When selecting a provider, consider whether you'd need to take on the engineering and maintenance costs of handling message ordering yourself.
Ultimately, each option requires balancing control and customization against resource limitations. Choosing the best approach lies in weighing what you can manage for the long-term in-house against freeing-up your team’s time by using an external service. If you’re still not sure which is the right approach for you, take a look at our .
SOCIAL SHARE CARD GENERATOR