Backpressure explained: where do the extra messages go?

Your server can handle a hundred messages a second. A thousand just showed up. When the producer is faster than the consumer, the other 900 have to go somewhere, and every messaging system has to decide where.

This is not an edge case. Traffic is bursty: a push notification goes out, a match kicks off, a batch job finishes and dumps its output. Any system that connects a producer to a consumer will, at some point, have the producer outrun it. The question is what happens then, and whether you chose it on purpose.

The simplest answer is to throw the extra messages away. For some data that is completely fine. If a temperature sensor reports every second and you lose a few readings, the next one arrives a second later and says nearly the same thing.

The danger is that dropping is silent. You can lose a large share of your messages while your dashboards still look healthy, because the dashboards are built from the messages that made it. If you drop, count what you drop and alert on it.

The next answer is a queue. Extra messages wait in a buffer and the consumer works through them when it can. For a short spike this is ideal: the burst is absorbed, nothing is lost, and the queue drains when traffic calms down.

But a buffer only delays the question. If the producer is faster on average, not just during a spike, the queue grows without limit. An in-memory queue eventually runs the process out of memory and it crashes, taking everything in the buffer with it. A queue written to disk lasts longer but slows down as it grows. Ask: what happens when the buffer is full?

The third answer is backpressure, also called flow control. When the buffer is full, the consumer tells the producer to wait. The producer blocks, or slows down, until there is room again. Nothing is lost and memory stays bounded; the cost is that the slowness travels upstream to whoever is producing.

You already use this every day. TCP does exactly this: the receiver advertises how much buffer space it has left, and the sender is not allowed to send more than that. Unix pipes do it too; if the reading process is slow, the writing process blocks.

queue = asyncio.Queue(maxsize=1000)  # bounded buffer

async def producer():
    while True:
        msg = await next_message()
        await queue.put(msg)  # waits when full: that wait IS the backpressure

And there is always the option of fixing the mismatch directly: add more consumers and split the work between them.

Pick by what one lost message costs you. A dropped sensor reading costs nothing. A dropped click in an analytics stream costs a rounding error. A dropped payment, order or chat message costs a real customer. Most real systems mix strategies: buffer to absorb spikes, push back when the buffer fills, and drop only the data that is explicitly allowed to be lossy.

Adding consumers sounds like the easy fix, but it changes how a traditional message broker behaves. Brokers in the AMQP/JMS style hand each message to one of several consumers, and only delete it once that consumer acknowledges it.

If a consumer crashes halfway through message 3, the broker never gets the acknowledgment. So it redelivers message 3 to another consumer, which by now may already be processing message 4. Message 3 ends up behind message 4. With load balancing plus redelivery, order is gone. For many workloads that's fine. For “create account” followed by “update account”, it isn't.

Log-based brokers like Apache Kafka take a different approach. A topic is an append-only log on disk, split into partitions. Messages within a partition are strictly ordered. Instead of tracking every message, each partition is read by exactly one consumer in a group, and that consumer just remembers how far it has read: the offset.

If a consumer crashes, another one takes over its partitions and starts reading from the last committed offset. Order inside a partition is preserved, and nothing is deleted when it's read, so you can even rewind and reprocess old messages. To keep related messages in order (all events for one account, say), give them the same key so they land in the same partition.

Because messages stay on disk, a slow consumer doesn't fill the broker's memory. It just falls behind; how far behind is its lag, which is the number you monitor.

Consumers commit offsets periodically, not after every message. If a consumer crashes after processing messages 101 to 110 but before committing offset 110, its replacement starts again from 100. Those ten messages are processed twice.

So make processing idempotent: running it twice has the same effect as running it once. A reliable way is to store the offset in the same database transaction as the result, and skip anything at or below the stored offset.

BEGIN;
  UPDATE balances SET amount = amount + 25 WHERE account = 'A';
  UPDATE consumer_offsets SET last_offset = 110
    WHERE partition = 3 AND last_offset < 110;  -- 0 rows? already done, roll back
COMMIT;