The Difference Between a Queue and a Stream

A worker falls behind overnight. By morning the queue is full, the dashboard is stale, and someone asks whether the system can just replay yesterday’s messages once the fix goes out. That question tells you what you actually built.

If those messages were work assignments, replay might be the wrong move. The email was either sent or it wasn’t. The image was either resized or it wasn’t. Running the same work again just creates duplicates. But if the messages were events, replay might be the whole point, because the original facts still matter to a different consumer, a fixed consumer, or a new consumer that needs to read them later.

Queues and streams both move messages between processes, and that overlap is what makes them easy to confuse. The difference shows up when something goes wrong.

A queue is a work assignment

A queue is built around ownership. One message goes in, one worker takes it, the worker finishes the job, and the message leaves the queue. That shape fits background work, because the point isn’t to preserve history. It’s to get work done: send the email, resize the image, charge the card, recompute the report.

Once the work is finished, keeping the original message around usually just adds noise. The system cares that the job completed, failed, retried, or landed in a dead letter queue. It doesn’t need every worker to read the same job independently. That’s the contract.

import json
import redis

r = redis.Redis()

def enqueue_email(order_id: str) -> None:
    r.lpush("email_jobs", json.dumps({"order_id": order_id}))

def worker() -> None:
    _, payload = r.brpop("email_jobs")
    job = json.loads(payload)
    send_order_email(job["order_id"])

This is intentionally small. No replay, no consumer offset, no fan-out. A worker removes a job and does the work, and that simplicity is the feature.

A stream keeps the event log

A stream is built around history. Messages go in, but they don’t disappear just because one consumer read them. Consumers track where they are in the log, so one service can read from the beginning, another can read only new messages, and a third can stop for an hour, catch up later, and still see the same ordered sequence. The stream isn’t a pile of tasks. It’s a record of things that happened.

from redis import Redis

r = Redis()

def publish_order(order_id: str, total: int) -> None:
    r.xadd("orders", {"order_id": order_id, "total": total})

def read_orders(last_id: str = "0-0") -> str:
    events = r.xread({"orders": last_id}, count=10, block=1000)
    for _, messages in events:
        for event_id, fields in messages:
            update_projection(fields)
            last_id = event_id
    return last_id

The important part isn’t the API, it’s the cursor. The consumer decides how far it has read, and the stream still has the events. That changes the failure mode.

Replay is the first real difference

If a queue consumer has a bug, the message may already be gone. You can retry failed jobs if the queue and worker acknowledge messages carefully, send poison messages to a dead letter queue, and make handlers idempotent so duplicate delivery doesn’t cause damage. But a queue is still designed around completion.

A stream gives you a different move: fix the consumer and read the events again. That matters when the consumer builds derived state. Search indexes, reporting tables, recommendations, fraud features, and customer timelines all depend on historical events being interpreted correctly, and if the interpretation changes, replay turns old facts into new projections. What changed, the event or the meaning of the event? That question is much easier to answer when the original event still exists.

Consumer lag means different things

Queues and streams both have lag, but it means different things. Queue lag means work is waiting. If email jobs pile up, customers don’t get email. If image jobs pile up, thumbnails arrive late. The pressure is operational and immediate. Stream lag means a reader is behind the log, and that may be completely fine. A batch analytics consumer can run five minutes late without hurting anyone. A fraud consumer probably can’t.

So the same metric points to different problems. A queue asks how fast workers can drain the backlog. A stream asks how far behind each reader is, and whether that reader’s delay actually matters. That second question drags product context into the infrastructure, because a delayed warehouse load, a delayed notification, and a delayed risk decision shouldn’t share the same urgency just because they all happen to read events.

Fan-out belongs naturally to streams

Queues can support multiple consumers, but the default mental model is competing workers. Ten workers read from the same queue so the work finishes faster. They divide the messages up between them, they don’t all receive every message. Streams make independent consumption natural instead. The billing service reads order events, the analytics pipeline reads the same order events, the email service reads them too, and each one owns its offset and its own failure handling.

That freedom has a cost. The producer now has to treat the event as a public contract. Field names, event meanings, ordering expectations, retention windows, backwards compatibility: none of those are local choices anymore. They affect every reader, including readers the producer team may not even know about. A queue keeps less history around. A stream creates more memory, and memory creates responsibility.

Ordering is not free

People often reach for streams when they need ordering. That can be reasonable, but ordering always comes with a boundary. Kafka orders records within a partition. Redis streams preserve order within a stream. A database table ordered by an incrementing ID gives you one kind of order, but not necessarily the order the business cares about. Order by what, exactly? Order by write time isn’t always order by payment time. Order by event arrival isn’t always order by user action. Order within one customer’s events isn’t the same as order across all customers.

Queues have the same problem, they just expose it differently. Two workers can process jobs out of order unless the queue design prevents it. That’s fine for image resizing and a disaster for account balance updates. The hard part was never picking the tool with ordering. It’s defining the ordering contract the system actually needs.

Retention changes debugging

With a queue, debugging usually starts from logs and traces. The job ran, the worker crashed, the retry happened, the dead letter queue has the payload. The question is almost always “why did this unit of work fail?” With a stream, debugging can start from the event history itself. Which event introduced the bad state? Did every consumer see it, or did one skip it? Did a schema change make an older consumer misread a newer event?

That changes the operating model. Teams get more evidence, but they inherit more obligations along with it. Retention has a price: storage costs money, long-lived events may hold data that later has to be deleted, and old event formats keep old assumptions alive. Keeping history isn’t automatically virtuous. It’s a product and compliance decision wearing an infrastructure jacket.

Most systems need both

A stream can feed a queue. An OrderPlaced event lands in a stream because several systems care that it happened. The email service reads that event and creates a queue job to send the actual email. If the email provider is down, retrying the email job belongs in the queue. Replaying the original order event belongs in the stream. Those are two different problems: one is distribution of facts, the other is execution of work.

Mixing them makes a system harder to reason about. If every task becomes an event, consumers start depending on implementation details. If every event becomes a task, new readers can’t build their own view of history without changing the producer.

The boundary is usually simple:

• Use a queue when the message represents work to be done once.
• Use a stream when the message represents something that happened.
• Use a stream when independent consumers need their own pace.
• Use a queue when success means a worker finished the job.
• Use a stream when replay would fix a real future problem.

That last point matters most. Replay sounds attractive right up until someone has to design event schemas, retention, offsets, backfills, and consumer compatibility. If nobody is ever going to replay the data, a stream is just machinery you don’t need.

The tool choice is a contract choice

Redis lists, Celery, SQS, RabbitMQ, Kafka, Redis Streams, Kinesis, and database-backed outboxes can all move messages between processes. That category is far too broad to choose from. The useful question is what contract the message should have once it leaves the producer. Can one worker own it? Can it disappear after success? Do several consumers need to read it independently? Will a future version of the system need to rebuild state from it?

A queue answers those questions one way, a stream answers them another, and the mistake is treating that as an implementation detail. By the time a message broker sits between two parts of a system, it has already become part of the product’s memory. The only real question is whether that memory should be short, private, and task-shaped, or durable enough for other systems to build on.