The layer that decouples who SENDS work from who DOES it — a line of messages where a producer drops tasks and consumers pick them up when ready.
← / → to navigate · EN/VI switch below
A first-in-first-out line of small messages, standing between the part of the app that creates work and the part that does it.
Messages (usually small JSON strings) line up: first in, first out. The queue is just storage — a waiting line.
A producer PUBLISHES a message; a consumer CONSUMES it later. They never call each other directly.
The sender doesn't know who handles the work, when it happens, or how many handlers there are.
Work can pile up in the queue while consumers catch up — a shock absorber for your system.
The same 'user places an order' moment, with and without a queue in between.
User clicks 'order' → your app sends the email, calls the payment API, renders the invoice — all inside the same request. The user stares at a spinner for 8 seconds.
Slow — and if the payment API hiccups, the whole request fails.
User clicks 'order' → the app drops ONE message on the queue, replies in 50 ms, and is done. Workers handle the email/payment/invoice in the background.
Fast response — and a slow worker doesn't block anyone.
The web app, the workers and the third-party APIs all talk through the queue instead of calling each other.
1,000 orders in one minute? They line up; the workers chew through them at their own pace.
Emails, reports, video renders — anything that doesn't need to finish before the user sees 'done'.
Messages persist in the queue: if a worker crashes mid-job, the message is still there to be redelivered.
Three reasons, all about not waiting and not losing.
Response time stops depending on the slowest side effect. Eight seconds of work becomes 50 ms of enqueuing.
A failed worker is not a lost message — retry it, or park it in a dead-letter queue and investigate later.
Traffic doubled? Add another consumer. The queue is the rubber coupling between demand and capacity.
publish → store → route → consume → ack. Five words, one loop.
Follow one order from click to done.
Your app calls publish() with the message — e.g. an order JSON with id 42. That is the entire sending side.
The queue server (the broker) takes the message and keeps it — in memory and/or on disk, depending on the system.
Which queue does it go to? Simple systems: one queue per job type. Bigger systems route by name or topic.
A worker pulls the next message and does the work: send the email, call the API, render the invoice.
When done, the consumer acknowledges (ack) it. Crash mid-way? The message is redelivered — 'at-least-once' delivery.
Nobody calls anyone. Everything is a message moving down the line.
Flask · FastAPI · any app
Creates the work and drops it on the line: publish(order). Doesn't wait, doesn't know who picks it up.
⟶
publish()
RabbitMQ · Redis · Kafka
Holds the line: stores messages, routes them to the right queue, redelivers if a consumer dies. The shock absorber.
⟶
consume() + ack
worker-1 · worker-2 · worker-3 …
Loop forever: take the next message, do the job, ack. Too busy? Add another worker — the queue doesn't care.
The producer and the workers only ever meet the queue — never each other.
One message — one consumer, or many? That's the fork in the road.
One message, one consumer. A line of work: whichever worker is free takes the next task. Load gets spread automatically.
Use it for TASKS: 'process this order'.
One message, many consumers. Every subscriber gets its own copy — pub/sub style.
Use it for EVENTS: 'an order was placed'.
The payload — usually a JSON string plus metadata (id, type, timestamp). Small and self-contained.
The queue server that stores and routes messages: RabbitMQ, Kafka — or Redis standing in.
The two roles: who puts messages on the line, and who takes them off. The same app can be both.
The consumer's 'done, drop it'. Until the ack arrives, the queue keeps the message safe.
Failed messages get retried, then parked in a dead-letter queue — a parking lot for broken work.
The queue fills up when producers outpace consumers — a visible, honest signal to add capacity.
The whole 'async' trick from the web app's side: one line, and the request is free.
import redis
r = redis.Redis(host="localhost", port=6379)
def publish(order):
# put one message onto the queue
r.lpush("orders", order)
print("queued:", order)
publish({"id": 42, "item": "keyboard", "qty": 1})lpush = list push: the message lands at the head of the 'orders' list — a queue in one Redis command.
The worker side: take the next message, do the job, loop. Nothing is lost.
import redis, json
r = redis.Redis(host="localhost", port=6379)
while True:
item = r.brpop("orders", timeout=5)
if item is None:
continue # queue empty — keep waiting
order = json.loads(item[1])
print("processing:", order)
# work: send email, call API, render PDF…
# then loop — the next message is upbrpop = blocking pop: the worker sleeps until a message arrives (timeout = how long it naps).
LLEN = how many are waiting; LRANGE = peek inside; LPOP = take one off — that's a consume.
Three of these cover nearly every small-to-mid system you'll ship.
The classic general-purpose broker: queues, routing, acknowledgments, retries. When someone says 'message queue', this is usually what they mean.
You already met it as a cache — its lists double as a simple queue (LPUSH/BRPOP). Perfect for small systems; not built for hard guarantees.
The event-log giant: append-only, partitioned, replayable. Powers high-volume event streaming — a different (bigger) animal.
| RabbitMQ | Redis | Kafka | |
|---|---|---|---|
| What it really is | A message broker (queues + routing) | A cache that doubles as a simple queue | A distributed event log |
| Guarantees | ack + retry, solid | basic (in-memory by default) | durable log, replayable |
| Best for | General app work: jobs, decoupling | Small systems, quick async | High-volume event streaming |
Queues go very deep. These are real — but not your next step.
RabbitMQ's fine-grained routing (direct, fanout, topic). Overkill until you have many queues and rules.
Kafka's internal structure: partitions, offsets, consumer groups. Powerful — a whole course of its own.
Transforming streams on the fly (Kafka Streams, Flink). The frontier, not your next step.
A message queue is a rubber coupling in your system: the sender drops work on the line and keeps going; the workers take it at their own pace; and nothing is lost when someone hiccups.
Next: when one box isn't enough, the queue is the first thing you reach for.