Queue vs Topic (Pub/Sub)

The Coupling Problem

Imagine you just launched an e-commerce platform. When a customer places an order, three critical services need to act on that event: the inventory service must reserve stock, the email service must send a confirmation, and the analytics service must log the transaction. If you’re using a simple queue model, your order producer sends one message and exactly one consumer receives it. But you need all three services to respond. So you create three separate queues, and your producer publishes the same order to all three. This works for a while.

Then comes the dark day when someone asks, “Can we add a notification service that sends an SMS?” You modify the producer to write to a fourth queue. Then a sixth, eighth, tenth service appears on your architecture diagram. Suddenly, the producer is tightly coupled to every downstream service. You can’t add a new consumer without modifying production code. This is the problem that publish-subscribe (pub/sub) models were designed to solve.

In this chapter, we’ll explore two fundamental messaging patterns that power modern distributed systems and learn when to use each one—and why choosing wrong can create operational nightmares.

Point-to-Point vs. Broadcasting

The Queue Model: One Message, One Consumer

In a traditional message queue, a producer sends a message once. A single consumer picks it up and processes it. If multiple consumers are connected to the same queue, they act as competing consumers—each message goes to exactly one of them. This creates natural load balancing: if you have a queue with 1,000 pending tasks and 5 workers connected, each worker gets roughly 200 tasks.

This model is perfect for work distribution. If you have a batch of image resizing tasks, you don’t want five workers all processing the same image—you want to split the work. A queue guarantees that happens automatically.

Producer → Message Queue → [Consumer A]
                        ├→ [Consumer B]  (only one gets each message)
                        └→ [Consumer C]

Once a message is consumed and acknowledged, it’s typically deleted. The queue doesn’t keep history. This saves storage and makes cleanup simple.

The Topic Model: One Message, Many Subscribers

In pub/sub, a producer publishes a message to a topic. Every active subscriber to that topic receives a copy of the message—simultaneously and independently. If you add a new subscriber tomorrow, it won’t get messages from before it subscribed (unless configured otherwise), but it gets everything going forward.

Producer → Topic (Event) → [Subscriber A]
                        ├→ [Subscriber B]
                        └→ [Subscriber C]  (all get every message)

This decouples the producer from the consumer count. Add a tenth service? The producer doesn’t care. The new service just subscribes to the topic, and it starts receiving events. This is event-driven architecture at its core.

The Radio Station Analogy

Think of a message queue as a ticket counter at a bank. You take a number, and when your number is called, you approach an available window (consumer). Each ticket is handled once, by one person. If there are 10 people at 5 windows, some windows will be empty as others get busy. Load balances itself.

Now imagine a radio broadcast (pub/sub). The DJ plays a song. Every radio tuned to that station hears it simultaneously. People can tune in or out whenever they want. The broadcast doesn’t care how many people are listening—they all get the same signal. But if you tune in late, you miss the beginning.

Kafka consumer groups are like having multiple households, each with one radio. The broadcast still reaches every household, but within each household, only one person needs to listen (or rather, only one consumer in the group processes each message). So you get the broadcast reach of a topic, but the load balancing of a queue.

Consumer Groups: The Hybrid Model

Kafka introduced a brilliant concept that bridges these two worlds. When you create a consumer group, Kafka divides topic partitions among the consumers in that group. Each partition is assigned to exactly one consumer at a time. If you have a topic with 10 partitions and 3 consumers, roughly 3-4 partitions go to each consumer.

The result: message fan-out happens at the consumer-group level, and load balancing happens within the group. Add a new consumer group? The topic automatically fan-outs to it. Add a consumer to an existing group? Kafka rebalances automatically. The producer never changes.

graph LR
    Producer -->|publishes to| Topic["Topic: Orders"]

    Topic -->|partition 0-2| CG1["Consumer Group: Inventory"]
    Topic -->|partition 3-6| CG2["Consumer Group: Email"]
    Topic -->|partition 7-9| CG3["Consumer Group: Analytics"]

    CG1 -->|offset tracked| Consumer1["Consumer 1"]
    CG2 -->|offset tracked| Consumer2["Consumer 2"]
    CG3 -->|offset tracked| Consumer3["Consumer 3"]

This is why Kafka dominates in modern architectures. You get both capabilities.

How It Works: From RabbitMQ to Kafka

RabbitMQ: Exchanges and Routing

RabbitMQ implements both models explicitly through different exchange types:

Fanout Exchange (pure pub/sub): Every message published to a fanout exchange goes to all queues bound to it, regardless of routing keys.

Direct Exchange (queue-like): A message goes to a queue only if the routing key matches exactly. Many-to-one mapping.

Topic Exchange (flexible routing): Routing keys use pattern matching. A producer might use order.created, and a subscriber listening to order.* catches it. Another listening to #.failed catches any failure. This gives you both the decoupling of pub/sub and the selective routing of queues.

# RabbitMQ Topic Exchange Example
exchanges:
  - name: order_events
    type: topic
    durable: true

bindings:
  - queue: inventory_queue
    exchange: order_events
    routing_key: "order.created"   # Only order.created messages

  - queue: email_queue
    exchange: order_events
    routing_key: "order.*"         # All order.* messages

  - queue: analytics_queue
    exchange: order_events
    routing_key: "#"               # All messages (catch-all)

Kafka: The Append-Only Log

Kafka’s architecture is fundamentally different. Instead of deleting messages after consumption, Kafka treats every topic as an append-only log. Messages are stored durably, indexed by offset (position in the log).

When a consumer processes messages, it doesn’t delete them. Instead, it commits an offset: “I’ve processed up to message 42,000.” Next time it reconnects, it resumes from offset 42,001. This has profound implications:

1. Multiple independent consumers: Many consumer groups can process the same partition independently, each tracking their own offset.
2. Replay: If a service crashes and recovers, it can reprocess historical messages by seeking to an earlier offset.
3. Retention: Messages stay on disk for a configured period (default often 7 days, but configurable) or until a size limit, whichever comes first.

# Kafka Consumer Group Example (Python)
from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'orders',
    group_id='email_service',
    bootstrap_servers=['localhost:9092'],
    value_deserializer=lambda m: json.loads(m.decode('utf-8')),
    auto_offset_reset='earliest'  # Start from beginning if no offset
)

for message in consumer:
    email = message.value
    send_confirmation_email(email['customer_email'], email['order_id'])
    # Offset is automatically committed in background

If you add a second consumer group for a new service, the topic doesn’t change. The new group simply starts reading from the beginning (or from the latest, depending on configuration) and maintains its own offset pointer.

Real-World Systems

AWS: Uses SNS (Simple Notification Service) as a topic layer and SQS (Simple Queue Service) as queues. The “fan-out” pattern combines both: publish to SNS, have SNS push messages to multiple SQS queues, then have consumer applications read from those queues. This gives you topic decoupling with queue durability and per-queue rate limiting.

Google Pub/Sub: Similar philosophy—one topic, many subscriptions. Each subscription independently consumes messages and tracks its own acknowledgment position. No explicit consumer groups needed; each subscription acts like a consumer group.

Azure Service Bus: Topics with subscriptions (analogous to Kafka’s topic/consumer group model) plus queues for point-to-point messaging.

Message Ordering: The Hidden Gotcha

In a simple queue, messages are consumed in order (FIFO). Not so in pub/sub.

With RabbitMQ topic exchanges, there’s no global ordering guarantee. If two messages arrive with routing keys order.created and order.paid, different subscribers might process them in different orders.

Kafka gives you ordering within a partition. Since each partition is consumed by exactly one consumer in a consumer group, you get strict ordering per partition. But if you have 10 partitions and 10 consumers, message order across partitions is not guaranteed.

Pro tip: If you need global ordering, send all messages for a related entity (like all events for order #12345) to the same partition. Use the entity ID as the partition key.

Storage and Cost Implications

Queues typically delete messages after consumption. This is efficient for short-lived tasks like resizing an image or sending an email. But if you ever need to replay or debug, the messages are gone.

Kafka keeps everything for days or weeks. This is powerful for event sourcing and debugging, but it’s not free. A high-throughput topic with millions of events per day can consume gigabytes of storage.

RabbitMQ and other traditional brokers fall in between. Messages persist, but they’re bound to a queue and deleted on consumption.

Choose based on your needs:

• Queues: Good for work distribution, fire-and-forget tasks. Cleanup is automatic.
• Topics with short retention: Good for event notification. Lower storage cost than Kafka.
• Topics with long retention (Kafka): Good for event sourcing, audit logs, replay scenarios.

When to Use What: A Decision Framework

Use Case	Model	Reason
Image resizing tasks	Queue	Distribute work among workers; delete after processing.
Broadcast “order placed” to 5+ services	Pub/Sub Topic	Decouple producer from consumer count.
E-commerce with evolving services	Kafka Topic + Consumer Groups	Add services without code changes; replay events for debugging.
High-volume event stream (analytics, metrics)	Kafka	Long retention for historical analysis.
Point-to-point request-reply	Queue	Simple, single consumer, ordered.
Event sourcing / audit log	Kafka Topic	Immutable log; replay to rebuild state.

Common Pitfalls

Mistake 1: Using a topic when you need a queue. You set up a Kafka topic thinking “broadcast to everyone.” Then a month in, you realize multiple instances of the email service are processing the same event. Use consumer groups from the start.

Mistake 2: Publishing too much to a topic. You send every single database write to a pub/sub topic. Now you have 50,000 events per second, and downstream services can’t keep up. Be selective about what you publish. Publish domain events (things that matter), not implementation details (every row update).

Mistake 3: Ignoring message ordering. You process an “order.created” event before the corresponding “payment.confirmed” event, causing a refund on a non-existent order. Either enforce partitioning by entity ID or add ordering logic at the application layer.

Mistake 4: No dead-letter queue (DLQ). A message arrives that crashes the consumer. The consumer retries forever, blocking the entire partition. Always configure a dead-letter queue and alert on it.

Key Takeaways

• Queues (point-to-point) are for task distribution: one message, one consumer, automatic load balancing.
• Topics (pub/sub) are for event broadcasting: one message, many subscribers, producer-consumer decoupling.
• Kafka consumer groups blend both: a topic fans out to multiple groups, but within each group, messages load-balance across consumers.
• Message ordering is guaranteed in queues, partition-scoped in Kafka, and not guaranteed in traditional pub/sub unless you add logic.
• Storage and retention matter: queues delete messages on consumption (cheap, short-lived); Kafka retains for days/weeks (expensive, replay-friendly).
• Choose your model based on semantics: work distribution → queues; event notification → topics; high-volume event streams → Kafka with consumer groups.

Practice Scenarios

Scenario 1: The Scaling Nightmare

You have a payment processing system. When a payment completes, you need to:

• Update the user’s account balance
• Send a receipt email
• Trigger fraud detection analysis
• Log to analytics
• Notify the merchant API

You initially used a single queue with one consumer that did all five tasks sequentially. Now it’s a bottleneck. How would you redesign this with pub/sub and consumer groups? What if fraud detection takes 30 seconds but receipt emails take 1 second?

Scenario 2: The Replay Problem

A bug was introduced three weeks ago that caused the email service to send thank-you emails with the wrong order ID to 10,000 customers. You want to resend correct emails to only those affected customers. Would a traditional queue help? What about Kafka? Why?

Scenario 3: The Ordering Disaster

Your e-commerce system publishes both order.created and payment.confirmed events. A race condition causes some inventory services to see payment.confirmed before order.created, reserving stock for non-existent orders. You have a Kafka cluster with 8 partitions and 3 consumer instances. What’s one fix? (Hint: think about partition keys.)

---

We’ve now seen how the choice between queues and topics shapes your entire message architecture. Queues are the workhorses of task distribution; topics decouple producers from an ever-changing set of consumers. Kafka’s consumer groups give you the best of both worlds, but at the cost of operational complexity. In the next section, we’ll explore the guarantees these systems can and can’t provide: what happens when a message disappears mid-processing? When does at-least-once delivery matter more than idempotent consumers?