How Things Work

Mechanism 02Distributed SystemsInteractive · 8 min

How does data move between systems that never talk to each other?

They share a log. Producers append to the end of a tape, and each reader keeps its own place. Reading removes nothing, and most of what makes Kafka unusual follows from that.

One topic, one consumer group, one simulated clock. Each tick is a chance for every running consumer to process one record.

Keyed records stay in order, because one key means one partition. Turn “Use keys” off and send more: watch the sink flag records that overtake each other.

Producer
orders-apiKeep sending
Use keys
Topic: orders · each cell is record, offset
P0
c1
0
lag
P1
c1
0
lag
P2
c2
0
lag
Consumer group: demo
c1● running
P0 P1
c2● running
P2
ReadySend records and watch them land.

Ready. Send records and watch them land.

tick 0
What the application did, in orderdashed = duplicate · filled = out of key order
0Delivered
0Duplicates
0Never processed
0Out of key order
0Total lag
2Active consumers
1Generation
Topic
GroupPause, crash or stop consumers on their own cards.
Records per poll, then commit
Commit
Idempotent application
Restart from an offset
Or click any record on a tape.
Sheet 02 · Topic, partitions, consumer groupSingle broker view, no replicationTicks are not real time
The model

Four rules explain most of it

Rule 1

A partition is an append-only log

Records are added at the end and get the next offset. Retention removes old records later, but consuming never does. Two groups can read the same data independently.

Rule 2

Order exists within a partition

There's no order across a multi-partition topic. If events about one customer must stay in sequence, give them a key so they share a partition.

Rule 3

One owner per partition, per group

That's what keeps a partition's order intact while it's processed. It also means more consumers than partitions adds spares, not throughput.

Rule 4

Progress is a number the consumer commits

The broker doesn't track each record as done. A group stores an offset per partition, and lag is the gap between that offset and the end of the log. Everything about redelivery follows from when that number gets written.

Under the hood

The same state, as an operator sees it

Live: this table reads the simulation above

$ kafka-consumer-groups --describe --group demo

PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID
0          0               0               0     c1
1          0               0               0     c1
2          0               0               0     c2

This is the shape of kafka-consumer-groups --describe. CURRENT-OFFSET is what the group last committed, so it moves in steps and lags behind the yellow cursor while a batch is in progress. The dashed line on each tape marks it.

A consumer is a loop around poll(): fetch a batch, process it, commit. The order of those last two steps is the choice between processing a record twice and possibly not at all.

while (running) {  const batch = await consumer.poll();  for (const record of batch) {    await handle(record);    // crash here…  }  await consumer.commit();   // …and this never ran}
When it breaks

Guarantees depend on what you built

Commit after · at-least-once

Crash, and records come back

Experiment 4. The successor restarts from the last commit, so work between that commit and the crash runs again. Nothing is lost, and side effects may repeat.

Commit before · at-most-once

Crash, and records vanish

Set “Commit: Before”, send 10 records, and crash a consumer while it's working. “Never processed” climbs: the group has moved past records nobody handled.

The practical answer

Assume delivery can repeat

Make handlers idempotent, for instance with a unique constraint on a record's ID. Turn on “Idempotent application” and duplicates arrive but do nothing.

Kafka's transactions give exactly-once processing for read-process-write flows inside Kafka. They don't extend to an arbitrary database or an email provider on the other side of your handler. That part is still yours to design.

Test yourself

Two predictions

A topic has six partitions. A group has eight consumers. How many can actively consume this topic at the same time?

Records A1 and B1 land in different partitions, with A1 written first. In what order will a consumer group process them?

Fine print

What this model leaves out

Brokers, replication and leaders

The tapes behave as one reliable log. Real partitions are replicated across brokers with a leader and followers, and acks and min.insync.replicas decide what “written” means.

Partitioning and assignment

The default partitioner hashes keys with murmur2, and keyless records go in sticky batches, far larger than the two records used here. Assignment here is a simple range. Real strategies include cooperative-sticky, which avoids stopping every consumer during a rebalance.

Commits and timing

Commits are offsets stored in an internal topic, and clients often commit on a timer. Lag here is measured from the committed offset, as the CLI does. A running consumer's real position is usually ahead of it. Sessions are measured in seconds, not ticks.

Pausing a consumer

This models the client's pause(), which stops fetching while polling continues. A consumer that simply stops calling poll() for too long gets removed from the group, which looks more like the crash path.

Keep going

Connected mechanisms