Kafka and the alternatives: when you do not need it
O que você vai aprender
- to build a queue out of one table and one query
- to take free rows and silently skip the ones a neighbor is holding at that very second
- to name what a queue broker can do and what a log simply does not have
- to answer three selection questions — and run your own feed through them, where the answer comes out the other way
09:20. Six messages a day
2 November 2184. The dispatch room on Vault-9's comms level. The rack hums, the feed runs steadily. And a wave of enthusiasm hits you: since the feed already works, why not route absolutely everything through it? Including the notice “the darkroom is out of film”. There are six such notices a day.
Without opening his eyes, QUERY rolls the previous shift's log out onto the panel. A topic like that has existed before — film_out, six messages a day.
Five years ago, on 14 March, its consumer — the program that picked up the notices and called people in — died at 02:10. Nobody noticed until 06:30 — when the curator came in and saw that the darkroom had stopped. For four hours twenty minutes nobody knew a thing.
Why did nobody notice? Whether the consumer was alive was judged by one number — its lag: how many messages it had not read yet. On a topic with six messages a day, a dead consumer looks exactly like a live one: a lag of zero. Not a single message arrived in the topic overnight; the first one was only due around noon. The sensor stayed silent precisely because there was nothing to shout about.
QUERY: I remember the names of topics that no longer exist. This one was not closed because of Kafka.
Why the whole level was closed, QUERY will not say: “Ask me the first time an acknowledged write disappears on you.” Today's question is a different one — does the film need Kafka at all.
dsp_queue is a queue table: one row is one job, and the status column says whether the job is free or already taken. Worker w1 claims the three oldest free jobs with ONE statement. FOR UPDATE SKIP LOCKED is the key part here: the query takes free rows and silently skips the ones another worker is holding at that very second — instead of standing there waiting for it.A queue is a table and one query
There are three jobs on the screen, and nothing else was needed: no separate system, no network, no configuration. If you already have a database, you already have a queue. A second worker running the same query gets the next three free jobs, not these: it skips the rows that are taken.
Where does such a table give up? Under a heavy stream: every job taken is a write to the database, finished rows have to be cleaned up by , and by tens of thousands of messages a second the database works for the queue alone. And where one message is needed by a hundred recipients: a row is taken by just one of them. That is what queue brokers are for — RabbitMQ and its kin. A queue broker is a middleman: it keeps a message until the recipient acknowledges it, and hands an unacknowledged one to someone else. Once acknowledged, the message is gone from the broker.
Do not confuse it with a Kafka broker: that is one machine of a Kafka cluster — the one that keeps logs on its disk. That is why RabbitMQ and its kin go by two words from here on — a “queue broker”.
What a log cannot do in principle
What people most often want from a queue is one thing: to retry ONE failed message. A log cannot do that. A consumer remembers not a list of what is done but a single number per partition — the offset it has read up to. There is no way to write “everything up to the eighth is done except the fourth” with that number. So when the handler fails on the fourth message out of eight, there are three ways out, and every one of them costs something:
- back — rewind the position: the fourth goes again, but the fifth to eighth wait, and the partition stands still;
- forward — move the position on: the group skips the fourth for good — it is still in the log, but the group will not come back to it;
- aside — move the fourth to a separate place and carry on: seven of eight are done, one waits for a human.
Three selection questions
Now you have both prices, and three questions decide the choice:
| question | if the answer is | what we take | what it costs |
|---|---|---|---|
| how many systems receive one event? | one | a queue table or a queue broker | history cannot be reread |
| do messages need to be retried one at a time? | yes | a queue, not a log | order is not preserved |
| how much data, and how fast is the answer needed? | little, minutes don't matter | a queue table in your own database | you hit the database's limits as it grows |
film_out | one system, one-by-one retries, 6 a day | a queue table | nothing: the database is already there |
For the film the answer is obvious: one table in a database that already exists. Kafka was superfluous there — and that is exactly why its dead consumer hid behind a zero.
Now the same questions about our feed
First, one word. The systems the feed goes to are called downstream systems: today it is the reading room; by the end of the watch there will be six of them. Do not confuse them with a consumer: a consumer is the program that reads a topic, while a downstream system is whoever uses the result.
We ask the same three questions — and the answer comes out the other way:
- one feed message has many downstream systems, each with its own pace;
- one-by-one retries are not needed: a message that failed to parse is set aside, and the feed moves on;
- there is a lot of data, and it never stops: a queue table in a database would choke on such a stream. And the reading room's answer shelf life is 30 seconds. Yesterday's belt gave a worst age of 30 h 12 min, an hourly run would give 64 minutes, and streaming gives 1.2 seconds — numbers from the cell of the lesson “Batch report or stream: how long the answer waits”, which you printed yourself.
And something a queue table does not have at all: what is read from a log does not disappear — you can reread it, and the same log can be turned into a directory. That is what the lessons “A topic is a log: segments and ” and “Compaction: the latest value per key” are about. This is where Kafka belongs — and the numbers show it.
One caveat for the future: in Kafka 4 some queue scenarios are covered differently from this lesson (in detail: the lesson “Share groups: Kafka as a queue” in the “Going deeper” track).
What in this lesson is real and what is not
The query with FOR UPDATE SKIP LOCKED is real PostgreSQL. But one cell cannot run two workers at once, so you do not see the race for rows — only the shape of the query and the result. That a second worker gets a different three jobs is a PostgreSQL rule: we state it, we do not show it.
The three outcomes of a failed message are shown with a diagram, without a run: they follow directly from a group having one position per partition. The Kafka 4 mode from the caveat above does not run here either — the training Kafka shows it in the lesson “Share groups: Kafka as a queue” in the “Going deeper” track.
Interview question
How this comes up in interviews
The most common question: “Kafka or RabbitMQ?” The worst answer is “Kafka is faster”: it shows the candidate does not understand how the two models differ.
A good answer starts with two counter-questions: how many different destination systems does one event have, and does an individual message need to be resent. Many destinations, each with its own pace, history that needs rereading — Kafka. One worker, one-by-one retries, priorities, delayed delivery — a queue broker (message broker: RabbitMQ, ActiveMQ, NATS JetStream).
The second question: “And when would you not take a queue broker at all?” They expect a queue table in a database you already have, and an honest estimate of the load it can handle. The industry name of the technique is SELECT … FOR UPDATE SKIP LOCKED, and the table itself is a queue table.
The third question is recent: “Did anything change in Kafka 4?” It did — Kafka itself now covers some queue scenarios. Answering “Kafka is not a queue, full stop” is no longer complete (in detail: the lesson “Share groups: Kafka as a queue” in the “Going deeper” track).
Principais pontos
| task | what we take | why |
|---|---|---|
| darkroom notices, 6 a day | a queue table in your own database | one destination, one-by-one retries |
| the feed of twelve antennas | Kafka | many downstream systems, each with its own pace, an answer needed within 30 s |
A queue is one table and one query: FOR UPDATE SKIP LOCKED takes what is free and skips what is taken. A queue broker retries messages one at a time; a log cannot: a group has one position per partition. Kafka is needed where an event has many destinations and history gets read again.
QUERY: Leave the film to the database. The feed is ours. This evening we'll look at where it really lives.