The second signal: Kotomarket's black box

Year 2184. The second signal

Station Vault-9's receiver had been silent for eight months — ever since you took the Kotomarket archive apart down to the last table. Today at 03:11 it caught a second signal from the same dead sector of the Net.

This is not a storefront snapshot like last time. This is a black box — the company's entire analytics archive covering the last two and a half years of its life: app events (every launch, every click, every abandoned search), daily metrics, the logs of "Pult" the BI system — and an incident diary signed with a single letter: "L."

The final entry was made nine days before the company shut down:

"The data was screaming. Nobody listened. If anyone ever finds this archive — teach them to listen. — L."

QUERY opens the journal's first page and purrs unusually quietly:

"An archivist reads what happened. An investigator finds out why. You are promoted. Don't thank me — I checked you were ready. Repeatedly."

But before opening the cases you have to learn the language they are written in. It starts with a single word: event.

An event is a row in a table

The black box's first table is called events and holds 1.9 billion rows.

An event is a fact: someone did something at a particular moment. In data a fact looks unremarkable — like a row in a table. Here are five rows from the evening of March 14:

user_id | event_name      | event_time | product_id
--------+-----------------+------------+-----------
4412    | product_view    | 19:00:12   | 812
4412    | add_to_cart     | 19:01:47   | 812
4412    | checkout_start  | 19:02:03   | 812
7781    | product_view    | 19:02:15   | 305
2043    | app_open        | 19:02:41   |

The first rule of reading such a table:

One row = one action.

Not one user, not one order, not one product — one action by one person in one second.

Read the first three rows out loud: user 4412 viewed product 812 (the "Murzik hammock bed") at 19:00:12, added it to the cart a minute and a half later, and started checkout sixteen seconds after that. Then — silence. They never bought. Without computing anything you reconstructed a slice of someone's evening out of four columns.

Notice what the table does not have: no "mood" column, no "why they left". An event records the bare fact, with no explanation. The explanation is invented and tested by the analyst — that is the whole profession.

The chain the whole craft rests on

Every number a company's management sees travels the same road. Let us walk it step by step — on the same five rows.

Step 1. The action. Barsik the cat pokes the "Add to cart" button.

Step 2. The event. The app sends the server a message: "user 4412, add_to_cart, 19:01:47, product 812". The server appends a row to the events table. The action has become a fact in the data. And remember right away: there is no way back — whatever was not recorded never happened as far as the analyst is concerned.

Step 3. The metric. A metric is a number computed from facts by an agreed rule. For instance DAU (daily active users) — the count of unique user_ids for a day. Let us assume these are all the rows for the day, and calculate:

event rows:     4412, 4412, 4412, 7781, 2043   ← five rows
unique ids:     4412, 7781, 2043               ← three users
DAU = 3

Five events — but DAU is three, because 4412 shows up three times while the metric counts people, not actions. This is where beginners stumble: there are always more events than people, often hundreds of times more.

Step 4. The chart and the dashboard. A single number is the state of a single day; whether it is good or bad you cannot tell from it. You need a series: DAU on March 13, on the 14th, on the 15th… The series becomes a line, the line lands on a dashboard — the screen management looks at every morning.

Three things everybody calls "data"

The distinction is practical, not philological. An event cannot "drop 30%" — it either happened or it did not. What can drop is a metric: by changing the counting rule, by losing part of the events, by breaking the code that sends them.

That is where the investigator's work begins: when the line collapses, you must work out what actually fell — the company's life, or its measuring instrument.

Two databases — two kinds of questions

Last time you worked with the shop's production database: products, orders, users. Its job is to keep the app running right now.

Saying "the production DB stores state and the warehouse stores history" would be convenient and wrong: order history lives in the production DB too. The border runs not through the data but through the shape of the question.

Questions for the production database: "What is the current status of order #812?", "How many hammock beds are left in the Myausino warehouse?", "What is user 4412's delivery address?" — all about one object right now. The answer is needed in milliseconds because a phone screen is waiting; there are thousands of such queries per second, each touching a couple of rows.

Questions for the analytics warehouse: "How did the cancellation share move week by week over the year?", "What share of March newcomers survived until June?", "How much did the 'food' category earn per city last quarter?" — all about many objects over a long stretch of time. The answer is needed in seconds or minutes — a human is waiting, not a screen; there are dozens of such queries a day, each chewing through millions of rows.

That is why the warehouse is built separately: send a heavy year-long query into the production database and it falls over, taking the shop with it. Data reaches the warehouse as a copy: usually once a day, and the event stream almost immediately.

A data analyst lives in the second column. In this course you get Kotomarket's entire black box — and every unsolved case in L.'s journal.