Queries first, store second
O que você vai aprender
- describe what a store needs to do by asking: what do we look up, how often, and how fresh must the answer be;
- tell a lookup by exact key from a search by condition;
- avoid choosing a store just because of the format the data arrived in.
The Commandant looks at the darkened archives. “Five stores. Who decided what went where?”
QUERY is the first to find a clue — a log kept by Kotomarket’s programs of their requests for data. For the last minute before 18:00, it records what they asked for and how many times. Next to each entry, the shop engineers noted how fresh each answer had to be.
QUERY: Looks like the choice didn’t start with the name of a technology. First, they counted the questions.
Questions before the store
The shop has data about carts, products, parcels, and orders. You could open a list of technologies right away and pick a familiar name. But that can leave you with an awkward system: the program keeps asking for data one way, while the chosen store is designed for another.
So an engineer starts with what the program does. What will it ask of the data? What will it change? Which answers are needed constantly, and which only rarely? Only then does it make sense to choose a store.
For example, for customer 17's cart, the program already knows the exact name of the record it needs — the key cart:17. It does not need to scan every cart and check each one. This is a lookup by exact key.
Revenue is different. The question “all orders from yesterday over a thousand rubles” does not contain a known record number. The program first has to find the records that match the conditions. This is a search by condition.
Frequency changes the requirements too. The program may ask for some data once a day and other data thousands of times a minute. Reading and writing are considered separately: retrieving a saved value and changing data are different jobs for a store. If almost every request only reads a product card, that creates one kind of load. If requests keep increasing a view counter, the store has to accept changes again and again.
There is also the question of freshness. Suppose a value has already changed, but for a while the program still sees the old one. For a product card, that delay may sometimes be acceptable. For a cart or parcel location, the answer is needed within seconds. So the engineer writes down in advance how old an answer can be before the user notices a problem.
Finally, they look at the amount and shape of the data. Ten rows and a billion rows require different room for growth. Uniform records, where objects have a similar set of fields, and records that differ greatly can also lead to different choices.
That gives us four questions.
- What is used to find the data. By exact key — when the program knows the name of the record it needs or builds it from an ID. Or by condition, when matching records still need to be found.
- How often. Once a day or thousands of times a minute — and how much of that is reading versus writing.
- How fresh the answer must be. Is it fine to show what was true a minute ago, or must it be exactly what is true now?
- How much data there is and what shape it has. Ten rows or a billion, uniform records or completely different ones.
This picture is called an access pattern: it shows exactly how the program works with the data. It is like choosing a road based on the route: first you need to know where people travel and how often, and only then decide which road fits. If you start with a favorite technology, you end up forcing the problem to fit the tool.
The cell below will print a log from the archive. show_questions() is a training function in this course: it simply displays the log as a table. In real work, a summary like this is built from server logs and load charts.
What a “fresh answer” means
Suppose the contents of a cart have already changed. If the program immediately shows the new contents, the answer is fresh. If it keeps showing the previous state for a while, the answer is lagging behind the latest change.
That lag is not always a problem. In the log, product view counts are allowed to lag by a minute, while product cards may lag by minutes. The requirements are stricter for carts and parcel locations: there, the answer is needed within seconds.
It is like a timetable display. If the information updates a little after the event itself, the display keeps showing the previous state for a while. For an engineer, the practical question is: how much of that delay can this particular task tolerate?
Reading the log
The letter N in the log stands for a specific number — a customer, product, or parcel ID. The “answer freshness” column says how far the answer may lag behind the latest changes. “Must not be lost” is no longer a freshness requirement, but a durability requirement.
Let's start with the cart. The question “what is in customer N's cart” comes up 1,200 times a minute. The customer ID is already known, so the program can go straight to the exact key. The answer is needed within seconds: if a product has already been added or removed, the old state quickly becomes noticeable.
Views work differently. The question “+1 to views for product N” comes up 4,800 times a minute. Here the program also knows the product ID in advance, but the request changes data: the counter needs to increase. The displayed number, however, may lag by a minute.
Product card N is read 3,000 times a minute. Again, the product ID is known in advance. The freshness requirement is softer: the answer may lag by minutes.
Parcel location is requested 900 times a minute. The parcel ID is known, so once again an exact-key lookup fits. But an old location quickly becomes misleading, so the answer is needed within seconds.
These four rows are similar in how they search: the program knows the exact key and wants to get the record linked to it immediately. That makes a “key → value” store a good fit. But the rows differ in frequency and freshness, which is exactly why saying only “we search by key” is not enough.
Product search by words works differently. The program does not know the IDs of matching products in advance: it receives words and has to find matches. This kind of search is usually handed to a separate search engine, which we will cover in chapter 8. The main database can handle simple word searches too.
Revenue by category is calculated once a day and by condition: take all of yesterday's orders, add up the amounts, and split the result by category. There is no known key in advance. But there is a table and SQL, which work well for this kind of query.
An order is placed 60 times a minute. Compared with thousands of reads, that is infrequent, but there is a different requirement here: the record must not be lost. So orders live in the main database.
This is why the log separates these questions. Two rows may access data equally often, but one mostly reads while the other changes it. Or both may use a key, but one can lag by minutes while the other needs an answer within seconds. The choice of store comes from the whole picture.
QUERY: The shape of the data matters too, but it is not enough by itself. First find out how the program will read and change that data.
A little more Python: a dictionary
A dictionary stores “question → answer” pairs inside curly braces: the question is on the left of the colon, the answer is on the right, and pairs are separated by commas. A line like answers = {…} gives the dictionary a name — the checker will use it to find your answers.
The word None means “there is no answer yet.” Replace each one with the word «ключ» (key) or «условие» (condition) in quotation marks — single or double. Leave the questions, colons, commas, and braces exactly as they are.
A little later you will see . This format often appears in responses from websites and apps, in configuration files, and when programs exchange data. You can also see fields and their values inside it. But the appearance of a message still does not tell you where it is best stored.
None with the word «ключ» (key) or «условие» (condition) — in quotes, as in the dictionary example above.TermsWhat the new words mean
| term | in plain language |
|---|---|
| access pattern | how the program works with the data: what it looks for, how often, and how fresh the answer must be |
| key lookup | the exact name of the needed record is known in advance — for example, customer 17's cart |
| search by condition | only the criteria are known — city, amount; the matching records still have to be found |
Principais pontos
- Before choosing a store, write down the questions the program asks of the data: what it searches by, how often, how fresh the answer must be, how much data there is, and what shape it has.
- In the shop log, frequent exact-key requests fit a “key → value” store, while revenue calculations fit tables and SQL. These are examples of choices, not a law: SQL can handle frequent queries too. Orders that must not be lost are kept where their durability is best protected — in the main database.
- The format alone decides nothing: can live perfectly well in the main database too.
When you face your own task, start with the same list of questions. Take one program scenario and write down what it searches for, reads, or changes, how often it does that, and how much answer lag is acceptable. The technology name comes later.
The log had an “answer freshness” column. Where does an out-of-date answer come from at all if the data is already in the store? Next to Kotomarket's cache stands its twin. That is where we will look next.