Case #0: "Conversion dropped" — your first investigation

The brief

L.'s journal, entry #0 — the earliest one, before the numbered cases:

"Monday. The CEO in chat: 'End-to-end conversion fell from 6.0% to 4.9% in a week. WHAT IS GOING ON?!' Gave myself one hour to answer."

This is everything there is. Kotomarket's funnel over two weeks, counted in unique users, seven-day window from the first step, identical definitions in both weeks:

One extra fact from the change log: on Monday of week 11 marketing launched a new ad campaign. The computation method was not touched in between — that is the first thing L. checked.

The rest is your job. Before reading the template, look at the table for a minute and try to phrase what exactly bothers you about it.

The investigation template: six steps

This template is the skeleton of every case in the course. Copy it down; from here on it only gains detail.

Step 1. What exactly changed. Name the metric, both values, the window and the unit. Here: end-to-end conversion in unique users, 6.0% → 4.9%, week over week. Immediately express the change both ways: that is −1.1 pp and roughly −18% relative. And check the methodology first: definitions, windows, timezone — did anything move? Half of all "drops" end at this step.

Step 2. Decompose the metric. End-to-end conversion is the product of three step conversions. So the fall could come from any one of the three steps, or several at once. Compute every step conversion for both weeks.

Step 3. Find the first level of divergence. Compare the steps pairwise and locate where exactly the series parted. A good analysis always ends with a precise address: not "conversion dropped" but "this particular transition sagged".

Step 4. Collect the facts. A fact is something visible in the data that requires no faith. Absolutes, shares, changes. Always look at both the numerator and the denominator: a share can fall with no deterioration of the numerator at all.

Step 5. Formulate hypotheses. A hypothesis is an explanation consistent with the facts but not yet tested. There should be several, and they should differ in nature: product, traffic, data, the outside world.

Step 6. Name the data that will test them. Every hypothesis comes with a concrete next query: "split cart→payment conversion by traffic source", "compare new versus returning users". An investigation ends not with a verdict but with a next step.

The discipline of the whole template rests on one separation: facts apart, hypotheses apart. In a report they occupy different paragraphs and different words: "the data shows" versus "a possible explanation".

Your verdict — before you read the debrief

Before opening the next block, phrase the answer to the CEO yourself. Here is the form to fit it into — fill it in mentally or on paper:

WHAT CHANGED
  metric: ______, from ______ to ______ (____ pp, ____% relative)
  methodology: ______________________

FACTS (visible in the data)
  1. transitions 1 and 2: ______________________
  2. transition 3: ______________________
  3. funnel entry: ______________________
  4. reached payment: ____________________
  5. shortfall against the old rate: ______ paying users

HYPOTHESES (not yet tested)
  1. ______________________
  2. ______________________
  3. ______________________

NEXT CHECK
  ______________________

Two hints so the form does not stay blank.

First: look at the absolute number who reached payment in both weeks. It is the same — 720. This is precisely the case from the lesson on shares: end-to-end conversion fell while the numerator did not move at all. The numerator did not drop — the denominator grew.

Second: the hypotheses must differ in nature. One about the product ("something broke at checkout"), one about traffic ("different people arrived"), one about the data ("some payments stopped reaching the log"). If all three are about the same thing, you were not generating hypotheses — you were justifying a hunch.

Filled it in? Now you may open the debrief.

L.'s debrief

What changed. End-to-end conversion in unique users, 6.0% → 4.9% week over week: −1.1 pp, about −18% relative. The methodology was untouched — same definitions, same window, same timezone. That was the first thing I checked, and this is not that case.

Facts.

1. The "opened → viewed" and "viewed → cart" transitions did not change at all: 50% and 30% in both weeks.
2. The "cart → paid" transition sagged: 40% → 32.7% (−7.3 pp, −18% relative).
3. The funnel entry grew: 12,000 → 14,700, that is +22.5%. A new campaign launched on Monday.
4. Users who paid are exactly level — 720 and 720. In absolute terms the number of users who paid did not change.
5. Had cart conversion held at 40%, there would have been 882 users who paid. The shortfall is 162 users who paid.

Hypotheses, none tested.

• Traffic. The campaign brought people who behave normally up to the cart but pay less often: a different segment, different intent, different ability to pay.
• Product. Something in checkout genuinely got worse in week 11 — a new payment method, a delivery-price change, a broken form. Then returning users would have sagged too.
• Data. Some payments stopped being recorded (a new screen not firing its event, say). Then there is no drop at all, only a hole in tracking.

Verdict. I have no grounds to claim checkout is broken: the first two transitions are untouched, the number of users who paid is stable in absolute terms, and the entire effect sits on the last step while the entry grew. But neither can I claim traffic is to blame — that is a hypothesis, not a fact.

Next check. Split "cart → payment" by traffic source and separately compare new users against returning ones. If returning users still convert at 40% and only the campaign arrivals sagged, the question is about traffic quality, not the product. If everyone sagged, go into the product and the tracking.

What this case teaches

Never conclude from absolutes without a denominator — or from percentages without a volume. Here both views are needed at once: the percentage fell, the number reaching payment did not move. The percentage alone says "catastrophe"; the absolute alone says "all fine". The truth is in between: the product had the same number of paying users while spending 22.5% more incoming traffic to do it.

Separate facts from hypotheses. In the report above the facts are numbered and checkable, and the hypotheses are called hypotheses. The phrase "I have no grounds to claim that…" is not cowardice but precision: it does not stop teams from working, and it does not send them to fix something that is not broken.

End with a next step. Case #0 is not closed — it is taken to the state where it is clear which single query stands between you and the answer. That is a normal result for an hour of work.