10 chapters are open
First lesson: 09:00. Yesterday's mart is empty, the day before is doubled
Interactive course · Data Engineering
Yesterday’s mart is empty.
From today the load is yours.
Not an "introduction to data engineering" but one pipeline you assemble yourself: source → raw → staging → DDS → a mart in ClickHouse, on a schedule, with checks and re-runs. Real Airflow 3 and Kafka signatures, live SQL in PostgreSQL and ClickHouse, zero installation: no Docker, no cloud account.
- live cells in PostgreSQL and ClickHouse
- a real airflow.sdk import
- zero install: just a browser tab
from airflow.sdk import dag, task @dag(schedule="0 3 * * *", catchup=True) def night_shift(): @task(retries=3) def load(logical_date): ...
03:14 load failed
03:19 load success · rows ×2The same date, run twice
The offer
Six failures you have already seen
Each gets a chapter, and each is reproduced live: first you break the belt yourself, then you fix it — and the grader looks at how the warehouse behaves, not at the text of your query.
Who it is for
A delta course, not a from-scratch one
We know exactly what you already know, because we taught it — the SQL, analytics, performance and BigQuery courses sit right next door. So not a paragraph of them is retold here. And for the same reason the course is not for everyone.
It is for you
- An analyst or SQL developer who has been handed the nightly load — and now carries the pager for it
- Anyone confident with SELECT, JOINs, GROUP BY and window functions, comfortable in basic Python, who has never written a DAG
- Anyone reading Data Engineer job ads and seeing Airflow, ClickHouse and Kafka — three tools you will run here, not read about
- Anyone who finished our analytics course: the capstone rebuilds the very cohort mart you once computed by hand in a notebook
It is not for you
- A beginner without SQL. There is not one paragraph on SELECT, JOINs, GROUP BY or window functions — that is the neighbouring course, and this one starts where it ends
- Anyone looking for "a new profession in eight months". The promise here is narrower and more honest: the move from analyst to data engineer
- Anyone who needs Spark, Hadoop or Kubernetes. They are not in the course, and that is stated in a section of its own below, not in small print
- Anyone who needs a diploma and a job placement. The course gives you the skill, a working pipeline and a platform certificate — not a guaranteed offer
Three scenes
What breaks on the belt
Three mechanisms the course is built around — drawn rather than described: two runs of the same date, a high-water mark with a late row, and ClickHouse parts where a new version does not replace a row.
A repeat must leave no trace
A retry, a backfill, a manual re-run — ordinary life for a pipeline. A bare insert doubles the day; replacing the partition leaves the number untouched. The course checks it with a snapshot after two runs.
Take only what is new — and lose nothing
The high-water mark lives in a table, not in process memory. A backdated edit arrives BEHIND the mark — and a naive `>` will never see it.
ClickHouse runs on different physics
A Postgres-style UPDATE does not work here: rows live in parts, and a new version lands next to the old one. A plain SELECT shows both — FINAL and argMax collapse them.
How it works
Code you can paste into production
There is no teaching DSL of our own — it would void the course’s only promise. What is emulated is the execution, not the language: on a virtual clock three attempts with a five-minute retry_delay take milliseconds, and a ninety-date backfill runs in a single cell.
Two real databases
PostgreSQL and ClickHouse are live platform sandboxes, not screenshots. One table arrives in two physics: an ordinary table in PG, a MergeTree, ReplacingMergeTree or SummingMergeTree in CH.
Real signatures
`from airflow.sdk import dag, task, Asset` and `consumer.poll(timeout_ms=…)` — Airflow 3 and Kafka exactly as job ads name them. The emulator is stricter than production: a DataFrame in XCom and a `datetime.now()` inside a task do not pass.
Behaviour is what is graded
Not the text of a query nor the shape of its output: the actual topological order of tasks, the number of attempts, the number of partitions after a backfill, and a snapshot of the warehouse after two runs of one date.
The capstone starts at lesson one
Not a project bolted on at the end but a belt that grows: by chapter three you own a working stretch, by chapter seven a pipeline that survives a re-run. You cannot finish half of this course and walk away with nothing.
The intake level · Vault-9 · 2184
You take the shift at nine in the morning
The signals no longer arrive one at a time: every night the station’s receiver hauls in whole batches of archive snapshots, and the intake belt files them into the vault. Until today your predecessor ran it — the previous duty engineer, who left behind a run log and a set of unfinished checks. Yesterday’s mart is empty, the day before the rows are doubled, and the log shows a failure at 03:14 and a retry at 03:19. The story does not scatter into incidents: every chapter adds a stretch of belt that RUNS, and by the end of the shift the incident that started it all physically cannot happen again — written not in a comment but in the code of a check.
The belt did not break. It did exactly what it was told — twice. The difference between “it ran” and “it counted once” is what you are here to put into code.
- 03:14the nightly load fails
- 03:19the retry — and the day doubles
- 09:00shift taken: the mart is empty
- D+7the belt survives a re-run
- D+30the shift is handed over whole
The route
Four stretches of one belt
The program is an assembly, not a table of contents: the intake level, the vault, the beat and delivery, the duty. By the end of each stretch the belt can do one layer more — and that layer runs.
The intake level
Pull the data out of the source and keep it through a re-run
The vault
Grain, keys and version history — and code others run without you
Beat and delivery
A schedule, a columnar mart and a ninety-day re-run
On duty
Checks, the stream — and a shift handed over whole
The program
10 chapters, 58 lessons, 18.2 hours
The first chapter is open in full and free — together with the first real DAG, which you will run twice. The rest ship in waves, as BigQuery’s and Performance’s did.
The intake level
2 chapters · 13 lessonsPull the data out of the source and keep it through a re-run
The night shift: nine in the morning
Your first shift on the intake level of Vault-9: the role is handed over by an incident, not a definition — yesterday’s mart is empty, the day before is doubled, and the run log shows a failure at 03:14 and a retry at 03:19. By the end of the chapter you pull data from a source page by page, force it into a row contract, and run a real DAG twice — to see the duplicates and fix them.
Intake: raw, high water, late rows
The stretch from source to raw to staging: it takes only what is new, never loses a late edit, never multiplies rows and survives a re-run. The source answers 429 and cuts out mid-page — intake has to live through that. And the second extraction mode: a change stream, because otherwise a row deleted at the source lives in the mart forever.
The vault
2 chapters · 10 lessonsGrain, keys and version history — and code others run without you
The racks: grain, keys, version history
The vocabulary nothing works without: fact grain, surrogate keys, a historised dimension, the star. Two marts gave different numbers — because their grain differs; a dictionary was overwritten — and sales history drifted. The chapter deliberately precedes orchestration and ClickHouse: a backfill operates on a model the learner would otherwise not have yet.
The pipeline’s code: where the transformation lives
Turning a script into code other people run without you: where the transformation SQL physically lives, how it survives volume, where secrets belong and what a log line owes you. At 02:14 the loader eats the memory — because it pulled everything into one DataFrame. The chapter sits after modelling and before orchestration: you schedule finished code, not a draft.
Beat and delivery
3 chapters · 19 lessonsA schedule, a columnar mart and a ninety-day re-run
The beat: the DAG and time
Finished code goes on a schedule. The chapter does not repeat graph basics — those are already free in de1l5 — and opens with what no lecture conveys: the scheduler re-reads your file again and again. Logical date versus "today", a catchup that fires ninety runs, retries that take the neighbours down, and a DataFrame that went into XCom.
Delivery: the mart in ClickHouse
A different physics: a columnar engine where ORDER BY is the storage order, a partition is the unit of lifecycle, and versions live where UPDATE would. You create a million rows in the cell yourself and watch what MergeTree does with them. The first lesson is free — "different physics" only sells by being shown.
The transfer and the re-run
The two halves of the pipeline join, and the pipeline learns to survive the past: the PostgreSQL → ClickHouse transfer with every type trap, money in a float and an hour of timezone drift, batch size and "too many parts", partitions in PostgreSQL, a ninety-day backfill — and two runs of the same date colliding.
On duty
3 chapters · 16 lessonsChecks, the stream — and a shift handed over whole
Control: the check that fails the pipeline
An analyst’s checklist becomes code that stands in front of the mart and fails on its own. The data never came, yet the mart is not empty — because it is showing yesterday; a quiet dropna ate the revenue. Freshness, volume and observability are COMPUTED here by querying the run log, not described in prose.
The stream: Kafka and "duplicates or losses"
The idempotency thread closes onto a continuous input: key and partition decide the order, committing before or after processing decides what you get — duplicates or losses, and a third consumer triggers a rebalance. Exactly four lessons — exactly as many as the emulator can show honestly.
Shift handed over: capstone, map, exam
The pipeline gets built end to end in four twenty-two-minute passes: intake, a DDS with a star and SCD2, the ClickHouse mart, a schedule with checks and backfill. Then the map of the territory: an honest list of what the course does not cover — AFTER you pass, not before. And an exam made only of what the server can verify. The last entry in the duty log is the one the course was started for: the incident of your first shift can no longer happen.
The boundary
What this course does not cover
The course ends with a map lesson placed AFTER the capstone: you should learn the edge of your skill before a recruiter finds it for you. This page owes you the same — before you buy, not after.
Spark and Hadoop
pyspark needs a JVM, and there is none in a browser. Promising them would mean selling slides as practice.
Instead there is a bridge lesson: shuffle, broadcast joins and partitioning shown on what actually executes here, plus an interview card on "MapReduce vs Spark".
Kubernetes, Terraform, cluster administration
That is the platform team’s job, not the pipeline author’s. This course is about what a data engineer is paged for at night.
What of it comes up in interviews lives as inserts inside working lessons; the course is not padded with separate "interview stands".
dbt as the course’s backbone
In a live sample of Russian job ads dbt did not appear once, while Greenplum appeared in five cards out of fourteen. Building the course around dbt would mean teaching the wrong market.
dbt stays as inserts — the snapshot bookkeeping columns you will meet in someone else’s project — and the freed budget goes to Airflow, MPP and idempotency.
A real cluster and a live WAL
The platform sandbox runs as an unprivileged role without replication — otherwise it would stop being a sandbox.
The change stream arrives as a ready log shaped like a Debezium event, and you write the sink for it. Every emulated lesson carries a "what is faked here" notice — its presence is enforced by a gate, not by the author’s conscience.
What you keep
More than chapters you have read
The duty log
The notes you keep from lesson one: symptom, cause, and the check that stops it from happening again. By the end it is your own on-call checklist for a pipeline.
A working pipeline
A four-part capstone: intake, a DDS with a star and SCD2, the ClickHouse mart, and a DAG on a schedule with checks and backfill. Not a toy example but a stretch of belt that runs.
The pipeline exam
A final set of tasks on the shared "night shift" dataset: you fix someone else’s belt and finish your own. Pass — and take a personal platform certificate.
Questions
What people ask before starting
No. Airflow, Kafka and the data source run as emulators right in the tab; PostgreSQL and ClickHouse are real, inside the platform sandbox. No Docker, no cloud account, no credit card. Open a tab and start writing a DAG.
The shift starts at nine
The first chapter is open in full: 6 lessons free, live SQL and a real DAG right inside the lesson. Nothing to install.
A lesson takes 14–22 minutes, with no deadlines