Orders sit in a Postgres database and usage events arrive from the app. Finance wants a trusted daily revenue table, and the product team wants the same data for a recommendation feature. You build one pipeline both can rely on.
A data engineer builds the pipes that move, clean and model data so analysts and ML teams can trust it. In AI teams that also means preparing documents and features for retrieval and training, and keeping that data fresh and compliant.
Goal: Learn SQL and Python well enough to move data from a source to a clean table on a schedule, and explain what each step does.
Before you start
Beginner · 8 weeks
Week 1: What data engineering is
Sources, pipelines, warehouses, consumers · Batch versus streaming · Rows, columns, types · Vocabulary: schema, ETL, ELT
Practice: Draw a pipeline for a shop: where data comes from, where it goes, who uses it.
Practice: Draw a pipeline for a shop: where data comes from, where it goes, who uses it.
Common pitfalls
Think first, then reveal
What is the difference between a primary key and a foreign key?
Think first, then reveal
What does idempotent mean for a pipeline?
Think first, then reveal
What is the difference between INNER JOIN and LEFT JOIN?
Think first, then reveal
Why check row counts after a join?
List every source, its owner, how often it changes and what a "wrong" row looks like.
The databases, APIs and event streams your data starts in, plus the agreement on what they will send.
Why this pick
Most pipeline failures start upstream, when a source changes a column nobody told you about.
What the role must know
Beginner mistake
Core
Python
Platform
Quality
Think first, then reveal
What is the difference between a primary key and a foreign key?
Think first, then reveal
What does idempotent mean for a pipeline?
Think first, then reveal
What is the difference between INNER JOIN and LEFT JOIN?
Think first, then reveal
Why check row counts after a join?
Think first, then reveal
How do you handle late-arriving data?
Think first, then reveal
Explain a star schema and why use it.
Think first, then reveal
A dashboard number changed overnight. How do you investigate?
Think first, then reveal
At-least-once versus exactly-once delivery?
Think first, then reveal
How do you stop upstream schema changes from breaking you?
Think first, then reveal
How do you fix a skewed Spark-style job?
Think first, then reveal
Design deletion for GDPR-style requests in a data lake.
Think first, then reveal
How do you decide between batch and streaming?
Build one, measure it and put it on your CV. Each has tools, setup commands and milestones.