End-to-End Data Engineering

By the end, you can

1. Write production Python and SQL for data work
2. Model a warehouse with star schemas and slowly changing dimensions
3. Build ETL and ELT pipelines with dbt and schedule them with Airflow
4. Process big data with PySpark and Databricks, and pass the Databricks associate exam
5. Stream events with Kafka into a live table
6. Ship pipelines with Docker, Git and CI/CD, and step into a data engineer role

What you get

• Live classesTaught live on a schedule, with time to ask questions as they come up. Every class is recorded, so a missed session is never lost.
• Reading materialsRunnable, chapter-by-chapter notes with real outputs, in the portal. Read at your own pace and come back to revise any time.
• Topic-wise guided labsA step-by-step hands-on lab for every topic, with hints when you get stuck, so you learn by building and not just by watching.
• Chapter-wise quiz and flash cardsA short quiz after every chapter and spaced flash cards, so what you read actually sticks.
• Weekly assessmentsA short timed test every week on what was taught, scored on the portal, so gaps show up while there is still time to fix them.
• AssignmentsHands-on assignments to build and submit, with feedback, so you finish with real work to show.

Your path, stage by stage

6 stages, in the order you learn them. Each one has a clear goal and ends with something you build and keep. Every topic runs on your own machine first.

1. 1Code for data
2. 2Model the warehouse
3. 3Build pipelines
4. 4Process big data
5. 5Stream and ship
6. 6Capstone

1. 01

   Stage 1 of 6

   Code for data

   Write the Python and SQL that every data job runs on.

   You buildA batch ETL job from an API or CSV into a SQL database

   Python for Data Engineering (Foundation)11 topics

   Core Python Programming

   • Data types, Variables, I/O
   • Functions, Scoping, Lambda, Map/Filter
   • Control Structures, Loops
   • Exception Handling & Logging

   Data Structures

   • Lists, Tuples, Dictionaries, Sets
   • Comprehensions and Generators

   Working with Files & APIs

   • Reading and writing text, CSV, JSON, Parquet
   • Consuming REST APIs using requests
   • Working with environment variables & config files

   Essential Libraries

   • NumPy (arrays, slicing, broadcasting, vectorized operations)
   • Pandas (Series, DataFrames, indexing, grouping, pivoting, reshaping)

   SQL for Data Engineering8 topics

   Basics

   • SELECT, WHERE, ORDER BY, DISTINCT, LIMIT, ALIAS

   Intermediate SQL

   • JOINs (INNER, LEFT, RIGHT, FULL)
   • GROUP BY, HAVING
   • Subqueries, CTEs

   Advanced SQL

   • Window functions (ROW_NUMBER, RANK, LAG, LEAD)
   • Aggregations, Nested queries, Date/Time functions
   • Query optimization basics (indexes, execution plans)

   Practice

   • Real-world datasets (Sales, HR, E-commerce)
2. 02

   Stage 2 of 6

   Model the warehouse

   Shape data so reports are fast and right, with star schemas and slowly changing dimensions.

   You buildA star schema loaded into Snowflake or BigQuery

   Data Modeling & Warehousing10 topics

   Data Modeling Fundamentals

   • OLTP vs OLAP, Normalization vs Denormalization
   • Star Schema, Snowflake Schema
   • Fact tables vs Dimension tables
   • Slowly Changing Dimensions (SCD Types 1/2)

   Cloud Data Warehouses

   • Snowflake basics (architecture, virtual warehouses, loading data)
   • BigQuery basics (datasets, partitioning, clustering)
   • Redshift basics (overview & comparison)

   File Formats & Storage

   • CSV, JSON, Parquet, Avro, ORC
   • Delta Lake format (bridges into Databricks module)
   • Partitioning strategies
3. 03

   Stage 3 of 6

   Build pipelines

   Move data on a schedule with ETL and ELT, dbt and Airflow.

   You buildAn Airflow pipeline into a cloud warehouse with a dbt layer

   Data Pipelines & ETL/ELT9 topics

   ETL vs ELT Concepts

   • Batch vs streaming ingestion
   • Idempotency, incremental loads, backfills

   Data Cleaning & Transformation

   • Handling missing values, duplicates, outliers at scale
   • Schema validation and enforcement
   • Data quality checks (Great Expectations basics)

   dbt (Data Build Tool)

   • Models, sources, seeds
   • Materializations (view, table, incremental)
   • Tests and documentation
   • dbt + warehouse workflow (Snowflake/BigQuery)

   Apache Airflow (Full)14 topics

   Airflow Fundamentals

   • Architecture (Scheduler, Webserver, Executor, Metadata DB)
   • DAGs, Tasks, Operators
   • Installing and running Airflow locally / with Docker

   Building Pipelines

   • PythonOperator, BashOperator, and provider operators
   • Task dependencies, branching, trigger rules
   • Sensors, hooks, and XComs for passing data

   Scheduling & Orchestration

   • Scheduling intervals, backfills, catchup
   • Retries, SLAs, alerting (email/Slack)
   • Dynamic DAG generation

   Airflow in Practice

   • Connecting Airflow to databases, APIs, and cloud storage
   • Orchestrating a full ETL pipeline end-to-end
   • Airflow with Databricks and Spark jobs
   • Monitoring, logging, and debugging DAGs
   • Deploying Airflow (Docker Compose, Astronomer, managed services)
4. 04

   Stage 4 of 6

   Process big data

   Handle data too big for one machine with PySpark and Databricks.

   You buildA lakehouse pipeline on Databricks with Delta Lake

   PySpark & Big Data Processing6 topics

   PySpark Basics

   • Setting up SparkSession, RDDs vs DataFrames

   DataFrame API

   • Read/Write CSV, JSON, Parquet
   • Filter, Select, GroupBy, Join
   • Partitioning and performance tuning basics

   PySpark SQL & Streaming

   • Creating temporary views and writing SQL queries on Spark DataFrames
   • Structured Streaming basics (Reading from Kafka, writing to sinks)

   Databricks and the Certification Exam34 topics

   • Aligned to Databricks Certified Data Engineer Associate. Covers Lakeflow, Delta Lake, and Unity Catalog.

   Exam domain weighting

   • Databricks Intelligence Platform, 6%
   • Data Ingestion and Loading, 21%
   • Data Transformation and Modeling, 22%
   • Working with Lakeflow Jobs, 16%
   • Implementing CI/CD, 10%
   • Troubleshooting, Monitoring & Optimization, 10%
   • Governance and Security, 15%

   Databricks Intelligence Platform

   • Lakehouse architecture (Delta/Parquet)
   • Workspace components (Notebooks, Repos, Clusters, Jobs, DBFS/Volumes)
   • Compute types (All-Purpose, Job Clusters, SQL Warehouses, Serverless)

   Data Ingestion and Loading

   • Auto Loader (incremental ingestion, schema inference/evolution)
   • COPY INTO (SQL-based batch loading)
   • Lakeflow Connect & Structured Streaming fundamentals
   • Relational entities & Unity Catalog namespace

   Data Transformation and Modeling

   • Delta Lake internals (ACID, transaction log, time travel)
   • MERGE INTO, SCDs, Schema enforcement vs evolution
   • Performance optimizations (OPTIMIZE, ZORDER BY, Liquid Clustering, VACUUM)
   • Medallion architecture (Bronze → Silver → Gold)
   • ELT with Spark SQL and Python

   Working with Lakeflow Jobs

   • Lakeflow Jobs (DAG-based orchestration)
   • Lakeflow Declarative Pipelines (formerly DLT)
   • Data quality Expectations & AUTO CDC

   Implementing CI/CD

   • Databricks Repos / Git Folders
   • Databricks Asset Bundles (DABs) & Databricks CLI
   • Typical flow (Git → PR tests → Staging → Prod)

   Troubleshooting, Monitoring & Optimization

   • Reading the Spark UI (stages, tasks, skew, spill)
   • Cluster/job event logs
   • Common fixes (compaction, salting, AQE, broadcast joins)

   Governance and Security (Unity Catalog)

   • Three-level namespace (catalog.schema.table)
   • Centralized access control (GRANT/REVOKE)
   • Data lineage, Volumes, and Audit logging

   Databricks + Airflow Integration

   • Triggering Lakeflow Jobs/pipelines from Airflow
   • End-to-end lakehouse pipeline orchestration
5. 05

   Stage 5 of 6

   Stream and ship

   Process events as they arrive, and ship pipelines with Docker, Git and CI/CD.

   You buildKafka into a live Delta table, the whole pipeline containerised with CI/CD

   Streaming & Real-Time Data (Intro)3 topics

   • Messaging system concepts (Kafka basics, topics, producers, consumers)
   • Batch vs real-time architectures
   • Simple streaming pipeline (Kafka → Spark Structured Streaming → Delta table)

   Infrastructure, Version Control & CI/CD5 topics

   Git & GitHub

   • Version control basics for data pipelines, Branching strategy

   Docker

   • Containerizing pipelines and jobs, Docker Compose for local setups

   CI/CD Basics

   • GitHub Actions for testing/deploying, Automated testing (pytest, dbt tests)

   Cloud Fundamentals (Overview)

   • AWS (S3, IAM), GCP (Cloud Storage, BigQuery), Azure (Data Lake)
   • When to use managed services vs self-hosted tools
6. 06

   Stage 6 of 6

   Capstone

   Put every stage together into one pipeline you can show in an interview.

   You buildAn end to end data platform, from raw files to a clean warehouse

   Capstone Projects7 topics

   Beginner Level

   • Batch ETL Pipeline (Extract from API/CSV → clean with Pandas → load into SQL database)
   • SQL-based sales/HR data analysis with automated reporting

   Intermediate Level

   • End-to-End Airflow Pipeline, Ingest → transform → load into cloud warehouse (Snowflake/BigQuery)
   • dbt-based transformation layer on top of a warehouse, orchestrated by Airflow

   Advanced Level

   • Lakehouse Pipeline, Airflow + Databricks + Delta Lake, orchestrating multi-stage transformations
   • Real-time Pipeline, Kafka + Spark Structured Streaming + Delta table, monitored via Airflow
   • Fully Containerized Pipeline, CI/CD (Docker + GitHub Actions) deploying to cloud warehouse