• People being curious about data engineering.
• People who want to learn basic and advanced concepts about Apache Airflow.
• People who like hands-on approach.

• Although there is no background needed except basic Python knowledge or object-oriented programming experience, any knowledge of Machine Learning can help boost your learning.

Course Outline:

The scalable problem of Machine Learning Pipelines

• What problems arise when trying to create a Machine Learning model?
• The components of a Machine Learning platform
• Introducing Apache Airflow
• Airflow architecture
• How do we represent a Machine Learning Pipeline?
• Demo: Our first DAG
• Tasks, TaskFlows, and Operators
• Demo: First Pipeline
• Capstone Lab: Cresting the datasets for training

Creating our Machine Learning Pipeline

• Using custom operators
• Demo: Creating a Train Operator
• Creating TaskGroups vs subDAGs
• Sharing data with xCOMs
• Branching and Triggers
• Sensors and SmartSensors
• Demo: Adding a sensor to validate enough new data
• Capstone Lab: Adding training, validation and delivery steps to our pipeline

Mastering scheduling

• Execution_date, start_date and schedule_interval
• Handling non-default schedule_intervals
• Demo: Playing with time
• Capstone Lab: Using Sensors with a correct schedule_interval

Enabling concurrency and scalability

• Abandoning SQLite to PostgreSQL
• Executors: Debug, Local, Celery
• Concurrency and parallelism
• Demo: Concurrency with Celery

Hackathon: Sentiment Prediction from Twitter

Software Required

This Apache Airflow for Machine Learning Operations course is taught using Python > 3.5, Apache Airflow > 2.1, scikit-learn > 1.1, and PyTorch  > 1.8. On request, we can provide either a remote VM environment for the class or directions for configuring this environment on your local PCs.

Training material provided: Yes (Digital format)