• This course is designed for developers and data analysts.

• Introduction to Apache Spark version 3.x.

Course Outline:

 Day 1: Spark Libraries 

• Continue understanding Spark Dataframes
• Look up tables and Joins with Spark Dataframes
• Understanding data partitioning on Dataframes
• Data transformations with pipelines
• Understanding Machine Learning Use cases and Techniques
• Understand what is machine learning?
• Machine learning development v/s traditional software development
• Importance of data in machine learning
• Machine Learning Development
  • Learn about the steps involved in machine learning development
  • Understand how the machine learns?
  • Machine Learning Algorithms and tools supported in Spark ML Library
• Building Classification and Regression Models
  • Building regression models and evaluating model performance
  • Building classification models and evaluating model performance
  • Understanding feature engineering
  • Model persistence
• Multiple Hands-on
  • Using PySpark on Spark Dataframes
  • Data transformations on dataframe
  • Extra Credits

Day 2: Machine Learning Library and Streaming

• Data clustering with Spark Machine Learning library
  • Perform data clustering using Spark Machine learning library
  • Understand finding optimal clusters on a dataset
• Streaming Data
  • Understand what is streaming data?
  • Design Challenges with Streaming data
• Structured Streaming with Spark
  • Spark library streaming history from Spark 1.x to Spark 3.x
  • Enhancements to Structured Streaming library
  • Learn about streaming output modes
  • Perform aggregation on streaming data
• Spark ML library and Streaming library
  • Build machine learning model and persist it
  • Load model and use it to make predictions on streaming data
• Multiple Hands-on
  • Multiple hands-on sessions on the above topics
  • Extra Credits session

Lab Environment

• Each student will be provided a virtual machine for performing hands-on labs, they are expected to use these machines in class
• These machines will be configured for Spark-3.x release
• Instructions will be provided to students for setting up environment on their machines (there will be no support for debugging their environment)

Training material provided: Yes (Digital format)