Topic Outline:

• Review Hadoop architecture
• Understand the fundamental architecture of Spark
• Use the core Spark APIs to operate on data
• Understand the Spark Ecosystem
• Use PySpark and Python
• Create RDDs
• Create Dataframes
• Use RDDs and DataFrames
• Use SQL on DataFrames
• Use Jupyter notebook

Course Outline:

Day 1 Foundations

 Overview

• Lab Environment
• PySpark
• Spark Data

Day 2 Programming Techniques

• Spark Programming
• RDDs
• DataFrames
• Jupyter Notebook

Detail Course Topics

Day 1 Foundations

• Big Data and History
• Big Data & Hadoop Deployment
• Hadoop
• Architecture
• Understanding Storage and Processing with Hadoop
• Map Reduce and its limitations
• Eco System with Hadoop
• Spark
• Why Spark?
• Spark Eco System
• Spark Architecture
• Spark Cluster
• Understanding RDD and its value
• Spark Dataframes and Spark SQL
• Understanding PySpark
• Multiple Hands-on
• Using Jupyter Notebook
• Spark command line
• RDD and DataFrame using PySpark
• Assignment

Day 2 Programming Techniques

• RDD and DataFrames programming
• Understanding Spark internals when using RDD and DataFrames
• Schema definition
• Schema definition
• Partitioning
• User defined Functions
• Handling Corrupt Records
• Basic ETL task using PySpark
• Handling data transformations for downstream processing
• Multiple Hands-on
• Hands on Jupyter Notebooks with exercises
• Assignment Review
• Lab Environment
• Each student will be provided a virtual machine for performing hands-on labs, they are expected to use these machines in class
• These machines will be configured for Spark-2.4.7 release
• Instructions will be provided to students for setting up environment on their machines (there will be no support for debugging their environment