1. Data Engineers looking to optimize Spark applications for production
2. Data Scientists and Analysts seeking advanced data processing and transformation skills
3. Developers transitioning from traditional batch processing to distributed analytics

1. Completion of BDT167 (Beginner Apache Spark) or equivalent hands-on Spark experience
2. Working knowledge of Python, Jupyter notebooks, and pandas
3. Familiarity with Hadoop/distributed computing concepts and SQL queries

Course Outline:

Day 1: Spark 3.X Architecture & Advanced RDDs

Module 1: Spark 3.X Architecture Deep Dive

1. Spark 3.X enhancements: Performance improvements and new features
2. Driver-Executor architecture and task scheduling mechanisms
3. Catalyst query optimizer: Rule-based and cost-based optimization
4. Tungsten project: Off-heap memory management and binary encoding

Module 2: RDD Internals & Advanced Operations

1. RDD DAG (Directed Acyclic Graph) and lineage tracking for fault tolerance
2. Partitioning strategies: Hash, range, and custom partitioners
3. Shuffle operations: Understanding reduce-side joins, group-by-key, and coalesce
4. RDD persistence, caching strategies, and storage levels
5. Hands-on: Optimizing RDDs with repartitioning and custom partitions

Day 2: DataFrames/Datasets & Advanced SQL

Module 3: DataFrame & Dataset APIs Mastery

1. DataFrame vs RDD: Trade-offs and when to use each
2. Dataset API: Type safety, encoders, and performance characteristics
3. Complex transformations: Window functions, lateral views, and pivot operations
4. Joins: Inner, outer, semi, anti, cross—optimization and broadcast patterns
5. Aggregations: GroupBy, cube, rollup, and custom aggregation functions (UDAF)
6. Hands-on: Building multi-step ETL pipelines with DataFrames

Module 4: Advanced Spark SQL & Query Optimization

1. Catalyst Query Optimizer: Physical plan generation and execution
2. EXPLAIN plans: Analyzing query execution and identifying bottlenecks
3. Advanced SQL: CTEs, recursive queries, subqueries, and analytical functions
4. Columnar storage: Parquet, ORC formats and compression trade-offs
5. Query hints and hint-based optimization techniques
6. Hands-on: Query tuning, index strategies, and performance benchmarking