This course is ideal for:

1. Data engineers working with large-scale data lakes and lakehouse architectures
2. Big data developers and architects building pipelines using Spark, Flink, and Trino
3. Data platform engineers responsible for table format management and optimization
4. Analytics engineers integrating Iceberg with BI tools and reporting platforms

1. Experience with SQL and big data query engines (e.g., Spark SQL, Flink SQL)
2. Basic understanding of data lakes and distributed file systems
3. Familiarity with formats like Parquet, ORC, and Avro
4. Basic programming experience in Python or Java for hands-on labs

Course Outline:

Module 1: Introduction to Table Formats

• What are table formats?
• Limitations of traditional data lakes (Hive tables, Parquet-only approach)
• Overview of Apache Iceberg and Delta Lake
• Lab: Compare Hive table vs Iceberg table using Spark SQL

Module 2: Apache Iceberg Fundamentals

• Architecture overview
• Table structure (metadata tree, manifest files, snapshot files)
• Iceberg table types: Partitioned, Unpartitioned
• Iceberg table spec and schema evolution
• Lab: Create and inspect Iceberg table metadata using Spark

Module 3: Partitioning and File Management

• Hidden partitioning
• Partition evolution
• Metadata management
• Lab: Demonstrate hidden partitioning and partition evolution with Iceberg

Module 4: Schema Evolution and Compatibility

• Schema updates (add/drop/rename columns)
• Backward and forward compatibility
• Lab: Perform schema evolution on an Iceberg table and validate read compatibility

Module 5: Snapshot-based Isolation

• Snapshots and time travel
• Version rollback
• Lab: Use snapshot IDs and timestamps to time travel and rollback in Iceberg

Module 6: Read and Write Engines

• Integrations with Spark, Flink, Trino, Presto, Hive
• Reading and writing Iceberg tables using Spark and Flink
• Lab: Read and write Iceberg tables using PySpark and Apache Flink

Module 7: ACID Transactions in Iceberg

• How Iceberg supports ACID semantics
• Snapshot isolation and atomic commits
• Lab: Simulate concurrent writes and validate isolation in Iceberg

Module 8: Catalog Support

• Types: HadoopCatalog, HiveCatalog, RESTCatalog, Glue
• Setting up catalogs
• Lab: Setup and use HiveCatalog and RESTCatalog in Spark

Module 9: Table Maintenance Operations

• Expire snapshots
• Rewrite manifests
• Rewrite data files (compaction)
• Lab: Run expireSnapshots, rewriteDataFiles and observe file changes

Module 10: Advanced Query Optimization

• Predicate pushdown
• File pruning and partition pruning
• Vectorized reading
• Lab: Analyze query plans and measure performance gains with pruning

Module 11: Iceberg + Parquet + ORC Avro Formats

• Supported formats and file structure
• Compatibility matrix
• Lab: Read Iceberg tables in Parquet and ORC formats using Spark

Module 12: Integration with Lakehouse & BI Tools

• Iceberg in Lakehouse architectures

Module 13: Incremental Data Processing

• Using snapshot_id, timestamp, changed_data_files
• Lab: Implement incremental ETL with Apache Flink using Iceberg

Module 14: Governance, Security, and Best Practices

• Data retention and GDPR compliance
• Table access controls
• Audit logs and lineage
• Iceberg best practices in production

Training material provided: Yes (Digital format)