This course is ideal for:

1. Data Engineers and Software Developers
2. DevOps and Platform Engineers managing Kafka clusters
3. Streaming and Real-Time Analytics Professionals
4. Architects designing event-driven systems

1. Familiarity with Java (or any another programming language)
2. Basic understanding of distributed systems
3. Exposure to Linux command line

Course Outline:

Module 1: Big Data and Kafka Introduction

1. 3Vs of Big Data and evolution
2. Real-time vs batch processing
3. Industry use cases

Module 2: Overview of Hadoop & Streaming Frameworks

1. Apache Hadoop: HDFS and MapReduce overview
2. Spark and Storm for real-time data analysis

Module 3: Introduction to Zookeeper

1. Zookeeper basics and recipes (leader election, locking)
2. Installing/configuring Zookeeper
3. Hands-on: Ubuntu setup, SSH, and connecting with PuTTY

Module 4: Introduction to Apache Kafka

1. Kafka design principles
2. Kafka as a distributed commit log
3. Partitions, topics, replication, and message queues
4. Publish-subscribe pattern explained

Module 5: Kafka Installation and Configuration

1. Installing Kafka and configuring brokers
2. Cluster setup using Zookeeper
3. Hands-on: Installing and configuring Kafka + Zookeeper

Module 6: Kafka Producers and Consumers

1. Producer API and serializers (Avro, JSON, Protobuf)
2. Consumer API, groups, offsets, and delivery semantics
3. At-most-once, at-least-once, exactly-once guarantees

Module 7: Kafka Data Model and Java Integration

1. Topics, partitions, and offsets in practice
2. Compiling and running Kafka Java clients
3. Hands-on: Writing Kafka Producer and Consumer apps

Module 8: Kafka Streams API

1. Difference between batch and stream processing
2. Kafka Streams API: KStreams, KTables, stateful operations
3. Event-time processing and windowing
4. Hands-on: Real-time word count and stream joins

Module 9: Kafka Connect and Data Integration

1. Kafka Connect architecture
2. Source/sink connectors: JDBC, S3, HDFS, Elasticsearch
3. Building ETL-style pipelines using Kafka Connect
4. Hands-on: Configuring Kafka Connect with databases

Module 10: Real-World Kafka Development Lab

1. Build a full pipeline from producer → Kafka → stream processor → sink
2. Hands-on: End-to-end pipeline using Kafka Streams + Connect

Module 11: Kafka Cluster Operations

1. Broker lifecycle: adding/removing brokers
2. Topic configuration, replication, and leader election
3. Monitoring clusters using metrics and logs

Module 12: Kafka Security

1. Authentication with SSL and SASL
2. Access control with ACLs
3. Encryption at rest and in transit
4. Hands-on: Securing Kafka brokers and clients

Module 13: Kafka Performance Tuning

1. Tuning producers and consumers for throughput
2. Compression, batching, and partitioning strategies
3. Broker-level tuning

Module 14: Zookeeper for Kafka Operations

1. Zookeeper’s role in Kafka cluster metadata
2. Best practices and operational tips
3. Securing Zookeeper nodes

Module 15: Monitoring and Troubleshooting

1. Tools: Kafka Manager, Grafana, Prometheus
2. Common issues: unbalanced partitions, lag, GC, leader election issues
3. Hands-on: Benchmarking performance, tuning parameters, viewing metrics

Training Material Provided:

Course slides and reference guides