START HERE: Learn Apache Kafka 3.0 Ecosystem, Core Concepts, Real World Java Producers/Consumers & Big Data Architecture

What you'll learn
Understand Apache Kafka Ecosystem, Architecture, Core Concepts and Operations
Master Concepts such as Topics, Partitions, Brokers, Producers, Consumers
Start a personal Kafka development environment
Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
Create your Producers and Consumers in Java to interact with Kafka
Program a Real World Twitter Producer & ElasticSearch Consumer
Extended APIs Overview (Kafka Connect, Kafka Streams), Case Studies and Big Data Architecture
Practice and Understand Log Compaction
Requirements
A recent Windows / Mac / Linux machine with minimum 4GB of RAM, 5 GB of disk space
Some understanding of Java Programming
Good to have knowledge about Linux command line
Desire to learn something awesome and new!
Description
UPDATE MARCH 2022: Course newly recorded with Kafka 3.0!Welcome to the Apache Kafka Series! Join a community of 130,000+ students learning Kafka.Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart and LinkedIn. To learn Kafka easily, step-by-step, you have come to the right place! No prior Kafka knowledge is required=If you look at the documentation, you can see that Apache Kafka is not easy to learn.Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone. We'll take a step-by-step approach to learn all the fundamentals of Apache Kafka. At the end of this course, you'll be productive and you'll know the following:The Apache Kafka Ecosystem ArchitectureThe Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more!Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux Learn and Practice using the Kafka Command Line Interface (CLI)Code Producer and Consumers using the Java API Real world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumerOverview of Advanced APIs (Kafka Connect, Kafka Streams)Real World Case Studies and Big Use casesOverview of Advanced Kafka for AdministratorsAdvanced Topic ConfigurationsNote: The hands-on section is based on Java, which is the native Kafka programming language. But, good news! Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka=The Kafka for Beginners course is the best place to start your Kafka learning journey. Ask other students!5 stars: "That's one of the most high-quality on-line courses I ever took" – Paul L.5 stars: "This training was awesome and learned many things about KAFKA though having 0 years of experience in this technology" – Puneet G.5 stars: "Outstanding on all fronts! I wish all courses were that well presented. I especially appreciate the hands-on sections. I was able to get everything up and running with ease. Highly recommend this course and this instructor!" – David G. You can take this course risk-free and if you don't like it, you can get a refund anytime in the first 30 days! =InstructorMy name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let's kick start the course! You are in good hands!=This Course Also Comes With:Lifetime Access to All Future UpdatesA responsive instructor in the Q&A SectionLinks to interesting articles, and lots of good code to base your next applications ontoUdemy Certificate of Completion Ready for DownloadThis is the course that could improve your career!Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!I hope to see you inside the course!===Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:Learn Kafka for Beginners v2 (great to start)Kafka Connect Hands On LearningKafka Streams for Data ProcessingKSQL on ksqlDB - Hands On!Kafka Cluster Setup & AdministrationConfluent Schema Registry & Kafka REST ProxyKafka Security (SSL SASL ACL)Kafka Monitoring and OperationsHappy learning!

Overview

Section 1: Kafka Introduction

Lecture 1 Course Introduction

Lecture 2 Apache Kafka in 5 minutes

Lecture 3 Course Objectives

Lecture 4 Welcome! - About your instructor

Section 2: Code Download

Lecture 5 Code Download

Section 3: ====== Kafka Fundamentals ======

Lecture 6 Kafka Fundamentals

Section 4: Kafka Theory

Lecture 7 Topics, Partitions and Offsets

Lecture 8 Producers and Message Keys

Lecture 9 Consumers & Deserialization

Lecture 10 Consumer Groups & Consumer Offsets

Lecture 11 Brokers and Topics

Lecture 12 Topic Replication

Lecture 13 Producer Acknowledgements & Topic Durability

Lecture 14 Zookeeper

Lecture 15 Kafka KRaft - Removing Zookeeper

Lecture 16 Theory Roundup

Section 5: Starting Kafka

Lecture 17 Important: Starting Kafka & Lectures Order

Lecture 18 FAQ for Setup Problems

Lecture 19 Starting Kafka with Conduktor - Multi Platform

Lecture 20 Mac OS X - Download and Setup Kafka in PATH

Lecture 21 Mac OS X - Start Zookeeper and Kafka

Lecture 22 Mac OS X - Using brew

Lecture 23 Linux - Download and Setup Kafka in PATH

Lecture 24 Linux - Start Zookeeper and Kafka

Lecture 25 Windows WSL2 - Download Kafka and PATH Setup

Lecture 26 Windows WSL2 - Start Zookeeper & Kafka

Lecture 27 Windows WSL2 - How to Fix Problems

Lecture 28 Windows WSL2 - Extra Instructions

Lecture 29 Windows non-WSL2 - Start Zookeeper and Kafka

Section 6: Starting Kafka without Zookeeper

Lecture 30 Note: not ready for production yet

Lecture 31 Mac OS X - Start Kafka in KRaft mode

Lecture 32 Linux - Start Kafka in KRaft mode

Lecture 33 Windows WSL2 - Start Kafka in KRaft mode

Section 7: CLI (Command Line Interface) 101

Lecture 34 CLI Introduction

Lecture 35 WINDOWS WARNING: PLEASE READ

Lecture 36 Kafka Topics CLI

Lecture 37 Kafka Console Producer CLI

Lecture 38 Kafka Console Consumer CLI

Lecture 39 Kafka Consumers in Group

Lecture 40 Kafka Consumer Groups CLI

Lecture 41 Resetting Offsets

Section 8: Kafka UI

Lecture 42 Conduktor - Demo

Section 9: Kafka Java Programming 101

Lecture 43 Kafka SDK List

Lecture 44 Creating Kafka Project

Lecture 45 Java Producer

Lecture 46 Java Producer Callbacks

Lecture 47 Java Producer with Keys

Lecture 48 Java Consumer

Lecture 49 Java Consumer - Graceful Shutdown

Lecture 50 Java Consumer inside Consumer Group

Lecture 51 Java Consumer Incremental Cooperative Rebalance & Static Group Membership

Lecture 52 Java Consumer Incremental Cooperative Rebalance - Practice

Lecture 53 Java Consumer Auto Offset Commit Behavior

Lecture 54 Programming - Advanced Tutorials

Section 10: ===== Kafka Real World Project =====

Lecture 55 Real World Project Overview

Lecture 56 Real World Exercise - Solution

Section 11: Kafka Wikimedia Producer & Advanced Producer Configurations

Lecture 57 Wikimedia Producer Project Setup

Lecture 58 Wikimedia Producer Implementation

Lecture 59 Wikimedia Producer Run

Lecture 60 Wikimedia Producer - Producer Config Intros

Lecture 61 Producer Acknowledgements Deep Dive

Lecture 62 Producer Retries

Lecture 63 Idempotent Producer

Lecture 64 Safe Kafka Producer Settings

Lecture 65 Wikimedia Producer Safe Producer Implementation

Lecture 66 Kafka Message Compression

Lecture 67 linger.ms and batch.size Producer settings

Lecture 68 Wikimedia Producer High Throughput Implementation

Lecture 69 Producer Default Partitioner & Sticky Partitioner

Lecture 70 [Advanced] max.block.ms and buffer.memory

Section 12: OpenSearch Consumer & Advanced Consumer Configurations

Lecture 71 OpenSearch Consumer - Project Overview

Lecture 72 OpenSearch Consumer - Project Setup

Lecture 73 Setting up OpenSearch on Docker

Lecture 74 Setting up OpenSearch on the Cloud

Lecture 75 OpenSearch 101

Lecture 76 OpenSearch Consumer Implementation - Part 1

Lecture 77 OpenSearch Consumer Implementation Part 2

Lecture 78 Consumer Delivery Semantics

Lecture 79 OpenSearch Consumer Implementation Part 3 - Idempotence

Lecture 80 Consumer Offsets Commit Strategies

Lecture 81 OpenSearch Consumer Implementation Part 3 - Delivery Semantics

Lecture 82 OpenSearch Consumer Implementation Part 5 - Batching Data

Lecture 83 Consumer Offset Reset Behavior

Lecture 84 OpenSearch Consumer Implementation Part 6 - Replaying Data

Lecture 85 Consumer Internal Threads

Lecture 86 Consumer Replica Fetching - Rack Awareness

Section 13: Kafka Extended APIs for Developers

Lecture 87 Kafka Extended APIs - Overview

Lecture 88 Kafka Connect Introduction

Lecture 89 Kafka Connect Hands On: Warning

Lecture 90 Kafka Connect Wikimedia & ElasticSearch Hands On

Lecture 91 Kafka Streams Introduction

Lecture 92 Kafka Streams Hands-On

Lecture 93 Kafka Schema Registry Introduction

Lecture 94 Kafka Schema Registry Hands On

Lecture 95 Which Kafka API should I use?

Section 14: Real World Insights and Case Studies (Big Data / Fast Data)

Lecture 96 Choosing Partition Count & Replication Factor

Lecture 97 Kafka Topics Naming Convention

Lecture 98 Case Study - MovieFlix

Lecture 99 Case Study - GetTaxi

Lecture 100 Case Study - MySocialMedia

Lecture 101 Case Study - MyBank

Lecture 102 Case Study - Big Data Ingestion

Lecture 103 Case Study - Logging and Metrics Aggregation

Section 15: Kafka in the Enterprise for Admins

Lecture 104 Kafka Cluster Setup High Level Architecture Overview

Lecture 105 Kafka Monitoring & Operations

Lecture 106 Kafka Security

Lecture 107 Kafka Multi Cluster & MirrorMaker

Lecture 108 Advertised Listeners: Kafka Client & Server Communication Protocol

Section 16: ===== Advanced Kafka =====

Lecture 109 Advanced Kafka

Section 17: Advanced Topics Configurations

Lecture 110 Changing a Topic Configuration

Lecture 111 Segment and Indexes

Lecture 112 Log Cleanup Policies

Lecture 113 Log Cleanup Delete

Lecture 114 Log Compaction Theory

Lecture 115 Log Compaction Practice

Lecture 116 Unclean Leader Election

Lecture 117 Large Messages in Kafka

Section 18: ===== Next Steps =====

Lecture 118 What's Next?

Lecture 119 THANK YOU!

Lecture 120 Bonus Lecture

Developers who want to learn the Apache Kafka Fundamentals, start a cluster and write their first application,Architects who want to understand how Apache Kafka fits into their solution architecture,Anyone looking to learn the full theory of how Apache Kafka works as a distributed system