Course duration
- 2 days
Course Benefits
- Understand the use of Kafka for high performance messaging.
- Identify the usages for Kafka in Microservices.
- Explain the benefits of Kafka patterns.
- Differentiate between messaging and message brokers.
- Describe Kafka messaging environments.
- Develop producers and consumers for Kafka.
- Recognize how Kafka enables Cloud-native applications.
- Summarize characteristics and architecture for Kafka.
- Demonstrate how to process messages with Kafka.
- Design distributed high throughput systems based on Kafka.
- Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka.
Course Outline
- Introduction to KAFKA
- Messaging Architectures – What is Messaging?
- Messaging Architectures – Steps to Messaging
- Messaging Architectures – Messaging Models
- What is Kafka?
- What is Kafka? (Contd.)
- Kafka Overview
- Need for Kafka
- When to Use Kafka?
- Kafka Architecture
- Core concepts in Kafka
- Kafka Topic
- Kafka Partitions
- Kafka Producer
- Kafka Consumer
- Kafka Broker
- Kafka Cluster
- Why Kafka Cluster?
- Sample Multi-Broker Cluster
- Overview of ZooKeeper
- Kafka Cluster & ZooKeeper
- Who Uses Kafka?
- Summary
- Using Apache Kafka
- Installing Apache Kafka
- Configuration Files
- Starting Kafka
- Using Kafka Command Line Client Tools
- Setting up a Multi-Broker Cluster
- Using Multi-Broker Cluster
- Kafka Connect
- Kafka Connect – Configuration Files
- Using Kafka Connect to Import/Export Data
- Creating a Spring Boot Producer
- Adding Kafka dependency to pom.xml
- Defining a Spring Boot Service to Send Message(s)
- Defining a Spring Boot Controller
- Testing the Spring Boot Producer
- Creating a Nodejs Consumer
- Summary
- Building Data Pipelines
- Building Data Pipelines
- Considerations When Building Data Pipelines
- Timeliness
- Reliability
- High and Varying Throughput
- High and Varying Throughput (Contd.)
- Data Formats
- Data Formats (Contd.)
- Transformations
- Transformations (Contd.)
- Security
- Failure Handling
- Coupling and Agility
- Ad-hoc Pipelines
- Loss of Metadata
- Extreme Processing
- Kafka Connect Versus Producer and Consumer
- Kafka Connect Versus Producer and Consumer (Contd.)
- Summary
- Integrating Kafka with Other Systems
- Introduction to Kafka Integration
- Kafka Connect
- Kafka Connect (Contd.)
- Running Kafka Connect
- Key Configurations for Connect workers:
- Kafka Connect API
- Kafka Connect Example – File Source
- Kafka Connect Example – File Sink
- Kafka Connector Example – MySQL to Elasticsearch
- Kafka Connector Example – MySQL to Elasticsearch (Contd.)
- Write the data to Elasticsearch
- Building Custom Connectors
- Kafka Connect – Connectors
- Kafka Connect - Tasks
- Kafka Connect - Workers
- Kafka Connect – Workers (Contd.)
- Kafka Connect - Converters and Connect’s data model
- Kafka Connect - Offset management
- Alternatives to Kafka Connect
- Alternatives to Kafka Connect (Contd.)
- Introduction to Storm
- Other Components of Spark
- Integrating Storm with Kafka
- Integrating Storm with Kafka – Sample Code
- Integrating Storm with Kafka
- Introduction to Hadoop
- Hadoop Components
- Integrating Hadoop with Kafka
- Hadoop Consumers
- Hadoop Consumers (Contd.)
- Hadoop Consumers (Contd.)
- Hadoop Consumers – Produce Topic
- Hadoop Consumers – Fetch Generated Topic
- Summary
- Kafka Security
- Kafka Security
- Encryption and Authentication using SSL
- Encryption and Authentication using SSL (Contd.)
- Configuring Kafka Brokers
- Configuring Kafka Brokers – Optional Settings
- Authenticating Using SASL
- Authenticating Using SASL – Configuring Kafka Brokers
- Authenticating Using SASL – Configuring Kafka Brokers (Contd.)
- Authorization and ACLs
- Authorization and ACLs (Contd.)
- Securing a Running Cluster
- Securing a Running Cluster (Contd.)
- ZooKeeper Authentication
- ZooKeeper Authentication (Contd.)
- Summary
- Monitoring Kafka
- Introduction
- Metrics Basics
- JVM Monitoring
- Garbage collection
- Garbage Collection (Contd.)
- Java OS monitoring
- OS Monitoring
- OS Monitoring (Contd.)
- Kafka Broker Metrics
- Under-Replicated Partitions
- Active controller count
- Request handler idle ratio
- Intelligent Thread Usage
- All topics bytes in
- All topics bytes out
- All topics messages in
- Partition count
- Leader count
- Offline partitions
- Request metrics
- Request Metrics (Contd.)
- Logging
- Logging (Contd.)
- Client Monitoring
- Producer Metrics
- Overall producer metrics
- Overall producer metrics (Contd.)
- Per-broker and per-topic metrics
- Consumer Metrics
- Fetch Manager Metrics
- Per-broker and per-topic metrics
- Consumer coordinator metrics
- Quotas
- Quotas (Contd.)
- Lag Monitoring
- Lag Monitoring (Contd.)
- End-to-End Monitoring
- Summary
- Kafka Operational Aspects
- Kafka Administration
- Kafka Cluster Management
- Kafka Replica Distribution
- Partition Reassignment
- Partition Reassignment (Contd.)
- Kafka Topic Management
- Kafka Topic Management (Contd.)
- Kafka Cluster Mirroring
- Kafka Cluster Mirroring (Contd.)
- Integration with Other Tools
- Summary
Class Materials
Each student will receive a comprehensive set of materials, including course notes and all the class examples.
Class Prerequisites
Experience in the following would be useful for this Microservices class:
- Basic understanding of messaging, cloud, development, architecture, and virtualization would be beneficial.
Since its founding in 1995, InterSource has been providing high quality and highly customized training solutions to clients worldwide. With over 500 course titles constantly updated and numerous course customization and creation possibilities, we have the capability to meet your I.T. training needs.
Instructor-led courses are offered via a live Web connection, at client sites throughout Europe, and at our Geneva Training Center.
Instructor-led courses are offered via a live Web connection, at client sites throughout Europe, and at our Geneva Training Center.