Hope you are clear about What is Kafka? through my previous post. Continue, we will go through how we can install Apache Kafka on Windows and how to create topic, producer, consumer in Kafka.
Step 1: Download JDK, if your PC has installed JDK, you can skip this step.
You can download JDK at https://www.oracle.com/java/technologies/javase-downloads.html. To check JDK installed successfully you type java -version in terminal like this
Step2: Download and install Apache Kafka binary
Download Apache Kafka at https://kafka.apache.org/downloads and remember choose binary version, don’t choose source version.
Extract Apache Kafka to C:/kafka folder or any folder that you like, then we proceed to configure Kafka log folder and ZooKeeper folder.
Create data folder in C:/kafka then create kafka folder contains kafka server logs and zookeeper folder contains log data in this folder.
Open config/server.properties and change log.dirs to the previous created kafka folder.
Open config/zookeeper.properties and change dataDir to the previous created data folder
Step 3: Add Kafka bin folder to your Path environment variable.
Add C:\kafka\bin\windows to your Path enviroment variable
Step 4: Start ZooKeeper and Apache Kafka
Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc. Zookeeper it self is allowing multiple clients to perform simultaneous reads and writes and acts as a shared configuration service within the system. The Zookeeper atomic broadcast (ZAB) protocol is the brains of the whole system, making it possible for Zookeeper to act as an atomic broadcast system and issue orderly updates.
Start ZooKeeper by change your directory to C:\kafka\config and execute the following command
cd C:\kafka\config zookeeper-server-start zookeeper.properties
And make sure ZooKeeper started successfully (yellow highlight in the following figure). ZooKeeper’ll listen on 2181 port.
Start Apache Kafka
Open new command prompt, then navigate to config folder and Start Apache Kafka by the following command
After started successfully, Kafka will create a broker and listen on port 9092.
Step 5: Create Kafka topic
Now we will create a topic with the name “test” and a replication factor of 1 because we have only one Kafka server running.
What is Kafka replication factor?
A replication factor is the number of copies of data over multiple brokers. The replication factor value should be greater than 1 always (between 2 or 3). This helps to store a replica of the data in another broker from where the user can access it.
For example, suppose we have a cluster containing three brokers: Broker 1, Broker 2, and Broker 3. A topic, namely Topic-X is split into Partition 0 and Partition 1 with a replication factor of 2 as the following figure.
If you have a cluster with more than one Kafka server running, you can increase the replication-factor accordingly, which will increase the data availability and act like a fault-tolerant system.
Open new command prompt and type the following command to create topic test with only have 1 partition
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Step 6: Create Kafka producer
Open new command prompt and create Kafka producer subscribe test topic by the following command:
kafka-console-producer --broker-list localhost:9092 --topic test
Step 7: Create Kafka consumer
Next we will create a consumer subscribe topic test. When you create a consumer without specifying consumer group, Kafka will assign a default group for your consumer.
Open new commadn prompt and type the following command to create a consumer
kafka-console-consumer --bootstrap-server localhost:9092 --topic test
if you want to assign a group for your consumer, you need add –group <group_name> to the command
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --group my-created-consumer-group
Now when you type any character on producer terminal, you will receive same message on consumer terminal.
Suppose we add 2 consumers to same group (test-group), at this time, if you send any message on producer, you will receive same message on only a consumer instead of load balancing because we only have a partition.
Now we will create a topic with 2 partitions by the following command:
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic
You can review infomation of the created topic by the command
kafka-topics --zookeeper 127.0.0.1:2181 --topic test2partitions --describe
Next we will create 2 consumers subscribe test2partitions topic, 2 consumers will be assigned same group.
kafka-console-consumer --bootstrap-server localhost:9092 --topic test2partitions --group test-group
Some popular commands Kafka
To get messages from the beginning of the subscribed topic you need to add –from-beginning to the create consumer command like this
kafka-console-consumer --bootstrap-server localhost:9092 --test --from-beginning
Delete test topic from all brokers
kafka-run-class kafka.admin.TopicCommand --delete --test --zookeeper localhost:2181