In this section, we will see the most common operations that we can perform on Kafka. All Kafka tools are present under the “bin/” directory and each tool will print details on all possible command-line options if it is run with no arguments.
Let us see the cluster setup of Kafka for “Single node single broker” and “Single node multiple brokers”.
Before performing Kafka cluster setup let us start Zookeeper and Kafka broker.
Start Zookeeper Server
To start the Zookeeper Server we can use the below command. Open a new terminal and type the below command.
cloudduggu@ubuntu:~/kafka$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
Open another terminal and start Kafka Broker using the below command.
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-server-start.sh config/server.properties
Now we can verify daemons of Zookeeper and Kafka Broker using the “JPS” command.
cloudduggu@ubuntu:~/kafka$ jps
Now let us see Kafka Single Node-Single Broker Configuration.
1. Kafka Single Node-Single Broker Configuration
In this Kafka configuration, we have a single Zookeeper and broker ID instance.
Now let us see the steps to configure it.
1.1 Create Kafka Topic
We can create topics on the server by using the Kafka command-line tool “kafka-topics.sh”. Type below command to create Kafka topic.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic topic_name
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic Kafka_Tutorial
A topic has been created with the name “Kafka_Tutorial”.
Output:
1.2 List Kafka Topic
List command is used to provide a list of topics on the Kafka server.
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --list --zookeeper localhost:2181
We can see a topic has been created with the name “Kafka_Tutorial”.
Output:
1.3 Modify Kafka Topic
We can modify the already created topic using the “—Alter” parameter. We have already created a topic with the name “Kafka_Tutorial”. Let us modify the partition of this topic.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic_name --partitions count
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic Kafka_Tutorial --partitions 3
We can see the topic has been altered with three partition values.
Output:
1.4 Start Kafka Producer to Send Messages
To send messages using producer we are using the “kafka-console-producer.sh” tool which is present under “/bin”. Kafka Producer needs two-parameter as mentioned below and its configuration detail is mentioned in the “config/producer.properties” file.
- Broker-list: It is the list of the broker on which we are going to send the message. The port detail of the broker is present under the “config/server.properties” configuration file. In our case broker is listening on 9092 port.
- Topic: It contains the name of the topic.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic_name
Open a new terminal and type the below command. Once you type and press enter the producer will wait for an input, type a message and press enter it will again ask you to type messages.
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Kafka_Tutorial
Output:
Now let us start Kafka consumer to receive messages sent by Kafka producer.
1.5 Start Kafka Consumer to Receive Messages
To receive a message using consumer we are using the “kafka-console-consumer.sh” tool. The configuration detail of Kafka consumer is mentioned in “config /consumer.properties”.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_name --from-beginning
Open a new terminal and type the below command to receive messages from the Kafka producer.
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Kafka_Tutorial --from-beginning
From the below output we can see messages are delivered.
Output:
1.6 Delete Kafka Topic
We can delete the already created topic using the “—Delete” parameter. We have already created a topic with the name “Kafka_Tutorial”. Let us use the delete parameter to delete this topic.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic topic_name
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic Kafka_Tutorial
From the below output we can see the topic has been deleted.
Output:
2. Kafka Single Node-Multiple Brokers Configuration
In this Kafka configuration, we will create multiple brokers. We already have one broker present under the “config/server.properties” configuration file. We will copy this file and create two more files with the name “server_1.properties” and “server_2.properties” configuration file.
Please use the below commands to perform this operation.
cloudduggu@ubuntu:~/kafka/config$ cp server.properties server_1.propertiescloudduggu@ubuntu:~/kafka/config$ cp server.properties server_2.properties
Now edit both configuration files and put the below parameters.
For “server_1.properties” add the below parameters.
broker.id=1
port=9093
log.dirs=/tmp/kafka_1-logs
For “server_2.properties” add the below parameters.
broker.id=2
port=9094
log.dirs=/tmp/kafka_2-logs
After performing this activity start all three brokers using below commands.
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-server-start.sh config/server.propertiescloudduggu@ubuntu:~/kafka$ ./bin/kafka-server-start.sh config/server_1.properties
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-server-start.sh config/server_2.properties
2.1 Create Kafka Topic
We will use the Kafka tool “kafka-topics.sh” to create the topic. We will use replication factor value three as three brokers are running. If there are two brokers then you can use replication factor two.
Syntax:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic topic_name
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Kafka_Multi_Brocker
Output:
2.2 Start Kafka Producer to Send Messages
Output:
2.2 Start Kafka Producer to Send Messages
The process will be the same which was explained in the single broker configuration.
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Kafka_Multi_Brocker
Output:
2.3 Start Kafka Consumer to Receive Messages
2.3 Start Kafka Consumer to Receive Messages
The process will be the same which was explained in the single broker configuration.
Command:
cloudduggu@ubuntu:~/kafka$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Kafka_Multi_Brocker --from-beginning
Output:
