11 Steps to Setup Apache Kafka on Ubuntu 20.04 LTS

Apache Kafka is a free & open-source software platform developed by the Apache Software Foundation. It is written in Scala and Java .It helps to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It is used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

There are few steps to setup Apache Kafka on ubuntu:

Step 1: Update the System.

apt-get update

Step 2: Install Java.

apt-get install default-jdk

  • Check the Java version.

java -version

  • Here is the command output.

Step 3: Download the Apache Kafka.

wget https://www.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz

  • Here is the command output.

  • Extract the downloaded file.

tar xzf kafka_2.13-2.7.0.tgz

Step 4: Move the extracted folder to /usr/local/.

mv kafka_2.13-2.7.0 /usr/local/kafka

Step 5: Create a new systemd unit files for Zookeeper and Kafka service.

  • For Zookeeper service:

vim /etc/systemd/system/zookeeper.service

  • Add the following lines:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target

  • For Kafka service:

vim /etc/systemd/system/kafka.service

  • Add the following lines:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target

Step 6: Reload the systemd daemon & Start the ZooKeeper service.

systemctl daemon-reload
systemctl start zookeeper

  • Here is the command output.

Step 7: Start & Enable the Kafka service.

systemctl start kafka
systemctl enable kafka

  • Check the Kafka service status.

systemctl status kafka

  • Here is the command output.

Step 8:To create a topic in Kafka.

  • Go to /usr/local/kafka directory.

cd /usr/local/kafka

  • Run the following command for creating the Topic.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Topic-name (For example: exampleTopic)

  • Here is the command output.

Step 9: List the created Topic.

bin/kafka-topics.sh --list --zookeeper localhost:2181

  • Here is the command output.

Step 10: To send Messages to Kafka cluster.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Topic-name

  • Here is the command output.

  • To input the data.

Step 11: To Read the Kafka cluster data.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Topic-name --from-beginning

  • Here is the command output.

Leave a Reply