Kafka Not Authorized to Read From Group Id

Kafka is an open-source distributed messaging system to transport the bulletin in partitioned and unlike topics. Real-time data streaming tin be implemented by using Kafka to receive data between the applications. Information technology has three major parts. These are producer, consumer, and topics. The producer is used to send a bulletin to a detail topic and each message is fastened with a key. The consumer is used to read a message on a detail topic from the set of partitions. The data received from the producer and stored on the partitions based on a item topic. Many libraries exist in python to create producer and consumer to build a messaging system using Kafka. How the information from Kafka can exist read using python is shown in this tutorial.

Prerequisite

Y'all have to install the necessary python library to read data from Kafka. Python3 is used in this tutorial to write the script of consumer and producer. If the pip package is not installed before in your Linux operating system and so you have to install pip before installing the Kafka library for python. python3-kafka is used in this tutorial to read information from Kafka. Run the following command to install the library.

$ pip install python3-kafka

Reading simple text data from Kafka

Dissimilar types of information tin be sent from the producer on a particular topic that can be read by the consumer. How a simple text data can exist sent and received from Kafka using producer and consumer is shown in this part of this tutorial.

Create a file named producer1.py with the post-obit python script. KafkaProducer module is imported from the Kafka library. The broker list needs to define at the time of producer object initialization to connect with the Kafka server. The default port of Kafka is '9092'. bootstrap_servers argument is used to define the hostname with the port. 'First_Topic' is set every bit a topic name by which text bulletin will be sent from the producer. Adjacent, a simple text message, 'Hello from Kafka' is sent using send() method of KafkaProducer to the topic, 'First_Topic'.

producer1.py:

# Import KafkaProducer from Kafka library
from kafka import KafkaProducer

# Define server with port
bootstrap_servers = [ 'localhost:9092' ]

# Define topic name where the message will publish
topicName = 'First_Topic'

# Initialize producer variable
producer = KafkaProducer(bootstrap_servers = bootstrap_servers)

# Publish text in defined topic
producer.send (topicName, b'Hello from kafka...' )

# Print message
impress ( "Message Sent" )

Create a file named consumer1.py with the following python script. KafkaConsumer module is imported from the Kafka library to read data from Kafka. sys module is used here to terminate the script. The same hostname and port number of the producer are used in the script of the consumer to read data from Kafka. The topic proper name of the consumer and the producer must be the same that is 'First_topic'.  Adjacent, the consumer object is initialized with the three arguments. Topic name, group id and server information. for loop is used here to read the text send from Kafka producer.

consumer1.py:

# Import KafkaConsumer from Kafka library
from kafka import KafkaConsumer

# Import sys module
import sys

# Define server with port
bootstrap_servers = [ 'localhost:9092' ]

# Define topic proper noun from where the message will recieve
topicName = 'First_Topic'

# Initialize consumer variable
consumer = KafkaConsumer (topicName, group_id = 'group1' ,bootstrap_servers =
bootstrap_servers)

# Read and impress bulletin from consumer
for msg in consumer:
print ( "Topic Name=%southward,Message=%s"%(msg.topic ,msg.value ) )

# Terminate the script
sys.exit ( )

Output:

Run the following command from ane terminal to execute the producer script.

The following output will appear after sending the bulletin.

Run the post-obit command from another final to execute the consumer script.

The output shows the topic proper noun and the text message sent from the producer.

Reading JSON formatted data from Kafka

JSON formatted information can be sent by the Kafka producer and read by Kafka consumer using the json module of python. How JSON data can be serialized and de-serialized before sending and receiving the data using the python-kafka module is shown in this part of this tutorial.

Create a python script named producer2.py with the following script. Some other module named JSON is imported with KafkaProducer module here. value_serializer argument is used with bootstrap_servers argument here to initialize the object of Kafka producer. This argument indicates that JSON data volition exist encoded using 'utf-viii' graphic symbol prepare at the time of sending. Adjacent, JSON formatted information is sent to the topic named JSONtopic.

producer2.py:

# Import KafkaProducer from Kafka library
from kafka import KafkaProducer

# Import JSON module to serialize information
import json

# Initialize producer variable and set parameter for JSON encode
producer = KafkaProducer(bootstrap_servers =
[ 'localhost:9092' ] ,value_serializer= lambda 5: json.dumps (5).encode ( 'utf-8' ) )

# Ship data in JSON format
producer.send ( 'JSONtopic' , { 'name': 'fahmida' , 'email':'[electronic mail protected]' } )

# Print message
print ( "Bulletin Sent to JSONtopic" )

Create a python script named consumer2.py with the post-obit script. KafkaConsumer, sys and JSON modules are imported in this script. KafkaConsumer module is used to read JSON formatted data from the Kafka. JSON module is used to decode the encoded JSON data send from the Kafka producer. Sys module is used to terminate the script. value_deserializer statement is used with bootstrap_servers to define how JSON data will exist decoded. Adjacent, for loop is used to print all the consumer records and JSON data retrieved from Kafka.

consumer2.py:

# Import KafkaConsumer from Kafka library
from kafka import KafkaConsumer

# Import sys module
import sys

# Import json module to serialize data
import json

# Initialize consumer variable and gear up property for JSON decode
consumer = KafkaConsumer ( 'JSONtopic' ,bootstrap_servers = [ 'localhost:9092' ] ,
value_deserializer= lambda m: json.loads (1000.decode ( 'utf-8' ) ) )

# Read data from kafka
for message in consumer:
impress ( "Consumer records:\n" )
print (message)
print ( "\northReading from JSON information\north" )
print ( "Name:" ,message[ 6 ] [ 'name' ] )
print ( "Email:" ,message[ 6 ] [ 'email' ] )
# Terminate the script
sys.exit ( )

Output:

Run the following command from ane concluding to execute the producer script.

The script will print the following message after sending the JSON information.

Run the following control from another terminal to execute the consumer script.

The following output volition announced afterward running the script.

Conclusion:

The information can be sent and received in different formats from Kafka using python. The data can also exist stored into the database and retrieved from the database using Kafka and python. I domicile, this tutorial will aid the python user to start working with Kafka.

Most the author

I am a trainer of web programming courses. I similar to write article or tutorial on various It topics. I have a YouTube channel where many types of tutorials based on Ubuntu, Windows, Give-and-take, Excel, WordPress, Magento, Laravel etc. are published: Tutorials4u Help.

clarktherromes.blogspot.com

Source: https://linuxhint.com/read_data_kafka_python/

Share this post