NodeJS Kafka Producer - Using `kafka-node`

Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.

Below is a basic producer code.

below are the Server Details.

1. nodejs is the nodejs server.
2. kafka is the kafka server (single node).

Step 1: Copy the below script in a file called producer_nodejs.js.

 /*
     Basic producer to send data to kafka from nodejs.
     More Information Here : https://www.npmjs.com/package/kafka-node
 */

 //    Using kafka-node - really nice library
 //    create a producer and connect to a Zookeeper to send the payloads.
 var kafka = require('kafka-node'),
     Producer = kafka.Producer,
     client = new kafka.Client('kafka:2181'),
     producer = new Producer(client);

     /*
         Creating a payload, which takes below information
         'topic'     -->    this is the topic we have created in kafka. (test)
         'messages'     -->    data which needs to be sent to kafka. (JSON in our case)
         'partition' -->    which partition should we send the request to. (default)

                         example command to create a topic in kafka: 
                         [kafka@kafka kafka]$ bin/kafka-topics.sh \
                                     --create --zookeeper localhost:2181 \
                                     --replication-factor 1 \
                                     --partitions 1 \
                                     --topic test

                         If there are multiple partition, then we optimize the code here,
                         so that we send request to different partitions. 

     */
     payloads = [
         { topic: 'test', messages: 'This is the First Message I am sending', partition: 0 },
     ];


 //    producer 'on' ready to send payload to kafka.
 producer.on('ready', function(){
     producer.send(payloads, function(err, data){
         console.log(data)
     });
 });

 producer.on('error', function(err){}

Step 2 : Start the kafka cluster as we already did in Installation of Kafka. Assuming topic as test

Step 3 : Start the consumer service as in the below command.

 [kafka-admin@kafka kafka]$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Step 4 : Execute below command. This will send This is the First Message I am sending Message to the Kafka consumer.

 [nodejs-admin@nodejs nodejs]$ node producer_nodejs.js

Step 5 : Check on the consumer you will see the message sent from nodejs.

 [kafka-admin@kafka kafka_2.9.2-0.8.2.0]$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
 This is a message
 This is another message here 
 This is the First Message I am sending

​