Skip to main content

Ansible Playbook - Setup Kafka Cluster.

Ansible Playbook - Setup Kafka Cluster.

This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)
Before we start read more information about Zookeeper/Kafka in the below link.
  1. Setup Zookeeper.
  2. Setup Kafka. Server running on ports 9091/9092 ports on each server.

Before we start.

Download kafka_2.9.2-0.8.2.1.tgz to file_archives directory.
Download zookeeper-3.4.5-cdh5.1.2.tar.gz to file_archives directory.

Get the script from Github.

Below is the command to clone.
ahmed@ahmed-server ~]$ git clone https://github.com/zubayr/ansible_kafka_tarball

Step 1: Update Hosts File.

Update the host file to reflect your server IPs. Currently hosts file looks as below.
[zookeepers]
10.10.18.10 zookeeper_id=1
10.10.18.12 zookeeper_id=2
10.10.18.13 zookeeper_id=3

[kafka-nodes]
10.10.18.10 kafka_broker_id1=11 kafka_port1=9091 kafka_broker_id2=12 kafka_port2=9092
10.10.18.12 kafka_broker_id1=13 kafka_port1=9091 kafka_broker_id2=14 kafka_port2=9092
10.10.18.13 kafka_broker_id1=15 kafka_port1=9091 kafka_broker_id2=16 kafka_port2=9092

Step 2: Update group_vars information as required.

Update users/password and Directory information in group_vars/all file. Currently we have the below information.
# --------------------------------------
# USERs
# --------------------------------------

zookeeper_user: zkadmin
zookeeper_group: zkadmin
zookeeper_password: $6$rounds=40000$1qjG/hovLZOkcerH$CK4Or3w8rR3KabccowciZZUeD.nIwR/VINUa2uPsmGK/2xnmOt80TjDwbof9rNvnYY6icCkdAR2qrFquirBtT1

kafka_user: kafkaadmin
kafka_group: kafkaadmin
kafka_password: $6$rounds=40000$1qjG/hovLZOkcerH$CK4Or3w8rR3KabccowciZZUeD.nIwR/VINUa2uPsmGK/2xnmOt80TjDwbof9rNvnYY6icCkdAR2qrFquirBtT1


# --------------------------------------
# COMMON FOR INSTALL PATH
# --------------------------------------

# Common Location information.
common:
  install_base_path: /usr/local
  soft_link_base_path: /opt

Step 3: Update default information in default/main.yml.

Update the default values if required.

Step 4: Executing.

Below is the command.
ahmed@ahmed-server ansible_kafka_tarball]$ ansible-playbook ansible_kafka.yml -i hosts --ask-pass

Popular posts from this blog

[SOLVED] E: Unable to locate package oracle-java7-installer [Ubuntu 13]

Unable to locate package oracle-java7-installer [Ubuntu 13] Was installing Java today, this is an easy install thanks to “ppa:webupd8team/java“, but when I tried it was not working, but has worked for me all this while. Little goggling around found me a solution. Here is full conversation if you wanted to have a look http://ubuntuforums.org/showthread.php?t=2048793 Installing JAVA on Ubuntu is simple. Do the below to add the repository and run the below command ‘sudo apt-get install oracle-java7-installer’ to install. Error. ahmed@ubuntu:~$ sudo add-apt-repository ppa:webupd8team/java ahmed@ubuntu:~$ sudo apt-get update ahmed@ubuntu:~$ sudo apt-get install oracle-java7-installer ahmed@ubuntu:~$ sudo apt-get install oracle-java7-installer Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package oracle-java7-installer Solution. Elevating to root. ahmed@ubuntu:~$ sudo su Adding to deb. root@ubuntu:~# ech…

Setting up SNMP Trapper for Zabbix.

Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “trapped” by Zabbix. Usually traps are sent upon some condition change and the agent connects to the server on port 162 (as opposed to port 161 on the agent side that is used for queries). Using traps may detect some short problems that occur amidst the query interval and may be missed by the query data. Receiving SNMP traps in Zabbix is designed to work with snmptrapd and one of the built-in mechanisms for passing the traps to Zabbix - either a perl script or SNMPTT. The workflow of receiving a trap: snmptrapd receives a trapsnmptrapd passes the trap to SNMPTT or calls Perl trap receiverSNMPTT or Perl trap receiver parses, formats and writes the trap to a fileZabbix SNMP trapper reads and parses the trap fileFor each trap Zabbix finds all “SNMP trapper” items with host interfaces matching the received trap address. Note that only t…

Zabbix History Table Clean Up

Zabbix history table gets really big, and if you are in a situation where you want to clean it up.
Then we can do so, using the below steps. Stop zabbix server.Take table backup - just in case.Create a temporary table.Update the temporary table with data required, upto a specific date using epoch.Move old table to a different table name.Move updated (new temporary) table to original table which needs to be cleaned-up.Drop the old table. (Optional)Restart Zabbix Since this is not offical procedure, but it has worked for me so use it at your own risk.
Here is another post which will help is reducing the size of history tables - http://zabbixzone.com/zabbix/history-and-trends/ Zabbix Version : Zabbix v2.4
Make sure MySql 5.1 is set with InnoDB as innodb_file_per_table=ON Step 1 Stop the Zabbix serversudo service zabbix-server stop Script. echo"------------------------------------------"echo" 1. Stopping Zabbix Server "echo"----------------------------…