Skip to main content

Posts

Creating /etc/hosts file in Chef.

We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network. Goal for us is to create a /etc/hosts with all the nodes within the cluster with their private IP addresses. Chef Setup (Assumptions). We have multiple cluster. Each cluster has an environment set. We have a multiple interfaces on each node (But this solutions should work for single interface as well). Steps we do to get information from each node. We take 3 attributes to look for in the cluster. Each server with the cluster have specific fqdn . Private IP (string search). Interface to look for on the node. all_nodes_in_cluster will get all the nodes with that fqdn (This can be change based on requirement to node , tags , roles , hostname ). For every node we look for the specific interface and ge...

Enable Kerberos Using Cloudera API.

Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster. One of the awesome features of cloudera API is to setup kerberos for the cluster, which otherwise done manually (command line) is very tricky task (loads of this to go wrong). For our kerberos setup we will need to complete below steps. Cloudera Manager Configuration. Stop Cluster. Stop Cloudera Manager service. Service configuration for all the services which needs updates like HDFS , HBASE , KAFKA , ZOOKEEPER , HUE . Creating KT_RENEWER for HUE service. Start Cloudera Manager Services. Start Cluster. Steps To kerberize Cloudera Cluster Step 1: Configure cloudera with LDAP information. # # Deploy LDAP configuration for CM # self.deploy_cm_ldap_configuration() Step 2: Import Admin credentials, using the username / password, which h...