What I See

We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it. During installation we notice that the distribution is not working. There could be couple of reasons. This was the second time we are having this issue. Check /etc/hosts file if we have all the server names added correctly Second reason would be due to the fact that, one of the installation was terminated midway, leaving a stale config which set the status to ACTIVATING for CDH parcel. So when we try to install, parcel was not distributing. Again there could be similar issue if we do not have enough space on the node for /opt/cloudera . Solution: Deactivating parcel and retry. curl -u username:password -X POST http://adminnode: 7180 /api/v14/clusters/ /parcels/products/CDH/versions/ 5.10 .0 - 1. cdh5 .10 .0 .p0 .41 /commands/deactivate Check for space and increase space for /opt/cloudera/ Most of the time should see th...

We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network. Goal for us is to create a /etc/hosts with all the nodes within the cluster with their private IP addresses. Chef Setup (Assumptions). We have multiple cluster. Each cluster has an environment set. We have a multiple interfaces on each node (But this solutions should work for single interface as well). Steps we do to get information from each node. We take 3 attributes to look for in the cluster. Each server with the cluster have specific fqdn . Private IP (string search). Interface to look for on the node. all_nodes_in_cluster will get all the nodes with that fqdn (This can be change based on requirement to node , tags , roles , hostname ). For every node we look for the specific interface and ge...

Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster. One of the awesome features of cloudera API is to setup kerberos for the cluster, which otherwise done manually (command line) is very tricky task (loads of this to go wrong). For our kerberos setup we will need to complete below steps. Cloudera Manager Configuration. Stop Cluster. Stop Cloudera Manager service. Service configuration for all the services which needs updates like HDFS , HBASE , KAFKA , ZOOKEEPER , HUE . Creating KT_RENEWER for HUE service. Start Cloudera Manager Services. Start Cluster. Steps To kerberize Cloudera Cluster Step 1: Configure cloudera with LDAP information. # # Deploy LDAP configuration for CM # self.deploy_cm_ldap_configuration() Step 2: Import Admin credentials, using the username / password, which h...