What I See

We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it. During installation we notice that the distribution is not working. There could be couple of reasons. This was the second time we are having this issue. Check /etc/hosts file if we have all the server names added correctly Second reason would be due to the fact that, one of the installation was terminated midway, leaving a stale config which set the status to ACTIVATING for CDH parcel. So when we try to install, parcel was not distributing. Again there could be similar issue if we do not have enough space on the node for /opt/cloudera . Solution: Deactivating parcel and retry. curl -u username:password -X POST http://adminnode: 7180 /api/v14/clusters/ /parcels/products/CDH/versions/ 5.10 .0 - 1. cdh5 .10 .0 .p0 .41 /commands/deactivate Check for space and increase space for /opt/cloudera/ Most of the time should see th...

We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network. Goal for us is to create a /etc/hosts with all the nodes within the cluster with their private IP addresses. Chef Setup (Assumptions). We have multiple cluster. Each cluster has an environment set. We have a multiple interfaces on each node (But this solutions should work for single interface as well). Steps we do to get information from each node. We take 3 attributes to look for in the cluster. Each server with the cluster have specific fqdn . Private IP (string search). Interface to look for on the node. all_nodes_in_cluster will get all the nodes with that fqdn (This can be change based on requirement to node , tags , roles , hostname ). For every node we look for the specific interface and ge...

This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] , We installed the cloudera management services. Setting Up Zookeeper Services Using Cloudera API [Part 2] Installing Zookeeper service to the cluster. Now we will be installing the HDFS service to our cluster. Create a cluster. Install HDFS service to our cluster. Creating a Cluster and setting up parcel is part of earlier post . Install HDFS Service. HDFS service is installed in stages. Create a HDFS service (if not exist). Update configuration for our newly create HDFS service. Create HDFS roles ( NAMENODE , SECONDARYNAMENODE , DATANODE , GATEWAY ) on the Cluster. Format Namenode. Start HDFS service. Create Temporary /tmp directory in HDFS Create a HDFS service. This is simple create a service if it does not exist. def create_service (cluster) : try : zk_service = cluster.get_service( 'HDFS' ) logg...