Skip to main content

Cloudera Manager Setup Using Chef [CentOS 6.6]

This cookbook [currently as of now] can be used to setup a Cloudera Manager Server (Management Server) running on MySQL database.
But the intended use for this cookbook [rather a wishlist] is to do more. Simply put Auto Deployment of a Cloudera Hadoop Cluster using Chef, Python and Cloudera API. This will help create cluster for a development/test/preproduction/production environment on a click of a button.
  • Attributes
  • Recipe
  • Usage

Get the Cookbook.

Can be downloaded from the link. Cloudera Manager Setup

How would the Setup Look like.

Nodes and the runlist which will be assigned.
  • Cloudera Manager - Runlist cm_setup default runlist which will include all the installations.
  • All Other nodes - Runlist, will be assigning the Role base_node_install which we will create below.

What does cm_setup default cookbook have.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'
Creating a mysql setup. Setting up cloudera-manager on mysql.
include_recipe 'cm_setup::mysql_setup'
  include_recipe 'cm_setup::mysql_install'
  include_recipe 'cm_setup::mysql_configure'
  include_recipe 'cm_setup::mysql_user_setup
Installing mysql_connector.
include_recipe 'cm_setup::mysql_connector_setup'
Installing cloudera-daemons and agent.
include_recipe 'cm_setup::cloudera_install_setup'
  configuring database using the db script.
  Autostarting `cloudera-scm-server`.

What does the base_node_install Role have.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'

Role.

To setup non-mgmt nodes we can create a role and assign the nodes this role, so that the base setup on that node is completed.
Below is a JSON for the Role for base setup.
{
   "name": "base_node_install",
   "description": "Base Installation for Node other than the Clouder Manager Node.",
   "json_class": "Chef::Role",
   "default_attributes": {

   },
   "override_attributes": {

   },
   "chef_type": "role",
   "run_list": [
     "recipe[cm_setup::commons]",
     "recipe[cm_setup::sysctl_setup]",
     "recipe[cm_setup::ntpd_setup]",
     "recipe[cm_setup::users_setup]",
     "recipe[cm_setup::hostfile_setup]"
   ],
   "env_run_lists": {

   }
}
Creating role on the Chef Server.
┌─[ahmed][zubair-HP-ProBook][±][master U:2 ✗][~/work/chef-repo]
└─▪ knife role create base_node_install
Add the contents above to the role. Once we are done then we can list then using below command.
┌─[ahmed][zubair-HP-ProBook][±][master U:2 ✗][~/work/chef-repo]
└─▪ knife role list
base_node_install
testrole
Now we add the role to each on the nodes which act as a managed node like a namenode, standbynamenode, resourcemanager etc.

Before we assign the Role.

Before we assign the role, we need to bootstrap the node so that it is added to the Chef Server.
Below is the command to add the node to the Chef Server.
knife bootstrap "host_ip" --ssh-port "host_port" --ssh-user "username" --ssh-password "password" --sudo
NOTE : The user we use should have sudo privileges so that chef can install the client on the node.
Here is the output for the vagrant node which was used to test the Cookbook.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ knife bootstrap 127.0.0.1 --ssh-port 2222 --ssh-user vagrant --ssh-password vagrant --sudo
Doing old-style registration with the validation key at /home/ahmed/work/chef-repo/.chef/happy-minds-validator.pem...
Delete your validation key in order to use your user credentials instead

Connecting to 127.0.0.1
127.0.0.1 -----> Installing Chef Omnibus (-v 12)
127.0.0.1 downloading https://omnitruck-direct.chef.io/chef/install.sh
127.0.0.1   to file /tmp/install.sh.3341/install.sh
127.0.0.1 trying wget...
127.0.0.1 el 6 x86_64
127.0.0.1 Getting information for chef stable 12 for el...
127.0.0.1 downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=12&p=el&pv=6&m=x86_64
127.0.0.1   to file /tmp/install.sh.3361/metadata.txt
127.0.0.1 trying wget...
127.0.0.1 sha1    44e71beed0cc0db2481c3e3d2108ad218c32dade
127.0.0.1 sha256    e51559dc7747c03b446f9d1a3cdbb122f274352ba0ed7dd8fdac41e10514b9e2
127.0.0.1 url    https://packages.chef.io/stable/el/6/chef-12.11.18-1.el6.x86_64.rpm
127.0.0.1 version    12.11.18
127.0.0.1 downloaded metadata file looks valid...
127.0.0.1 downloading https://packages.chef.io/stable/el/6/chef-12.11.18-1.el6.x86_64.rpm
127.0.0.1   to file /tmp/install.sh.3361/chef-12.11.18-1.el6.x86_64.rpm
127.0.0.1 trying wget...
127.0.0.1 Comparing checksum with sha256sum...
127.0.0.1 Installing chef 12
127.0.0.1 installing with rpm...
127.0.0.1 warning: /tmp/install.sh.3361/chef-12.11.18-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
127.0.0.1 Preparing...                                                            (1########################################### [100%]
127.0.0.1    1:chef                                                               ( ########################################### [100%]
127.0.0.1 Thank you for installing Chef!
127.0.0.1 Starting the first Chef Client run...
127.0.0.1 Starting Chef Client, version 12.11.18
127.0.0.1 Creating a new client identity for localhost.localdomain using the validator key.
127.0.0.1 resolving cookbooks for run list: []
127.0.0.1 Synchronizing Cookbooks:
127.0.0.1 Installing Cookbook Gems:
127.0.0.1 Compiling Cookbooks...
127.0.0.1 [2016-06-24T15:25:45+02:00] WARN: Node localhost.localdomain has an empty run list.
127.0.0.1 Converging 0 resources
127.0.0.1
127.0.0.1 Running handlers:
127.0.0.1 Running handlers complete
127.0.0.1 Chef Client finished, 0/0 resources updated in 09 seconds

Logon to Chef Server and Edit Run List.

Chef Server and Edit Run List

Select Role to be Assigned.

Select Role to be Assigned

Assigned Role and Save.

Assigned Role and Save

Chef Role is Assigned.

Chef Role is Assigned

Executing sudo chef-client on Node.

Executing sudo chef-client on Node

Attributes.

Below are the set of attributes which can be changed as per requirement.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ tree
.
|____hosts_attr.rb
|____default.rb
|____mysql_attr.rb
|____sudo_attr.rb
|____cdh_attr.rb
|____sysctl_attr.rb
|____security_sssd_attr.rb
|____security_krb5_attr.rb
|____ntp_attr.rb

hosts_attr File

This file has the host information which need to be populated in the /etc/hosts file.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat hosts_attr.rb

#
# Server informatoin for the `/etc/hosts` file changes this as required
#

default['etc_hosts_entries']['9.1.1.1']['hostname'] = 'server9.ahmed.com'
default['etc_hosts_entries']['9.1.1.1']['aliases']  = ['server9']
default['etc_hosts_entries']['9.1.1.1']['comment']  = 'Server9'
default['etc_hosts_entries']['9.1.1.1']['action']   = :create_if_missing

mysql_attr File.

This has parameters related to mysql more attributes can be twicked more information can be found on the base cookbook mysql, mysql_connector, database.
mysql cookbook for creating the mysql instance. mysql_connector cookbook for creating the connector. database cookbook to create database and database users.
File here.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat mysql_attr.rb
#
# MySQL connector attributes
# => https://supermarket.chef.io/cookbooks/mysql_connector
# => https://supermarket.chef.io/cookbooks/mysql_connector/download
#

default['mysql_connector']['j']['install_paths'] = ['/usr/share/java']
default['mysql_connector']['j']['version'] = '5.1.36'

#
# MySQL user, Configuration and services
#
# #
# # Installing `mysqld`
# # => https://github.com/chef-cookbooks/mysql
# # => https://supermarket.chef.io/cookbooks/mysql#knife
# #
#
# #
# # Setting up user for the mysql database.
# # => https://github.com/chef-cookbooks/database
# # => https://supermarket.chef.io/cookbooks/database#knife
# #
#
#

default['mysql']['configuring']['database_service_name'] = 'default'
default['mysql']['configuring']['database_name'] = 'cmdb'
default['mysql']['configuring']['database_root_password'] = 'root@123'

default['mysql']['configuring']['database_user'] = 'cmadmin'
default['mysql']['configuring']['database_password'] = 'cmadmin@123'
default['mysql']['configuring']['database_user_privileges'] = [:all]
default['mysql']['configuring']['database_user_privileges_host'] = '%'

default['mysql']['configuring']['host_ip'] = '127.0.0.1'
default['mysql']['configuring']['port'] = '3306'
default['mysql']['configuring']['version'] = '5.5'

hostsfile File

Creating users and sudo users on the server.
Config File.
┌─[ahmed][zubair-HP-ProBook][±][master U:3 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat sudo_attr.rb
#
# Adding sudo attributes
#

#
# User Setup
#
# #
# # Creating a admin user/group for clouderamanager
# # https://github.com/chef-cookbooks/users
# # https://supermarket.chef.io/cookbooks/users#knife
#

default['users_setup']['groups'] = { 'sysadmin' => 2300, 'cmadmin' => 2301 }

#
# Creating sudo users
# => https://supermarket.chef.io/cookbooks/sudo#knife
# => https://github.com/chef-cookbooks/sudo
#

default['authorization']['sudo']['groups'] = ['cmadmin', 'sysadmin']
default['authorization']['sudo']['users'] = ['cmadmin', 'vagrant', 'sysadminuser']
default['authorization']['sudo']['passwordless'] = true

cdh_attr.rb File

Here we can configure information related to cdh.
  • Creating Repository
  • Installation Packages
  • Services
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat cdh_attr.rb

#
# Cloudera Manager installation and services
#

default['cdh_install']['install_packages'] = [
                                              'oracle-j2sdk1.7',
                                              'cloudera-manager-daemons',
                                              'cloudera-manager-server'
                                            ]
default['cdh_install']['cm_services'] = [
                                          'cloudera-scm-server'
                                        ]

#
# Repository Configuration
#
# # Setting up repos
# # => https://supermarket.chef.io/cookbooks/yum
# # => https://github.com/chef-cookbooks/yum/
# #
#

# description 'Extra Packages for Enterprise Linux'
# mirrorlist 'http://mirrors.fedoraproject.org/mirrorlist?repo=epel-6&arch=$basearch'
# gpgkey 'http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6'

default['yum_repository']['epel']['description'] = 'Extra Packages for Enterprise Linux'
default['yum_repository']['epel']['mirrorlist'] = 'http://mirrors.fedoraproject.org/mirrorlist?repo=epel-6&arch=$basearch'
default['yum_repository']['epel']['gpgkey'] = 'http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6'

# description 'Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 '
# baseurl 'https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/'
# gpgkey 'https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera'

default['yum_repository']['cm']['description'] = 'Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 '
default['yum_repository']['cm']['baseurl'] = 'https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/'
default['yum_repository']['cm']['gpgkey'] = 'https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera'

sysctl_attr File.

This file is to update sysctl.conf. All attributes are from the sysctl cookbook.
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat sysctl_attr.rb

#
# Setting up custom sysctl configuration
# TODO: We need to make the parameter to be added from attributes.
#
# template '/etc/sysctl.conf' do
#   source 'sysctl.conf.erb'
# end
#
# Setting up sysctl.conf
# => https://supermarket.chef.io/cookbooks/sysctl
# => https://github.com/svanzoest-cookbooks/sysctl
#

security_sssd_attr File.

This file is to setup (install and configure) sssd on the node.
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat security_sssd_attr.rb
#
# SSSD installation and configuration
#
#

#
# Installing and configuring SSSD
# => https://supermarket.chef.io/cookbooks/sssd_ldap
# => https://github.com/tas50/chef-sssd_ldap
#

security_krb5_attr File.

This file attributes are to install and configure krb5 for a node.
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat security_krb5_attr.rb
#
# krb5 installation and configuration
#

#
# Installing and configuring krb5
# => https://supermarket.chef.io/cookbooks/krb5
# => https://github.com/atomic-penguin/cookbook-krb5
#

ntp_attr File.

Setting up ntp on a node.
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat ntp_attr.rb
#
# configuring servers
# => https://supermarket.chef.io/cookbooks/ntpd#knife
# => https://github.com/rogerdelph/cookbook-ntpd
#

default['ntp']['mode_servers'] = ['0.pool.ntp.org', '1.pool.ntp.org']

default Recipe Details.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'
Creating a mysql setup. Setting up cloudera-manager on mysql.
include_recipe 'cm_setup::mysql_setup'
  include_recipe 'cm_setup::mysql_install'
  include_recipe 'cm_setup::mysql_configure'
  include_recipe 'cm_setup::mysql_user_setup
Installing mysql_connector.
include_recipe 'cm_setup::mysql_connector_setup'
Installing cloudera-daemons and agent.
include_recipe 'cm_setup::cloudera_install_setup'
  configuring database using the db script.
  Autostarting `cloudera-scm-server`.
Installation and Configuration of sssd. [ Unit Test complete - Need to do TEST on live environment ]
Installation and Configuration of krb5. [ Unit Test complete - Need to do TEST on live environment ]
Configuration of Cloudera Using Cloudera API. [TODO]

Usage.

Below are the steps to setup and environment to execute this cookbook.

Update the .kitchen.yml file with below content [ if required - OPTIONAL ]

File can be found in the ${CHEF_COOKBOOK_HOME}/.kitchen.yml.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ cat .kitchen.yml
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

# Uncomment the following verifier to leverage Inspec instead of Busser (the
# default verifier)
# verifier:
#   name: inspec

platforms:
  - name: grtjn/centos-6.5

suites:
  - name: default
    run_list:
      - recipe[cm_setup::default]
    attributes:

Check the for the vagrant box which will be used.

Command
kitchen list
Output
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/starter]
└─▪ kitchen list
Instance                 Driver   Provisioner  Verifier  Transport  Last Action
default-grtjn-centos-65  Vagrant  ChefSolo     Busser    Ssh

create node.

Command
kitchen create
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:1 ?:3 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen create
-----> Starting Kitchen (v1.8.0)
-----> Creating ...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'grtjn/centos-6.5'...
==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'grtjn/centos-6.5' is up to date...
       ==> default: Setting the name of the VM: kitchen-starter-default-grtjn-centos-65_default_1466270503111_60773
       ==> default: Fixed port collision for 22 => 2222. Now on port 2200.
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 (guest) => 2200 (host) (adapter 1)
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: 127.0.0.1:2200
           default: SSH username: vagrant
           default: SSH auth method: private key
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
           default: The guest additions on this VM do not match the installed version of
           default: VirtualBox! In most cases this is fine, but in rare cases it can
           default: prevent things such as shared folders from working properly. If you see
           default: shared folder errors, please make sure the guest additions within the
           default: virtual machine match the version of VirtualBox you have installed on
           default: your host and reload your VM.
           default:
           default: Guest Additions Version: 4.3.8
           default: VirtualBox Version: 5.0
       ==> default: Setting hostname...
       ==> default: Machine not provisioned because `--no-provision` is specified.
       [SSH] Established
       Vagrant instance  created.
       Finished creating  (0m52.53s).
-----> Kitchen is finished. (0m52.65s)

Login to the node.

Command
kitchen login
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:1 ?:3 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen login
Last login: Sat Jun 18 17:22:13 2016 from 10.0.2.2
[vagrant@default-grtjn-centos-65 ~]$ cat /etc/hosts
127.0.0.1   default-grtjn-centos-65 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[vagrant@default-grtjn-centos-65 ~]$ exit
logout
Connection to 127.0.0.1 closed.

converge - Cookbook with the node.

Command
kitchen converge
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:2 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen converge
-----> Starting Kitchen (v1.8.0)
-----> Converging ...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.3.3...
       Removing non-cookbook files before transfer
       Preparing data_bags
       Preparing validation.pem
       Preparing client.rb
-----> Installing Chef Omnibus (install only if missing)
       Downloading https://www.chef.io/chef/install.sh to file /tmp/install.sh
       Trying wget...
       Trying curl...
       Download complete.
       el 6 x86_64
       Getting information for chef stable  for el...
       downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=&p=el&pv=6&m=x86_64
         to file /tmp/install.sh.1983/metadata.txt
       trying wget...
       sha1 44e71beed0cc0db2481c3e3d2108ad218c32dade
       sha256 e51559dc7747c03b446f9d1a3cdbb122f274352ba0ed7dd8fdac41e10514b9e2
       url  https://packages.chef.io/stable/el/6/chef-12.11.18-1.el6.x86_64.rpm
       version  12.11.18
       downloaded metadata file looks valid...
       downloading https://packages.chef.io/stable/el/6/chef-12.11.18-1.el6.x86_64.rpm
         to file /tmp/install.sh.1983/chef-12.11.18-1.el6.x86_64.rpm
       trying wget...
       trying curl...
       Comparing checksum with sha256sum...

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       You are installing an omnibus package without a version pin.  If you are installing
       on production servers via an automated process this is DANGEROUS and you will
       be upgraded without warning on new releases, even to new major releases.
       Letting the version float is only appropriate in desktop, test, development or
       CI/CD environments.

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       Installing chef
       installing with rpm...
       warning: /tmp/install.sh.1983/chef-12.11.18-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
       Preparing...                                                            (100%########################################### [100%]
          1:chef                                                               (  1%########################################### [100%]
       Thank you for installing Chef!
       Transferring files to 
       Starting Chef Client, version 12.11.18
       Creating a new client identity for default-grtjn-centos-65 using the validator key.
       resolving cookbooks for run list: ["cm_setup::default"]
       Synchronizing Cookbooks:
         - hostsfile (2.4.5)
         - sudo (2.9.0)
         - users (2.0.3)
         - cm_setup (0.1.0)
         - sysctl (0.7.5)
         - mysql (7.1.1)
         - yum (3.11.0)
         - smf (2.2.8)
         - ohai (3.0.1)
         - database (5.1.2)
         - build-essential (6.0.0)
         - rbac (1.0.3)
         - openssl (4.4.0)
         - yum-mysql-community (0.2.0)
         - apt (4.0.0)
         - chef-sugar (3.3.0)
         - mingw (1.2.0)
         - seven_zip (2.0.1)
         - postgresql (4.0.6)
         - compat_resource (12.10.6)
         - windows (1.43.0)
         - chef_handler (1.4.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       [2016-06-18T17:40:28+00:00] WARN: Chef::Provider::AptRepository already exists!  Cannot create deprecation class for LWRP provider apt_repository from cookbook apt
       [2016-06-18T17:40:28+00:00] WARN: AptRepository already exists!  Deprecation class overwrites Custom resource apt_repository from cookbook apt
       [2016-06-18T17:40:28+00:00] WARN: Cloning resource attributes for hostsfile_entry[3.3.3.3] from prior resource (CHEF-3694)
       [2016-06-18T17:40:28+00:00] WARN: Previous hostsfile_entry[3.3.3.3]: /tmp/kitchen/cache/cookbooks/cm_setup/recipes/hostfile_setup.rb:27:in `from_file'
       [2016-06-18T17:40:28+00:00] WARN: Current  hostsfile_entry[3.3.3.3]: /tmp/kitchen/cache/cookbooks/cm_setup/recipes/hostfile_setup.rb:34:in `from_file'
       Converging 37 resources

       ##
       ###################### VERBOSE ########################
       ##

       Recipe: cm_setup::cloudera_install_setup
         * yum_package[oracle-j2sdk1.7] action install (up to date)
         * yum_package[cloudera-manager-daemons] action install (up to date)
         * yum_package[cloudera-manager-server] action install (up to date)
       Recipe: cm_setup::mysql_setup
         * mysql_service[default] action restart
           * service[default :restart mysql-default] action restart
             - restart service service[default :restart mysql-default]

       Running handlers:
       Running handlers complete
       Chef Client finished, 20/106 resources updated in 07 minutes 05 seconds
       Finished converging  (7m25.80s).
Above we have no issues and the cookbook converged successfully.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen list
Instance                 Driver   Provisioner  Verifier  Transport  Last Action
default-grtjn-centos-65  Vagrant  ChefZero     Busser    Ssh        Converged

Server Spec Verification using Kitchen.

Command
kitchen verify
Output.
┌─[ahmed][zubair-HP-ProBook][±][master U:1 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen verify
-----> Starting Kitchen (v1.8.0)
-----> Verifying ...
       Preparing files for transfer
-----> Busser installation detected (busser)
       Installing Busser plugins: busser-serverspec
       Plugin serverspec already installed
       Removing /tmp/verifier/suites/serverspec
       Transferring files to 
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/tmp/verifier/gems/gems/rspec-core-3.4.4/lib /opt/chef/embedded/bin/rspec --pattern    /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec

       cm_setup::default
         File "/etc/yum.repos.d/cloudera-manager.repo"
           should exist
         File "/etc/mysql-default/conf.d/default.cnf"
           should exist
           should be file
           should contain "max_connections = 550"
         File "/etc/sysctl.d/99-chef-attributes.conf"
           should exist
           should be file
           should contain "vm.dirty_ratio"
           should contain "vm.swappiness"
           should contain "vm.nr_hugepages"
         Package "ntp"
           should be installed
         Package "oracle-j2sdk1.7"
           should be installed
         Package "cloudera-manager-daemons"
           should be installed
         Package "cloudera-manager-server"
           should be installed
         File "/etc/hosts"
           should exist
           should be file
           should contain "namenode.ahmed.com"
           should contain "standbynamenode.ahmed.com"
           should contain "resourcemanager.ahmed.com"
         File "/etc/cloudera-scm-server/db.properties"
           should exist
           should be file
           should contain "user=cmadmin"
           should contain "name=cmdb"
         User "cmadmin"
           should exist
           should have home directory "/home/cmadmin"
         User "sysadminuser"
           should exist
           should have home directory "/home/sysadminuser"
         MySQL config parameters
           Mysql config "innodb_flush_log_at_trx_commit"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:55
           Mysql config "socket"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:59
           Mysql config "innodb_flush_method"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:66
           Mysql config "innodb_log_file_size"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:70
         Yumrepo "epel"
           should exist
           should be enabled
         Yumrepo "cloudera-manager"
           should exist
           should be enabled

       Finished in 1.32 seconds (files took 0.52476 seconds to load)
       34 examples, 0 failures

       Finished verifying  (0m6.22s).
-----> Kitchen is finished. (0m6.66s)

Cloudera Manager UI.

Logon to the node and open up a browser and hit.
http://127.0.0.1:7180/
You will see the cloudera manager UI. NOTE: This will take a while for the first time, as clouder will initialize the database for first time use.
​
Labels:

Comments

  1. PandaGeneralMarch 21, 2022 at 3:15 PM

    Congenital Diaphragmatic Hernia
    The diaphragm typically forms during the first eight weeks of pregnancy. In CDH patients, the size of the hole in the diaphragm will determine how much a baby’s lungs, heart, and other internal organs will be affected.

    ReplyDelete

Post a Comment

Popular posts from this blog

Zabbix History Table Clean Up

Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps. Stop zabbix server. Take table backup - just in case. Create a temporary table. Update the temporary table with data required, upto a specific date using epoch . Move old table to a different table name. Move updated (new temporary) table to original table which needs to be cleaned-up. Drop the old table. (Optional) Restart Zabbix Since this is not offical procedure, but it has worked for me so use it at your own risk. Here is another post which will help is reducing the size of history tables - http://zabbixzone.com/zabbix/history-and-trends/ Zabbix Version : Zabbix v2.4 Make sure MySql 5.1 is set with InnoDB as innodb_file_per_table=ON Step 1 Stop the Zabbix server sudo service zabbix-server stop Script. echo "------------------------------------------" echo " 1. Stopping Zabbix Server ...
1 comment
Read more

Installing Zabbix Version 2.4 Offline (Zabbix Server without Internet).

There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a custom repo on the remote/zabbix server so that we can install zabbix using rpms Here is how we are planning to do this. Download all the dependency rpms on a machine which has internet connection, using yum-downloadonly or repotrack . Transfer all the rpms to the remote server. Create a repo on the remote server. Update yum configuration. Install. NOTE: This method can be used to install any application, but here we have used zabbix as we had this requirement for a zabbix server. Download dependent rpms . On a machine which has internet connection install the package below. And download all the rpms . Make sure the system are similar (not required to be identical - At-least the OS should be of same version) mkdir /zabbix_rpms yum install yum-downloadonly Downloading all the rpms to location /zabbix_rpms/ ,...
1 comment
Read more

Access Filter in SSSD `ldap_access_filter` [SSSD Access denied / Permission denied ]

Access Filter Setup with SSSD ldap_access_filter (string) If using access_provider = ldap , this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on this host. If access_provider = ldap and this option is not set, it will result in all users being denied access. Use access_provider = allow to change this default behaviour. Example: access_provider = ldap ldap_access_filter = memberOf=cn=allowed_user_groups,ou=Groups,dc=example,dc=com Prerequisites yum install sssd Single LDAP Group Under domain/default in /etc/sssd/sssd.conf add: access_provider = ldap ldap_access_filter = memberOf=cn=Group Name,ou=Groups,dc=example,dc=com Multiple LDAP Groups Under domain/default in /etc/sssd/sssd.conf add: access_provider = ldap ldap_access_filter = (|(memberOf=cn=System Adminstrators,ou=Groups,dc=example,dc=com)(memberOf=cn=Database Users,ou=Groups,dc=example,dc=com)) ldap_access_filter accepts standa...
1 comment
Read more