Skip to main content

Enable Kerberos Using Cloudera API.

Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster.
One of the awesome features of cloudera API is to setup kerberos for the cluster, which otherwise done manually (command line) is very tricky task (loads of this to go wrong).
For our kerberos setup we will need to complete below steps.
  1. Cloudera Manager Configuration.
  2. Stop Cluster.
  3. Stop Cloudera Manager service.
  4. Service configuration for all the services which needs updates like HDFS, HBASE, KAFKA, ZOOKEEPER, HUE.
  5. Creating KT_RENEWER for HUE service.
  6. Start Cloudera Manager Services.
  7. Start Cluster.

Steps To kerberize Cloudera Cluster

Step 1: Configure cloudera with LDAP information.
 #
 # Deploy LDAP configuration for CM
 #
 self.deploy_cm_ldap_configuration()
Step 2: Import Admin credentials, using the username / password, which has permission to create/delete AD service accounts.
 #
 # Creating Admin Credentials
 #
 self.cm_import_admin_credentials()
Step 3: Stopping cloudera cluster services.
 #
 # Stopping Cluster
 #
 self.gets_cm_cluster.stop().wait()
 logging.debug("Waiting for CLUSTER to stop completely !!!")
 time.sleep(5)
Step 4: Stopping cloudera management services.
 #
 # Stopping CM services
 #
 self.get_mgmt_services.stop().wait()
 logging.debug("Waiting for CM to stop completely !!!")
 time.sleep(5)
Step 5: Update Zookeeper configuration.
 #
 # Deploy Kerberos Config Zookeeper
 #
 logging.debug("Deploy Service Configuration!!!")
 zk_service = Zookeeper(self.cluster, self.config, self.cloudera_manager)
 zk_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 6: Update HDFS configuration.
 #
 # Deploy Kerberos Config Hdfs
 #
 logging.debug("Deploy Service Configuration!!!")
 hdfs_service = Hdfs(self.cluster, self.config, self.cloudera_manager)
 hdfs_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 6: Update HBASE configuration.
 #
 # Deploy Kerberos Config Hbase
 #
 logging.debug("Deploy Service Configuration!!!")
 hbase_service = Hbase(self.cluster, self.config, self.cloudera_manager)
 hbase_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 7: Update KAFKA configuration.
 #
 # Deploy Kerberos Config Kafka
 #
 logging.debug("Deploy Service Configuration!!!")
 kafka_service = Kafka(self.cluster, self.config, self.cloudera_manager)
 kafka_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 8: Update HUE configuration.
 #
 # Deploy Kerberos Config Hue
 #
 logging.debug("Deploy Service Configuration!!!")
 hue_service = Hue(self.cluster, self.config, self.cloudera_manager)
 hue_service.update_configuration()
 hue_service.add_service_kt_renewer_to_cluster()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 9: Generating Credentials.
 #
 # Generated kerberos credentials in AD.
 #
 self.cm_generate_credentials()
 time.sleep(5)
Step 10: Deploy Client Configuration.
 #
 # Deploy Client Configuration
 #
 logging.info("Deploying Client Config...")
 self.deploy_client_configuration()
 logging.debug("Waiting for CLUSTER to deploy completely !!!")
 time.sleep(5)
Step 11: Starting CM Services.
 #
 # Starting CM services.
 #
 self.get_mgmt_services.start().wait()
 logging.debug("Waiting for CM to start completely !!!")
 time.sleep(5)
Step 12: Starting cluster services.
 #
 # Restart Cluster based on stale configuration and redeploy config if required.
 #
 self.gets_cm_cluster.start().wait()
 logging.info("Cluster Kerberos Deployment Complete.")

Important configuration information.

LDAP setup.
 'cm_ldap': {
     'KDC_HOST': 'adserver1.service.ahmedinc.com',
     'SECURITY_REALM': 'SERVICE.AHMEDINC.COM',
     'KRB_ENC_TYPES': 'rc4-hmac',
     'KDC_ACCOUNT_CREATION_HOST_OVERRIDE': 'adserver2.service.ahmedinc.com',
     'AD_KDC_DOMAIN': 'OU=accounts,OU=test-lab-ou,OU=bigdata,DC=service,DC=ahmedinc,DC=com',
     'AD_DELETE_ON_REGENERATE': True,
     'AD_ACCOUNT_PREFIX': 'Lab1Test'
   }
LDAP credentials for user with create, delete, modify permissions.
 'cm_kdc_import_credentials': {
     'kdc_password': 'Ahm3d@123',
     'kdc_username': 'service-acc-lab@SERVICE.AHMEDINC.COM'
 }
Configuration to connect to Cloudera Manager.
 'cm': {
     'username': 'admin',
     'tls': False,
     'host': 'server-admin-node.ahmedinc.com',
     'api-version': 13,
     'password': 'admin',
     'port': 7180
 }
Cluster configuration.
 'cluster': {
     'version': 'CDH5',
     'hosts': [
       'server-admin-node.ahmedinc.com',
       'server-edge-node.ahmedinc.com',
       'server-worker-node.ahmedinc.com'
     ],
     'name': 'AutomatedHadoopCluster',
     'fullVersion': '5.8.3'
 }
Service Configuration Hdfs.
 'HDFS': {
   'config': {
     'hadoop_security_authentication': 'kerberos',
     'trusted_realms': 'USERACC.AHMEDINC.COM',
     'hadoop_security_authorization': True
   },
   'roles': [
     {
       'config': {
         'dfs_datanode_http_port': 1006,
         'dfs_datanode_data_dir_perm': 700,
         'dfs_datanode_port': 1004
       },
       'hosts': [
         'server-admin-node.ahmedinc.com',
         'server-edge-node.ahmedinc.com',
         'server-worker-node.ahmedinc.com'
       ],
       'group': 'DATANODE'
     }
   ]
 }
Service Configuration Hbase.
 'HBASE': {
   'config': {
     'hbase_thriftserver_security_authentication': 'auth',
     'hbase_security_authorization': True,
     'hbase_security_authentication': 'kerberos'
   }
 }
Service Configuration Zookeeper.
 'ZOOKEEPER': {
   'config': {
     'enableSecurity': True
   }
 }
Service Configuration Kafka.
 'KAFKA': {
   'config': {
     'kerberos.auth.enable': True
   }
 }
Service Configuration Hue.
 'HUE': {
   'roles': [
     {
       'group': 'KT_RENEWER',
       'hosts': [
         'server-admin-node.ahmedinc.com'
       ]
     }
   ]
 }
NOTE : In a kerberos setup, when we do a update_config, generate_missing_credentials command is triggered.
  • we need to wait for this command to complete, before we start the next update_config or else generate_credential command will FAIL.
  • get_commands will return a List of ApiCommand responses. we iterate through all the command to complete. [currently we will receive only one command]
  • get_commands
  • Code snippet.
    current_running_commands = self.cm.get_commands()
logging.debug("Current Running Commands: " + str(current_running_commands))
if not current_running_commands:
  logging.info("Currently no command is running.")
elif current_running_commands:
  for cmd in current_running_commands:
      if cmd.wait().success:
          logging.info("Command Completed. moving on..")
else:
  logging.error("We are in ELSE, something went wrong. :(")
  sys.exit(1)

logging.info("Configuration Updated Successfully..")
time.sleep(5)
    Update Configuration for Cloudera Manager LDAP.
    def deploy_cm_ldap_configuration(self):
  """
      Updating LDAP configuratio no CM server.
      This will require a `sudo servive cloudera-scm-server restart`, we are doing a restart in the end of the script.
  :return:
  """
  cm_api_handle = ApiResource(config['cm']['host'],
                                        config['cm']['port'],
                                        config['cm']['username'],
                                        config['cm']['password'],
                                        config['cm']['tls'],
                                        version=config['cm']['api-version'])
  cluster = cm_api_handle.get_cluster(config['cluster']['name'])
  cloudera_manager = cm_api_handle.get_cloudera_manager()
  cloudera_manager.update_config(self.config['cm_ldap'])
    Example Configuration update for Zookeeper.
    def update_configuration(self):
  """
      Update service configurations
  :return:
  """
  cm_api_handle = ApiResource(config['cm']['host'],
                                        config['cm']['port'],
                                        config['cm']['username'],
                                        config['cm']['password'],
                                        config['cm']['tls'],
                                        version=config['cm']['api-version'])
  cluster = cm_api_handle.get_cluster(config['cluster']['name'])
  get_service = cluster.get_service('ZOOKEEPER')
  get_service.update_config(config['ZOOKEEPER']['config'])
  logging.info("Service Configuration Updated.")

  #   In a kerberos setup, when we do a `update_config`
  #      `generate_missing_credentials` command is triggered.
  #      we need to wait for this command to complete,
  #      before we start the next `update_config`
  #      or else `generate_credential` command will FAIL.

  cm = cm_api_handle.get_cloudera_manager()
  current_running_commands = cm.get_commands()
  logging.debug("Current Running Commands: " + str(current_running_commands))
  if not current_running_commands:
      logging.info("Currently no command is running.")
  elif current_running_commands:
      for cmd in current_running_commands:
          if cmd.wait().success:
              logging.info("Command Completed. moving on..")
  else:
      logging.error("We are in ELSE, something went wrong. :(")
      sys.exit(1)

  logging.info("Configuration Updated Successfully..")
  time.sleep(5)
    Code snippet for kt_renewer
def create_kt_renewer_service(self):


     cm_api_handle = ApiResource(config['cm']['host'],
                                           config['cm']['port'],
                                           config['cm']['username'],
                                           config['cm']['password'],
                                           config['cm']['tls'],
                                           version=config['cm']['api-version'])
     cluster = cm_api_handle.get_cluster(config['cluster']['name'])
     get_service = cluster.get_service('HUE')


     for role in self.config['HUE']['roles']:
         role_group = self.get_service.get_role_config_group("{0}-{1}-BASE".format('HUE', role['group']))
         # Update the group's configuration.
         # [https://cloudera.github.io/cm_api/epydoc/5.10.0/cm_api.endpoints.role_config_groups.ApiRoleConfigGroup-class.html#update_config]
         role_group.update_config(role.get('config', {}))
         self.create_roles(get_service, role, role['group'])
Creating Roles.
 def create_roles(self, service, role, group):
     """
     Create individual roles for all the hosts under a specific role group

     :param role: Role configuration from yaml
     :param group: Role group name
     """
     role_id = 0
     for host in role.get('hosts', []):
         role_id += 1
         role_name = '{0}-{1}-{2}'.format('HUE', group, role_id)
         logging.info("Creating Role name as: " + str(role_name))
         try:
             service.get_role(role_name)
         except ApiException:
             service.create_role(role_name, group, host)
We are done.

Useful Links.

​
Labels:

Comments

Post a Comment

Popular posts from this blog

Cloudera Manager - Duplicate entry 'zookeeper' for key 'NAME'.

We had recently built a cluster using cloudera API’s and had all the services running on it with Kerberos enabled. Next we had a requirement to add another kafka cluster to our already exsisting cluster in cloudera manager. Since it is a quick task to get the zookeeper and kafka up and running. We decided to get this done using the cloudera manager instead of the API’s. But we faced the Duplicate entry 'zookeeper' for key 'NAME' issue as described in the bug below. https://issues.cloudera.org/browse/DISTRO-790 I have set up two clusters that share a Cloudera Manger. The first I set up with the API and created the services with capital letter names, e.g., ZOOKEEPER, HDFS, HIVE. Now, I add the second cluster using the Wizard. Add Cluster->Select Hosts->Distribute Parcels->Select base HDFS Cluster install On the next page i get SQL errros telling that the services i want to add already exist. I suspect that the check for existing service names does n
2 comments
Read more

Zabbix History Table Clean Up

Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps. Stop zabbix server. Take table backup - just in case. Create a temporary table. Update the temporary table with data required, upto a specific date using epoch . Move old table to a different table name. Move updated (new temporary) table to original table which needs to be cleaned-up. Drop the old table. (Optional) Restart Zabbix Since this is not offical procedure, but it has worked for me so use it at your own risk. Here is another post which will help is reducing the size of history tables - http://zabbixzone.com/zabbix/history-and-trends/ Zabbix Version : Zabbix v2.4 Make sure MySql 5.1 is set with InnoDB as innodb_file_per_table=ON Step 1 Stop the Zabbix server sudo service zabbix-server stop Script. echo "------------------------------------------" echo " 1. Stopping Zabbix Server &quo
1 comment
Read more

Access Filter in SSSD `ldap_access_filter` [SSSD Access denied / Permission denied ]

Access Filter Setup with SSSD ldap_access_filter (string) If using access_provider = ldap , this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on this host. If access_provider = ldap and this option is not set, it will result in all users being denied access. Use access_provider = allow to change this default behaviour. Example: access_provider = ldap ldap_access_filter = memberOf=cn=allowed_user_groups,ou=Groups,dc=example,dc=com Prerequisites yum install sssd Single LDAP Group Under domain/default in /etc/sssd/sssd.conf add: access_provider = ldap ldap_access_filter = memberOf=cn=Group Name,ou=Groups,dc=example,dc=com Multiple LDAP Groups Under domain/default in /etc/sssd/sssd.conf add: access_provider = ldap ldap_access_filter = (|(memberOf=cn=System Adminstrators,ou=Groups,dc=example,dc=com)(memberOf=cn=Database Users,ou=Groups,dc=example,dc=com)) ldap_access_filter accepts standa
Post a Comment
Read more