标签：cloudera CDH6.3 cmf 离线 server 部署 64 node0 ntpd
Cloudera（CDH）简介
官方文档https://www.cloudera.com/documentation/enterprise/latest.html CDH(Clouder's Distribution Including Apache hadoop)，基于web的用户界面，支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、pig、Hbase、Zookeeper、Sqoop,简化了大数据平台的安装和使用难度。一、准备环境
1、4台虚拟机，8核32G，1T硬盘
node1的IP地址为：192.168.2.180 node2的IP地址为：192.168.2.181 node3的IP地址为：192.168.2.182 node4的IP地址为：192.168.2.1832、关闭所有节点的selinux
vi /etc/selinux/config 将SELINUX=enforcing改为SELINUX=disabled 设置后需要重启才能生效，现在先不要重启机器
3、设置所有节点的hosts文件
[root@localhost ~]# echo "192.168.2.180 node0">> /etc/hosts [root@localhost ~]# echo "192.168.2.181 node1">> /etc/hosts [root@localhost ~]# echo "192.168.2.182 node2">> /etc/hosts [root@localhost ~]# echo "192.168.2.183 node3">> /etc/hosts [root@node0 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.180 node0 192.168.2.181 node1 192.168.2.182 node2 192.168.2.183 node3
4、关闭所有节点的防火墙和清空防火墙规则链
# systemctl stop firewalld
# systemctl disable firewalld
# iptables -F
5、关闭所有节点的selinux
[root@localhost ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
重启服务器
# reboot
6、设置所有节点的时区一致及时钟同步
所有节点安装ntp # yum install -y ntp 选取node0为ntp的主节点 # vi /etc/ntp.conf
添加内容如下：
#添加一个网段 restrict 192.168.2.254 mask 255.255.255.0 nomodify notrap #time 设置时间同步服务器 server 0.asia.pool.ntp.org server 1.asia.pool.ntp.org server 2.asia.pool.ntp.org server 3.asia.pool.ntp.org #当外部时间不可用时，可使用本地硬件时间 server 127.127.1.0 iburst #local clock
开启ntpd及查看状态
[root@node0 ~]# systemctl start ntpd
[root@node0 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2020-07-21 23:08:36 CST; 4s ago
Process: 3209 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 3210 (ntpd)
CGroup: /system.slice/ntpd.service
└─3210 /usr/sbin/ntpd -u ntp:ntp -g
Jul 21 23:08:36 node0 ntpd[3210]: proto: precision = 0.105 usec
Jul 21 23:08:36 node0 ntpd[3210]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Jul 21 23:08:36 node0 ntpd[3210]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Jul 21 23:08:36 node0 ntpd[3210]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listen and drop on 1 v6wildcard :: UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listen normally on 3 ens192 192.168.2.180 UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listen normally on 4 lo ::1 UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listen normally on 5 ens192 fe80::664e:b20e:244f:99bd UDP 123
Jul 21 23:08:36 node0 ntpd[3210]: Listening on routing socket on fd #22 for interface updates
[root@node0 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
y.ns.gin.ntt.ne .INIT. 16 u - 64 0 0.000 0.000 0.000
ntp.hkg10.hk.le .INIT. 16 u - 64 0 0.000 0.000 0.000
111.230.189.174 .INIT. 16 u - 64 0 0.000 0.000 0.000
LOCAL(0) .LOCL. 5 l - 64 0 0.000 0.000 0.000
设置开机自启动
# systemctl enable ntpd
其他从节点停止禁用ntpd服务
由于是新安装的服务，默认是不启动的。
# systemctl stop ntpd # systemctl disable ntpd
node1、node2、node3分别同步node0的时间
# ntpdate node0
设置node1、node2、node3每天凌晨同步node0节点时间的计划任务
# crontab -e
添加如下内容
00 00 * * * /usr/sbin/ntpdate node0 >/dev/null &
7、每个节点部署JDK【Java的路径必须放在/usr/java下】
# mkdir /usr/java
# tar -xzvf jdk-8u181-linux-x64.tar.gz -C /usr/java/
# chown -R root:root /usr/java/jdk1.8.0_181
# echo "export JAVA_HOME=/usr/java/jdk1.8.0_181" >> /etc/profile
# echo "export PATH=/usr/java/jdk1.8.0_181/bin:${PATH}" >> /etc/profile
# source /etc/profile
# which java
8、MySQL数据库相关操作
创建CDH的元数据库和用户、amon服务的数据库及用户
[root@mysql5 ~]# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1800190 Server version: 5.7.27 MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create database cmf DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> create database amon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> create user 'cmf' identified by 'Cnbisoft@123'; Query OK, 0 rows affected (0.00 sec) mysql> create user 'amon' identified by 'Cnbisoft@123'; Query OK, 0 rows affected (0.00 sec) mysql> grant all on cmf.* to 'cmf'; Query OK, 0 rows affected (0.00 sec) mysql> grant all on amon.* to 'amon'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)
9、node0节点部署MySQL jdbc jar
# mkdir -p /usr/share/java/
# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
二、CDH部署
1、离线部署cm server 及agent
所有节点创建目录及解压
# mkdir /opt/cloudera-manager
# tar -xzvf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager/
2、选择node0为cm server，不下载依赖包直接部署
# cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64 # rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force # rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force[root@node1 x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force warning: cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY Preparing... ################################# [100%] Updating / installing... 1:cloudera-manager-daemons-6.3.1-14################################# [100%] ( 10%) [root@node1 x86_64]# [root@node1 x86_64]# [root@node1 x86_64]# [root@node1 x86_64]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force warning: cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY Preparing... ################################# [100%] Updating / installing... 1:cloudera-manager-server-6.3.1-146################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.
3、所有节点（包含node0）为cm agent，不下载依赖包直接部署
# cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64 # rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force # rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force4、所有节点修改agent的配置，指向server的节点node0
# sed -i "s/server_host=localhost/server_host=node0/g" /etc/cloudera-scm-agent/config.ini5、主节点修改server的配置
# vi /etc/cloudera-scm-server/db.properties
# Copyright (c) 2012 Cloudera, Inc. All rights reserved. # # This file describes the database connection. # # The database type # Currently 'mysql', 'postgresql' and 'oracle' are valid databases. com.cloudera.cmf.db.type=mysql # The database host # If a non standard port is needed, use 'hostname:port' #com.cloudera.cmf.db.host=localhost com.cloudera.cmf.db.host=192.168.2.240 # The database name #com.cloudera.cmf.db.name=cmf com.cloudera.cmf.db.name=cmf # The database user #com.cloudera.cmf.db.user=cmf com.cloudera.cmf.db.user=cmf # The database user's password #com.cloudera.cmf.db.password= com.cloudera.cmf.db.password=Cnbisoft@123 # The db setup type # After fresh install it is set to INIT # and will be changed post config. # If scm-server uses Embedded DB then it is set to EMBEDDED # If scm-server uses External DB then it is set to EXTERNAL #com.cloudera.cmf.db.setupType=INIT com.cloudera.cmf.db.setupType=EXTERNAL
6、node0节点部署离线parcel源
安装httpd服务
# yum install -y httpd
部署离线parcel源
# mkdir -p /var/www/html/cdh6_parcel # mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel/ # mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha # mv manifest.json /var/www/html/cdh6_parcel/ 启动httpd，window查看 # systemctl start httpd # systemctl enable httpd http://192.168.2.180/cdh6_parcel/
7、node0节点启动Server
# systemctl start cloudera-scm-server
# cd /var/log/cloudera-scm-server/
8、所有节点启动Agent
# systemctl start cloudera-scm-agent
# systemctl status cloudera-scm-agent
三、接下来，全部Web界面操作
访问地址：http://192.168.2.180:7180 账号密码：admin/admin
欢迎使用Cloudera Manager
最终用户许可条款与条件。勾选
欢迎使用Cloudera Manager--您想要部署哪个版本？选择Cloudera Express免费版本
继续
BigData
管理当前的主机，继续
选择更多选项
添加本地URL库
自动刷新页面，显示CDH
正在安装
end
标签：cloudera,CDH6.3,cmf,离线,server,部署,64,node0,ntpd 来源： https://www.cnblogs.com/djlsunshine/p/13356168.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。