标签：xml hadoop DFS site CentOs7 Master yarn 版本 Hadoop2.7
Hadoop-2.7.7集群安装
下载地址，可选择版本：https://archive.apache.org/dist/hadoop/common/
只在Master节点上安装
安装步骤
- 解压安装
- 用户授权
$ cd ~/Downloads
$ rz
# 上传下载到windows内的hadoop文件
$ sudo tar -zxf ./hadoop-2.7.7.tar.gz -C /usr/local
$ cd /usr/local
# 更改文件夹名称便于后面的配置
$ sudo mv ./hadoop-2.2.7/ ./hadoop
$ sudo chown -R qinphy ./hadoop
环境配置
$ vi ~/.bashrc
配置内容
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin
$ source ~/.bashrc
# 检验安装及环境配置
$ cd ~
$ hadoop version
hadoop-2.2.7
...
集群配置
$ vi slaves
workers文件是指定从节点
每行一个
Slave1 Slave2
$ vi core-site.xml
core-site.xml是全局参数配置文件
主要是用于配置NameNode与DataNode通信地址
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://Master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/tmp</value> </property> </configuration>
$ vi hdfs-site.xml
hdfs-site.xml配置内容：声明
dfs.replication是从节点个数
<configuration> <property> <name>dfs.namenode.http-address</name> <value>Master:50090</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>Slave1:50070</value> </property> <property> <name>dfs.replication</name> <value>4</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoop/tmp/dfs/data</value> </property> </configuration>
$ cp ./mapred-site.xml.template ./mapred-site.xml
$ rm -f ./mapred-site.xml.template
$ vi mapred-site.xml
mapred-site.xml配置
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>Master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>Master:19888</value> </property> </configuration>
$ vi yarn-site.xml
yarn-site配置
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>Master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>Master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>Master:8031</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>Master:8030</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>Master:8100</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> </configuration>
配置好从节点
# 转发到每个结点
$ cd /usr/local
$ sudo tar -zcf ~/hadoop.master.tar.gz ./hadoop
$ scp ~/hadoop.master.tar.gz qinphy@Slave1:/home/qinphy
$ scp ~/hadoop.master.tar.gz qinphy@Slave2:/home/qinphy
# 每个子节点上解压
$ sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local
# 授予权限
$ sudo chown -R qinphy /usr/local/hadoop
初始化
$ hdfs namenode -format
启动与关闭
$ start-all.sh
# 等同于start-dfs.sh+start-yarn.sh
查看效果：
# 在每个节点上
$ jps
# 在Master结点上
$ hdfs dfsadmin -report
# 停止Hadoop
$ stop-all.sh
jps结果：
# Master $ jps 10433 Jps 10317 ResourceManager 9967 NameNode # Slave1 $ jps 9796 NodeManager 9881 Jps 9722 SecondaryNameNode 9661 DataNod # 其余Slave结点 $ jps 9634 DataNode 9778 Jps 9715 NodeManagerreport结果：能够看到从节点信息
$ hdfs dfsadmin -report Configured Capacity: 36477861888 (33.97 GB) Present Capacity: 30168911872 (28.10 GB) DFS Remaining: 30168903680 (28.10 GB) DFS Used: 8192 (8 KB) DFS Used%: 0.00% Replicated Blocks: Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 Erasure Coded Block Groups: Low redundancy block groups: 0 Block groups with corrupt internal blocks: 0 Missing block groups: 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 ------------------------------------------------- Live datanodes (2): Name: 192.168.1.121:9866 (Slave1) Hostname: Slave1 Decommission Status : Normal Configured Capacity: 18238930944 (16.99 GB) DFS Used: 4096 (4 KB) Non DFS Used: 3154427904 (2.94 GB) DFS Remaining: 15084498944 (14.05 GB) DFS Used%: 0.00% DFS Remaining%: 82.70% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri May 01 08:42:26 CST 2020 Last Block Report: Fri May 01 08:41:45 CST 2020 Num of Blocks: 0 Name: 192.168.1.122:9866 (Slave2) Hostname: Slave2 Decommission Status : Normal Configured Capacity: 18238930944 (16.99 GB) DFS Used: 4096 (4 KB) Non DFS Used: 3154522112 (2.94 GB) DFS Remaining: 15084404736 (14.05 GB) DFS Used%: 0.00% DFS Remaining%: 82.70% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri May 01 08:42:26 CST 2020 Last Block Report: Fri May 01 08:41:37 CST 2020 Num of Blocks: 0查看web：192.168.1.120:50090
标签：xml,hadoop,DFS,site,CentOs7,Master,yarn,版本,Hadoop2.7 来源： https://blog.csdn.net/weixin_42631782/article/details/115438008
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。