标签：10 cdh5.7 15 140 Distributed Pseudo hadoop Deployment 2.6
Pre-condition and software
Hadoop version: hadoop-2.6.0-cdh5.7.0
JDK version：jdk-8u45-linux-x64
Linux install ssh service
1.Create user and upload hadoop installation package
wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
rz (upload command)
[root@10-9-15-140 ~]# useradd hadoop
[root@10-9-15-140 ~]# su - hadoop
[hadoop@10-9-15-140 ~]$
[hadoop@10-9-15-140 ~]$ mkdir app
[hadoop@10-9-15-140 ~]$ cd app/
[hadoop@10-9-15-140 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
或
[hadoop@10-9-15-140 app]$ rz
2.jdk deployment
CDH environment:
mkdir /usr/java jdk deployment
mkdir /usr/share/java CDH deployment need mysql jdbc jar package
rz upload jdk-8u45-linux-x64.gz
unzip
[root@10-9-15-140 java]# tar -xzvf jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 uucp 143 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz
Change permission
[root@10-9-15-140 java]# chown -R root:root jdk1.8.0_45
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# vi /etc/profile
#env
export JAVA_HOME=/usr/share/java/jdk1.8.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
[root@10-9-15-140 java]# source /etc/profile
[root@10-9-15-140 java]# which java
/usr/share/java/jdk1.8.0_45/bin/java
3.unzip hadoop package
[hadoop@10-9-15-140 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 app]$ ll
total 304288
drwxr-xr-x 14 hadoop hadoop 4096 Mar 24 2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r-- 1 hadoop hadoop 311585484 Jun 30 23:59 hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ ll
total 76
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 bin *execution script
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 bin-mapreduce1
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 cloudera
drwxr-xr-x 6 hadoop hadoop 4096 Mar 24 2016 etc *configuration menu(conf)
drwxr-xr-x 5 hadoop hadoop 4096 Mar 24 2016 examples
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 examples-mapreduce1
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 include
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 lib *jar package munu
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 libexec
-rw-r--r-- 1 hadoop hadoop 17087 Mar 24 2016 LICENSE.txt
-rw-r--r-- 1 hadoop hadoop 101 Mar 24 2016 NOTICE.txt
-rw-r--r-- 1 hadoop hadoop 1366 Mar 24 2016 README.txt
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 sbin *hadoop component‘s start and stop script
drwxr-xr-x 4 hadoop hadoop 4096 Mar 24 2016 share
drwxr-xr-x 17 hadoop hadoop 4096 Mar 24 2016 src
4.Configuration
Use the following:
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
5.Config ssh localhost non password trusted connection
[hadoop@10-9-15-140 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
53:21:86:1c:af:cd:f3:c4:45:a4:ca:72:ea:83:58:37 hadoop@10-9-15-140
The key's randomart image is:
+--[ RSA 2048]----+
| ..oo ..o |
| oo . + |
| . o . |
| = + . |
| o S o |
| . E+ = |
| o o.. . |
| . ... |
| .. |
+-----------------+
[hadoop@10-9-15-140 ~]$ cd .ssh
[hadoop@10-9-15-140 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Jul 2 01:18 id_rsa
-rw-r--r-- 1 hadoop hadoop 400 Jul 2 01:18 id_rsa.pub
[hadoop@10-9-15-140 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@10-9-15-140 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop 400 Jul 2 01:18 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul 2 01:18 id_rsa 私钥
-rw-r--r-- 1 hadoop hadoop 400 Jul 2 01:18 id_rsa.pub 公钥
ssh localhost date (need to input password, but this user do not need the password.
We need to setup the trusted connetion without password.
Change permission
[hadoop@10-9-15-140 .ssh]$ chmod 600 authorized_keys
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 68:d4:3b:cb:d1:f9:dc:23:65:81:a9:a0:fd:e9:ec:3b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Tue Jul 2 01:19:38 CST 2019
#ssh test successfully
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
Tue Jul 2 01:19:53 CST 2019
6.Format
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format
7.Start
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/07/02 09:48:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-10-9-15-140.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-10-9-15-140.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-10-9-15-140.out
19/07/02 09:48:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
29539 Jps
29222 DataNode
29405 SecondaryNameNode
29134 NameNode
Two issues:
Issue One:
start-dfs.sh启动出现Error:cannot find configuration directory:/etc/hadoop
将hadoop-env.sh中的
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
改为绝对路径
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
Issue Two:
Hadoop安装完后，启动时报Error: JAVA_HOME is not set and could not be found.
解决办法：
修改/etc/hadoop/hadoop-env.sh中设JAVA_HOME。
应当使用绝对路径。
export JAVA_HOME=$JAVA_HOME //错误，原来就这样的 不该就报错
export JAVA_HOME=/usr/share/java/jdk1.8.0_45/bin/java //正确，应该这么改
8.环境变量
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH
9.正式使用
通过浏览器查看NameNode
http://NameNodeIP机器IP地址:50070/
注意：云主机需要在防火墙添加端口
10.停止NameNode和DataNode
$ sbin/stop-dfs.sh
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
19/07/03 00:10:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/07/03 00:10:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
23642 Jps
参考资料：
apache hadoop： hadoop.apache.org
cdh hadoop: http://archive.cloudera.com/cdh5/cdh/5/
标签：10,cdh5.7,15,140,Distributed,Pseudo,hadoop,Deployment,2.6 来源： https://blog.csdn.net/weixin_41158277/article/details/94892674
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。