三少个人自留地

hadoop安装完成后，需要对hadoop进行一些安装后的测试，证明hadoop确实安装成功。以下内容记录一些hadoop的安装后的基本测试：

1：format namenode

安装完成后，用如下命令进行namenode的format

[root@master dfs]# hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master.localdomain/172.16.3.217
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.21.0
STARTUP_MSG: classpath = /root/jz/hadoop-0.21.0/bin/../conf:/usr/java/jdk1.6.0_27/lib/tools.jar:/root/jz/hadoop-0.21.0/bin/..:/root/jz/hadoop-0.21.0/bin/../hadoop-common-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-common-test-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0-sources.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-hdfs-ant-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-hdfs-test-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-hdfs-test-0.21.0-sources.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-mapred-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-mapred-0.21.0-sources.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-mapred-examples-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-mapred-test-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../hadoop-mapred-tools-0.21.0.jar:/root/jz/hadoop-0.21.0/bin/../lib/ant-1.6.5.jar:/root/jz/hadoop-0.21.0/bin/../lib/asm-3.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/aspectjrt-1.6.5.jar:/root/jz/hadoop-0.21.0/bin/../lib/aspectjtools-1.6.5.jar:/root/jz/hadoop-0.21.0/bin/../lib/avro-1.3.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-cli-1.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-codec-1.4.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-el-1.0.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-httpclient-3.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-lang-2.5.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-logging-1.1.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-logging-api-1.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/commons-net-1.4.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/core-3.1.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/ftplet-api-1.0.0.jar:/root/jz/hadoop-0.21.0/bin/../lib/ftpserver-core-1.0.0.jar:/root/jz/hadoop-0.21.0/bin/../lib/ftpserver-deprecated-1.0.0-M2.jar:/root/jz/hadoop-0.21.0/bin/../lib/hsqldb-1.8.0.10.jar:/root/jz/hadoop-0.21.0/bin/../lib/jackson-core-asl-1.4.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/jackson-mapper-asl-1.4.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/jasper-compiler-5.5.12.jar:/root/jz/hadoop-0.21.0/bin/../lib/jasper-runtime-5.5.12.jar:/root/jz/hadoop-0.21.0/bin/../lib/jdiff-1.0.9.jar:/root/jz/hadoop-0.21.0/bin/../lib/jets3t-0.7.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/jetty-6.1.14.jar:/root/jz/hadoop-0.21.0/bin/../lib/jetty-util-6.1.14.jar:/root/jz/hadoop-0.21.0/bin/../lib/jsp-2.1-6.1.14.jar:/root/jz/hadoop-0.21.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/root/jz/hadoop-0.21.0/bin/../lib/junit-4.8.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/kfs-0.3.jar:/root/jz/hadoop-0.21.0/bin/../lib/log4j-1.2.15.jar:/root/jz/hadoop-0.21.0/bin/../lib/mina-core-2.0.0-M5.jar:/root/jz/hadoop-0.21.0/bin/../lib/mockito-all-1.8.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/oro-2.0.8.jar:/root/jz/hadoop-0.21.0/bin/../lib/paranamer-2.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/paranamer-ant-2.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/paranamer-generator-2.2.jar:/root/jz/hadoop-0.21.0/bin/../lib/qdox-1.10.1.jar:/root/jz/hadoop-0.21.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/root/jz/hadoop-0.21.0/bin/../lib/slf4j-api-1.5.11.jar:/root/jz/hadoop-0.21.0/bin/../lib/slf4j-log4j12-1.5.11.jar:/root/jz/hadoop-0.21.0/bin/../lib/xmlenc-0.52.jar:/root/jz/hadoop-0.21.0/bin/../lib/jsp-2.1/*.jar:/root/jz/hadoop-0.21.0/hdfs/bin/../conf:/root/jz/hadoop-0.21.0/hdfs/bin/../hadoop-hdfs-*.jar:/root/jz/hadoop-0.21.0/hdfs/bin/../lib/*.jar:/root/jz/hadoop-0.21.0/bin/../mapred/conf:/root/jz/hadoop-0.21.0/bin/../mapred/hadoop-mapred-*.jar:/root/jz/hadoop-0.21.0/bin/../mapred/lib/*.jar:/root/jz/hadoop-0.21.0/hdfs/bin/../hadoop-hdfs-*.jar:/root/jz/hadoop-0.21.0/hdfs/bin/../lib/*.jar
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by ‘tomwhite’ on Tue Aug 17 01:02:28 EDT 2010
************************************************************/
11/09/26 14:12:29 INFO namenode.FSNamesystem: defaultReplication = 3
11/09/26 14:12:29 INFO namenode.FSNamesystem: maxReplication = 512
11/09/26 14:12:29 INFO namenode.FSNamesystem: minReplication = 1
11/09/26 14:12:29 INFO namenode.FSNamesystem: maxReplicationStreams = 2
11/09/26 14:12:29 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks = false
11/09/26 14:12:29 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/09/26 14:12:29 INFO namenode.FSNamesystem: fsOwner=root
11/09/26 14:12:29 INFO namenode.FSNamesystem: supergroup=supergroup
11/09/26 14:12:29 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/09/26 14:12:29 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/09/26 14:12:30 INFO common.Storage: Image file of size 110 saved in 0 seconds.
11/09/26 14:12:30 INFO common.Storage: Storage directory /hadoopdata/dfs/name has been successfully formatted.
11/09/26 14:12:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.localdomain/172.16.3.217
************************************************************/
2:检查格式化好后的文件系统的状态：
继续阅读 →

最近有时间看看hadoop的一些东西，而且在测试的环境上做了一些搭建的工作。仅记录之；
首先，安装前需要做一些准备工作。
1：准备三台虚拟机。使用一台pcserver作为测试服务器，同时使用Oracle VM VirtualBox来作为虚拟机的服务器。
新建了三个虚拟机以后，安装linux，我安装的linux的版本是redhat linux 5.4 x64版本。安装完成以后，
设置其中的ip地址，master服务器为：172.16.3.217；slave01为172.16.3.218；slave02为172.16.3.219；
2：给虚拟机安装java环境，由于hadoop推荐使用java的1.6版本。而实际上redhat 5.4自带的为1.4.2，
因此要先删除，然后再安装心的版本；java 1.6的版本到java的官方网站下载即可。下载到java的安装包以后。
首先删除系统自带的java 1.4.2；
在root用户下执行：
rpm -qa|grep gcj
能看到回显的结果中有类似：
java-1.4.2-gcj-compat-1.4.2.0-40jpp.115这样的内容
然后
rpm -e java-1.4.2-gcj-compat-1.4.2.0-40jpp.115
这个有可能无法执行成功，提示有依赖关系，这个时候，加上–nodeps再执行，就能成功了；
然后安装jdk 1.6，方法如下：
执行下载下来的bin文件：
./jdk-6u27-linux-x64-rpm.bin
然后安装生成的rpm文件：
rpm -ivh jdk-6u27-linux-amd64.rpm
等到安装完成以后，检查java安装的情况
java -version
一般会得到类似如下的信息：
java version “1.6.0_27″
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
安装完成以后，需要修改系统的文件让java的路径输出给所有的用户，一般修改的是/etc/profile,在这个文件中增加如下的内容：
export JAVA_HOME=/usr/java/jdk1.6.0_27
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC
export HADOOP_HOME=/root/jz/hadoop-0.21.0
export PATH=$HADOOP_HOME/bin:$PATH
在三个机器上都安装好java以后，可以继续下面的操作了；
3：在hadoop的官方网站 http://www.apache.org/dist/hadoop/common/hadoop-0.21.0/
下载文件hadoop-0.21.0.tar.gz 这个文件是hadoop 0.21.0版本的安装文件了。下载下来以后，
在master机器上的/root下新建一个jz目录，将hadoop的文件拷贝到这个目录下；
4：配置master到slave01和slave02的互信；
在master上执行：
ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
执行完成以后，会在~/.ssh/生成两个文件：id_dsa 和id_dsa.pub。这两个是成对出现，
类似钥匙和锁。再把id_dsa.pub 追加到授权key 里面
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
这个时候，如果你执行
ssh localhost
应该是不需要密码的。这表明你对本机的ssh互信成功。现在需要把这个文件拷贝给slave01和slave02
以下的操作需要对两个slave机器都做一次。以slave01为例：
scp id_dsa.pub root@172.16.3.218:/root/
然后到slave01上执行：
cat id_dsa.pub >> .ssh/authorized_keys
现在从master上直接ssh登陆slave01应该无需密码的，如果还是需要密码，要确保互信成功才能继续下面的操作；
5：修改master机器上的hosts文件为如下内容：
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
172.16.3.217 master master.localdomain
172.16.3.218 slave01 slave01.localdomain
172.16.3.219 slave02 slave02.localdomain
6：将下载的hadoop文件解压：
tar -xvf hadoop-0.21.0.tar
7：修改hadoop的配置
修改/root/jz/hadoop-0.21.0/conf/目录下的master文件，内容如下：
172.16.3.217
修改/root/jz/hadoop-0.21.0/conf/目录下的slaves文件，内容如下：
172.16.3.218
172.16.3.219
修改/root/jz/hadoop-0.21.0/conf/hadoop-env.sh文件的环境变量：
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/jdk1.6.0_20
修改/root/jz/hadoop-0.21.0/conf/core-site.xml配置文件，内容如下：
< ?xml version=”1.0″?>
< ?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoopdata</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>excludes</value>
</property>
</configuration>
修改/root/jz/hadoop-0.21.0/conf/hdfs-site.xml配置文件，内容如下：
< ?xml version=”1.0″?>
< ?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
修改/root/jz/hadoop-0.21.0/conf/mapred-site.xml配置文件，内容如下：
< ?xml version=”1.0″?>
< ?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
8:将/root/jz/hadoop-0.21.0整个目录拷贝到slave01和slave02上；将/etc/profile拷贝到slave01和slave02上；
将/etc/hosts拷贝到slave01和slave02上；
9：启动和关闭hadoop
启动hadoop：
sh /root/jz/hadoop-0.20.2/bin/start-all.sh
关闭hadoop：
sh /root/jz/hadoop-0.20.2/bin/start-all.sh
如果都成功，则安装hadoop成功。
—未完待续—