怎么在CentOS7中编译Hadoop2.x
这篇文章主要介绍“怎么在CentOS7中编译Hadoop2.x”，在日常操作中，相信很多人在怎么在CentOS7中编译Hadoop2.x问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答”怎么在CentOS7中编译Hadoop2.x”的疑惑有所帮助！接下来，请跟着小编一起来学习吧！
成都创新互联长期为1000多家客户提供的网站建设服务，团队从业经验10年，关注不同地域、不同群体，并针对不同对象提供差异化的产品和服务；打造开放共赢平台，与合作伙伴共同营造健康的互联网生态环境。为旌德企业提供专业的成都网站设计、成都网站建设，旌德网站改版等技术服务。拥有十多年丰富建站经验和众多成功案例,为您定制开发。
环境
VirtualBox-5.0, CentOS-7.1, Hadoop-2.7.1
1、关于Hadoop本地库
1.1、可能存在的问题
Hadoop发行包自带的库文件可能与本地操作系统不兼容。特别是在Hadoop-2.5.1之前的版本，自带的本地库都是32位的，64位操作系统无法使用，只能使用java自带的，而java自带的速度会比较慢。
查看本地库版本
[wukong@bd01 native]$ pwd /home/wukong/local/hadoop-2.7.1/lib/native [wukong@bd01 native]$ ll total 4928 -rw-r--r-- 1 wukong wukong 1383476 Jun 29 14:15 libhadoop.a -rw-r--r-- 1 wukong wukong 1634592 Jun 29 14:15 libhadooppipes.a lrwxrwxrwx 1 wukong wukong 18 Jun 29 14:15 libhadoop.so -> libhadoop.so.1.0.0 -rwxr-xr-x 1 wukong wukong 807942 Jun 29 14:15 libhadoop.so.1.0.0 -rw-r--r-- 1 wukong wukong 476666 Jun 29 14:15 libhadooputils.a -rw-r--r-- 1 wukong wukong 447388 Jun 29 14:15 libhdfs.a lrwxrwxrwx 1 wukong wukong 16 Jun 29 14:15 libhdfs.so -> libhdfs.so.0.0.0 -rwxr-xr-x 1 wukong wukong 282388 Jun 29 14:15 libhdfs.so.0.0.0 [wukong@bd01 native]$ file ./libhadoop.so.1.0.0 ./libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xd3720bf1ab8325bdde6e129bc386a86c888accc9, not stripped [wukong@bd01 native]$
1.2、可能存在的WARN级别日志
启停hdfs/mapreduce程序后可能出现的告警日志
2015-08-01 18:15:57,940 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2、编译过程
2.1、安装编译相关的包（需联网）
[root@bd01 ~]# yum install svn [root@bd01 ~]# yum install autoconfautomake libtool cmake [root@bd01 ~]# yum install ncurses-devel [root@bd01 ~]# yum install openssl-devel [root@bd01 ~]# yum install gcc*
2.2、安装maven
# 到apache官网下载最新稳定版 wget # 解压 tar -xzvf apache-maven-3.3.3-bin.tar.gz # 放到/opt下面 mv apache-maven-3.3.3 /opt # 配置环境变量 export JAVA_HOME=/opt/jdk1.7.0_79 export M2_HOME=/opt/apache-maven-3.3.3 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/li b/tools.jar export PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin export MAVEN_OPTS=-Xmx512m # 更新环境变量 [root@bd01 apache-maven-3.3.3]# source /etc/profile # 检查是否配置成功 [root@bd01 apache-maven-3.3.3]# echo $PATH /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jdk1.7.0_79/bin:/home/wukong/local/hadoop-2.7.1/bin:/home/wukong/local/hadoop-2.7.1/sbin:/opt/jdk1.7.0_79/bin:/opt/apache-maven-3.3.3/bin [root@bd01 apache-maven-3.3.3]# mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00) Maven home: /opt/apache-maven-3.3.3 Java version: 1.7.0_79, vendor: Oracle Corporation Java home: /opt/jdk1.7.0_79/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix" [root@bd01 apache-maven-3.3.3]# exit exit [wukong@bd01 pack]$ mvn -version -bash: mvn: command not found [wukong@bd01 pack]$ source /etc/profile [wukong@bd01 pack]$ mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00) Maven home: /opt/apache-maven-3.3.3 Java version: 1.7.0_79, vendor: Oracle Corporation Java home: /opt/jdk1.7.0_79/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix" [wukong@bd01 pack]$
注意用户权限
root@bd01 apache-maven-3.3.3]# pwd /opt/apache-maven-3.3.3 [root@bd01 apache-maven-3.3.3]# cd .. [root@bd01 opt]# ll total 4 drwxr-xr-x 6 wukong wukong 92 Aug 2 15:11 apache-maven-3.3.3 drwxr-xr-x 8 10 143 4096 Apr 11 02:53 jdk1.7.0_79 [root@bd01 opt]# cd apache-maven-3.3.3/ [root@bd01 apache-maven-3.3.3]# ll total 32 drwxr-xr-x 2 wukong wukong 91 Aug 2 15:11 bin drwxr-xr-x 2 wukong wukong 41 Aug 2 15:11 boot drwxr-xr-x 3 wukong wukong 60 Apr 22 19:55 conf drwxr-xr-x 3 wukong wukong 4096 Aug 2 15:11 lib -rwxr-xr-x 1 wukong wukong 19091 Apr 22 19:58 LICENSE -rwxr-xr-x 1 wukong wukong 182 Apr 22 19:58 NOTICE -rwxr-xr-x 1 wukong wukong 2541 Apr 22 19:55 README.txt [root@bd01 apache-maven-3.3.3]#
2.3、安装protobuf
# 下载 [wukong@bd01 pack]$ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz # 解压 [wukong@bd01 pack]$ tar -xzvf ./protobuf-2.5.0.tar.gz # 编译及安装 [wukong@bd01 pack]$ cd protobuf-2.5.0 [wukong@bd01 protobuf-2.5.0]$ ./configure # 以下三步时间比较长，而且有warning [wukong@bd01 protobuf-2.5.0]$ make [wukong@bd01 protobuf-2.5.0]$ make check # make install时要使用root账户，否则会因为权限的问题报错 [root@bd01 protobuf-2.5.0]$ make install # 验证 [root@bd01 lib]# protoc --version libprotoc 2.5.0 [wukong@bd01 ~]$ protoc --version libprotoc 2.5.0
make check后可能会得到类似以下结果，检查看看是否有error，没有就ok
PASS: protobuf-test PASS: protobuf-lazy-descriptor-test PASS: protobuf-lite-test PASS: google/protobuf/compiler/zip_output_unittest.sh PASS: google/protobuf/io/gzip_stream_unittest.sh make[5]: Entering directory `/home/wukong/pack/protobuf-2.5.0/src' make all-am make[6]: Entering directory `/home/wukong/pack/protobuf-2.5.0/src' make[6]: Nothing to be done for `all-am'. make[6]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src' make[5]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src' ============================================================================ Testsuite summary for Protocol Buffers 2.5.0 ============================================================================ # TOTAL: 5 # PASS: 5 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ make[4]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src' make[3]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src' make[2]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src' make[1]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'
注意，和hadoop-2.7.1对应的protobuf版本是2.5.0，如果版本不对，会报错大致如下：
[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 47.107 s [INFO] Finished at: 2015-08-02T16:59:47+08:00 [INFO] Final Memory: 53M/135M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn-rf :hadoop-common
2.4、重新编译本地库
2.4.1、获取hadoop源代码
wget或者svn或者直接去官网下
[wukong@bd01 pack]$ ll total 233916 -rw-rw-r-- 1 wukong wukong 8042383 Apr 28 23:12 apache-maven-3.3.3-bin.tar.gz drwxr-xr-x 15 wukong wukong 4096 Jun 29 14:04 hadoop-2.7.1-src -rw-rw-r-- 1 wukong wukong 18226773 Jul 30 22:57 hadoop-2.7.1-src.tar.gz -rw-rw-r-- 1 wukong wukong 210606807 Jul 28 09:49 hadoop-2.7.1.tar.gz drwxr-xr-x 10 wukong wukong 4096 Aug 2 15:28 protobuf-2.6.1 -rw-rw-r-- 1 wukong wukong 2641426 Oct 23 2014 protobuf-2.6.1.tar.gz
2.4.1、解压
[wukong@bd01 pack]# tar -xzvf hadoop-2.7.1-src.tar.gz [wukong@bd01 pack]# cd hadoop-2.7.1-src [wukong@bd01 hadoop-2.7.1-src]$ pwd /home/wukong/pack/hadoop-2.7.1-src [wukong@bd01 hadoop-2.7.1-src]$ ll total 76 -rw-r--r-- 1 wukong wukong 13110 Jun 29 14:03 BUILDING.txt drwxr-xr-x 2 wukong wukong 4096 Aug 2 16:06 dev-support drwxr-xr-x 3 wukong wukong 30 Aug 2 16:06 hadoop-assemblies drwxr-xr-x 2 wukong wukong 20 Aug 2 16:06 hadoop-client drwxr-xr-x 10 wukong wukong 4096 Aug 2 16:06 hadoop-common-project drwxr-xr-x 2 wukong wukong 20 Aug 2 16:07 hadoop-dist drwxr-xr-x 6 wukong wukong 103 Aug 2 16:06 hadoop-hdfs-project drwxr-xr-x 9 wukong wukong 4096 Aug 2 16:06 hadoop-mapreduce-project drwxr-xr-x 3 wukong wukong 30 Aug 2 16:06 hadoop-maven-plugins drwxr-xr-x 2 wukong wukong 20 Aug 2 16:06 hadoop-minicluster drwxr-xr-x 3 wukong wukong 30 Aug 2 16:07 hadoop-project drwxr-xr-x 2 wukong wukong 37 Aug 2 16:06 hadoop-project-dist drwxr-xr-x 16 wukong wukong 4096 Aug 2 16:06 hadoop-tools drwxr-xr-x 3 wukong wukong 56 Aug 2 16:06 hadoop-yarn-project -rw-r--r-- 1 wukong wukong 15429 Jun 29 14:03 LICENSE.txt -rw-r--r-- 1 wukong wukong 101 Jun 29 14:03 NOTICE.txt -rw-r--r-- 1 wukong wukong 18893 Jun 29 14:03 pom.xml -rw-r--r-- 1 wukong wukong 1366 Jun 29 14:03 README.txt [wukong@bd01 hadoop-2.7.1-src]$
2.4.2、编译hadoop源代码
# 非常漫长非常漫长的等待 [wukong@bd01 hadoop-2.7.1-src]$ mvn clean package -Pdist,native -DskipTests -Dtar # 最终编译结果 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 7.458 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 5.576 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 6.810 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.343 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 3.293 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 5.637 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 6.570 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 14.101 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.555 s] [INFO] Apache Hadoop Common ............................... SUCCESS [03:54 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.327 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 22.779 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.053 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [08:56 min] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [02:19 min] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [01:36 min] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 9.181 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.065 s] [INFO] hadoop-yarn ........................................ SUCCESS [ 0.076 s] [INFO] hadoop-yarn-api .................................... SUCCESS [02:27 min] [INFO] hadoop-yarn-common ................................. SUCCESS [03:22 min] [INFO] hadoop-yarn-server ................................. SUCCESS [ 0.564 s] [INFO] hadoop-yarn-server-common .......................... SUCCESS [ 22.455 s] [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 40.593 s] [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 8.402 s] [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 20.187 s] [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 41.425 s] [INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 13.118 s] [INFO] hadoop-yarn-client ................................. SUCCESS [ 16.388 s] [INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 7.473 s] [INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.205 s] [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 5.137 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 3.379 s] [INFO] hadoop-yarn-site ................................... SUCCESS [ 0.192 s] [INFO] hadoop-yarn-registry ............................... SUCCESS [ 11.037 s] [INFO] hadoop-yarn-project ................................ SUCCESS [ 5.957 s] [INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.084 s] [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [01:05 min] [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 44.305 s] [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 8.763 s] [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 20.268 s] [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 17.986 s] [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 24.200 s] [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 3.840 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.104 s] [INFO] hadoop-mapreduce ................................... SUCCESS [ 4.579 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 21.775 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [01:16 min] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 11.316 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 27.178 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 8.318 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 5.153 s] [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 5.326 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 7.849 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 27.793 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 9.929 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 57.588 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 21.386 s] [INFO] Apache Hadoop Client ............................... SUCCESS [ 20.907 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 20.113 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 39.010 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 15.505 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.051 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [07:26 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 44:52 min [INFO] Finished at: 2015-08-03T00:37:02+08:00 [INFO] Final Memory: 134M/492M [INFO] ------------------------------------------------------------------------
2.4.2、编译后的成果
这里只看了native库文件
[wukong@bd01 native]$ pwd /home/wukong/pack/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1/lib/native [wukong@bd01 native]$ ll total 4644 -rw-rw-r-- 1 wukong wukong 1235998 Aug 3 00:30 libhadoop.a -rw-rw-r-- 1 wukong wukong 1606616 Aug 3 00:31 libhadooppipes.a lrwxrwxrwx 1 wukong wukong 18 Aug 3 00:30 libhadoop.so -> libhadoop.so.1.0.0 -rwxrwxr-x 1 wukong wukong 724522 Aug 3 00:30 libhadoop.so.1.0.0 -rw-rw-r-- 1 wukong wukong 471882 Aug 3 00:31 libhadooputils.a -rw-rw-r-- 1 wukong wukong 433780 Aug 3 00:30 libhdfs.a lrwxrwxrwx 1 wukong wukong 16 Aug 3 00:30 libhdfs.so -> libhdfs.so.0.0.0 -rwxrwxr-x 1 wukong wukong 272202 Aug 3 00:30 libhdfs.so.0.0.0 [wukong@bd01 native]$
可以把库文件拷贝到hadoop的相关目录中，或者直接使用用编译好的hadoop
到此，关于“怎么在CentOS7中编译Hadoop2.x”的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注创新互联网站，小编会继续努力为大家带来更多实用的文章！
网站标题：怎么在CentOS7中编译Hadoop2.x
文章分享：http://hbruida.cn/article/ggooec.html
