hadoop2.5.2配置httpfs服务
测试环境
- ubuntu 14.04 单机
- hadoop2.5.2 伪分布式
- jdk1.7
作用
- 通过HttpFs你可以在浏览器里面管理HDFS上的文件,功能同hadoop shell相似
- HttpFs还提供了一套REST 风格的API可以用来管理HDFS
缺点
- 网络安全性
- 需要配置acl
修改core-site.xml
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>localhost</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
添加上述两个配置
hadoop.proxyuser.root.hosts
允许通过httpfs方式访问hdfs的主机名或者域名;hadoop.proxyuser.root.groups
允许访问的客户端的用户组
配置httpfs环境变量
export CATALINA_BASE=/opt/work/hadoop/share/hadoop/httpfs/tomcat
临时配置,也可以在配置文件配置环境变量
重启hadoop集群
root@localhost:/opt/nfs#/opt/work/hadoop/sbin//stop-all.sh
root@localhost:/opt/nfs#/opt/work/hadoop/sbin//start-all.sh
启动httpfs服务
root@localhost:/opt/git/hadoop_dev/hdfsToInfoBright# /opt/work/hadoop/sbin/httpfs.sh start
Setting HTTPFS_HOME: /opt/work/hadoop
Setting HTTPFS_CONFIG: /opt/work/hadoop/etc/hadoop
Sourcing: /opt/work/hadoop/etc/hadoop/httpfs-env.sh
Setting HTTPFS_LOG: /opt/work/hadoop/logs
Setting HTTPFS_TEMP: /opt/work/hadoop/temp
Setting HTTPFS_HTTP_PORT: 14000
Setting HTTPFS_ADMIN_PORT: 14001
Setting HTTPFS_HTTP_HOSTNAME: localhost
Setting HTTPFS_SSL_ENABLED: false
Setting HTTPFS_SSL_KEYSTORE_FILE: /root/.keystore
Setting HTTPFS_SSL_KEYSTORE_PASS: password
Using CATALINA_BASE: /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Setting HTTPFS_CATALINA_HOME: /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Setting CATALINA_OUT: /opt/work/hadoop/logs/httpfs-catalina.out
Setting CATALINA_PID: /tmp/httpfs.pid
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dhttpfs.home.dir=/opt/work/hadoop -Dhttpfs.config.dir=/opt/work/hadoop/etc/hadoop -Dhttpfs.log.dir=/opt/work/hadoop/logs -Dhttpfs.temp.dir=/opt/work/hadoop/temp -Dhttpfs.admin.port=14001 -Dhttpfs.http.port=14000 -Dhttpfs.http.hostname=localhost -Dhttpfs.ssl.enabled=false -Dhttpfs.ssl.keystore.file=/root/.keystore -Dhttpfs.ssl.keystore.pass=password
Using CATALINA_BASE: /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Using CATALINA_HOME: /opt/work/hadoop/share/hadoop/httpfs/tomcat
Using CATALINA_TMPDIR: /opt/work/hadoop/share/hadoop/httpfs/tomcat//temp
Using JRE_HOME: /usr/local/jdk1.7.0
Using CLASSPATH: /opt/work/hadoop/share/hadoop/httpfs/tomcat//bin/tomcat-juli.jar:/opt/work/hadoop/share/hadoop/httpfs/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/httpfs.pid
Existing PID file found during start.
Removing/clearing stale PID file.
root@localhost:/opt/git/hadoop_dev/hdfsToInfoBright#
并查看启动日志,并确定环境变量的正确
CATALINA_BASE
的值
浏览器访问httpfs
http://localhost:14000/
页面显示:HttpFs service, service base URL at /webhdfs/v1.
根目录为:/webhdfs/v1
curl访问httpfs
root@localhost:/opt/nfs# curl -i -X PUT -T /opt/test.json "http://localhost:14000/webhdfs/v1/tmp/test.json?op=CREATE&data=true&user.name=root" -H "Content-Type:application/octet-stream"
将文件上传到hdfs
http访问文件:
http://localhost:14000/webhdfs/v1/tmp/test.json?user.name=root&op=open
即可下载test.json文件
考虑
- httpfs的安全性
- httpfs的性能:httpfs是一个tomcat启动的web服务,当数据文件非常大,上传下载的性能将骤减
- httpfs的并发访问:tomcat的并发问题
相关推荐
hadoop2.5.2开发所用到的基础jar包,本人亲测
hadoop 2.5.2 64位native包
hadoop2.5.2 + eclipse + win32位环境安装及开发环境搭建全套资源及说明 Hi:发帖目的是因为Hadoop开发环境搭建太麻烦了,涉及很多版本和工具组件的问题,自己也走了不少弯路,把资源集中一下,让大家少走弯路,也...
hadoop2.5.2在windows下搭建开发环境并进行测试可以运行。
文件是hadoop 2.5.2源码,从官方网站下载的,供国内访问不正常的同学下载
eclipse环境下集成hadoop2.5.2时候需要的jar包 ant已经编译好了的可以直接用
从0开始一步一步安装和部署Hadoop 2.5.2
hadoop-2.5.2.tar.gz
hadoop-2.5.2 hadoop2.5.2
网上百度的最高只有hadoop2.4的,于是自己想着也编译一版来,于是就基于hadoop2.5.2的源码在windows8 64位系统安装时自己编译的 包含 hadoop.dll hadoop.pdb hadoop.lib hadoop.exp winutils.exe winutils.pdb ...
2、大数据环境-安装Hadoop2.5.2伪分布式傻瓜教程 原创
本文针对Hadoop2.5.2的集群安装过程进行详细的讲解,根据本文可顺利完成Hadoop集群的安装和基本使用。
Hadoop2.5.2搭建真分布式,利用三台虚拟机搭建的简单真分布式例子
hadoop2.5.2的本地模式、伪分布式集群、分布式集群和HDFS系统的高可用的环境搭建.docx hadoop2.5.2的本地模式、伪分布式集群、分布式集群和HDFS系统的高可用的环境搭建.docx hadoop2.5.2的本地模式、伪分布式集群、...
CentOS7.0上搭建Hadoop2.5.2,云计算框架
hadoop2.5.2学习13-MR之新浪微博TF-IDF算法简介
Hadoop在win 7 x64的已编译文件,只包含windows下独有的文件。与官网hadoop.apache.org下的程序包合并使用,放于hadoop\bin文件夹下。 包含文件:hadoop.dll, hadoop.exp, hadoop.lib, hadoop.pdb, winutils.exe, ...
自己整理,亲测,已经安装了几遍,喜欢的可以下载看看,
在Windows7 x64 + Eclipse + Hadoop 2.5.2搭建MapReduce开发环境,下载的文件中包括下载的文件包括:hadoop 2.5.2.tar.gz,hadoop-common-2.2.0-bin-master.zip,hadoop-eclipse-plugin-2.5.2.jar。应用这些软件的...