AI and Big Data
Apache HBase is an open source column-oriented database. It is often described as a sparse, consistent, distributed, multi-dimensional sorted map. HBase is modeled after Google’s “Bigtable: A distributed Storage System for Structured Data”, which can host very large tables with billions of rows, X millions of columns.
项目如果没有采用Java开发，难道就不能用HBase了么？程序猿不会善罢甘休的，有什么语言就会有什么API存在，我还觉得用Java配置时各种缺包错误很烦呢，记得《数学之美》中曾说道：“做技术有术和道两个层面”，知道HBase的架构和一些底层细节是”道”，而使用各种配置和API开发应用则是”术”，而我们就来试试非Java连接HBase。
HBase的第三方接口有Shell, Java, REST和Thrift，可以参考《HBase in Action》chapter 6, REST接口比较慢，使用起来并没有Thrift好。而你可能疑惑什么是Thrift:
作者: Thomas Zhang
清楚地记得，第一次看《The Matrix》这部电影是在小学六年级，在一个英语班上，同小伙伴们一起，围着一个20英寸的小电视，一知半解地看完了一部完全没有中文字幕的英文版本。虽然那时的自己，对这个繁复庞杂的世界架构必然一知半解，但纵然如此，惊喜与震撼已经随着剧终Neo手中听筒的落地，而深埋内心。我开始像Mopheus一样的问自己。What is real? How to define real? 我开始问自己，这个世界是否真的有可能是一个Matrix中的映像，周遭的生活，是否真的有可能是镜花水月，空中楼台。转眼十几年过去了。这十几年中，《The Matrix》三部曲一直珍藏在我的硬盘中，时不时翻出来重新品味，每次又都有不一样的体验。[more…]
十年后的今天，我已经从一个懵懂的少年，成为一个软件工程专业的学生。不得不说，直到接触了软件，接触了计算机体系结构，接触了面向对象的设计思想，对于这部电影，我才有了一个更深刻，更不一样的理解和体会。而经过这一整个学期对面向对象技术的实papapa践与深入了解，对这部经典的理解，似乎又深入了许多。
对于这三部曲，《The Matrix》是开创性的，如疾风骤雨办将一个残酷的真相——现实劈头盖脸的丢到你面前，只留下目瞪口呆的你独自回味着扑面而来的震撼和冲击；《The Matrix ——Reload》是颠覆性的，将你冥思苦想，以为理解了的世界一把推翻，告诉你，真相，其实远远没有那么简单；《The Matrix —— Revolution》是整个三部曲的高潮与结尾，却更是一个开端，那一夜雨战之后，一切都恢复了原装，但是一切，又都变得不同了起来。这篇文章当然不是描述对电影艺术表现抑或情节内容的感受与分析，但是我们仍然要找一个切入点，一个面向对象世界中，和那个电影中的波澜壮阔的世界能够微妙的联系在一起的点。我选择了，Revolution——革命。
在那个雨夜大战之后，Neo和Smith这一对系统中的异常，终于被抹平，Matrix中的世界，似乎又恢复昔日的平静，一切都回到了原点。但，事实真的是这样么？Revolution，革命，究竟又指的是什么呢？
其实事实上，这时候的世界已经发生了本质的变化。还记得《Reload》里面的印度小姑娘Sati么？在从前，一个在系统中本应该被删除的，没有既定目标的冗余程序，现在竟然可以在Architecture眼皮底下自由地玩耍，并且可以完成一项似乎并没有“实用”功能性的代码改写——让夕阳变得美丽——来纪念逝去的人类朋友Neo。这意味着看似恢复原样的Matrix系统其实已经在不经意间完成了一次根本上变化，整个Matrix中，开始允许因为“爱”产生的冗余程序的存在。这是一个近乎根本性的有颠覆意义的改变。
也就是说，Revolution并不仅仅指的是在人类的角度上，Neo通过自我牺牲完成了对Zion的拯救，改变了脱离Matrix的人类宿命一样的被屠杀的命运；在另一个层面上，也指的是在Oracle的指引与Neo的协助之下，整个Matrix系统进行了一次根本性的“革命”。
更确切的对应到软件体系中，我们应该可以说，在电影中Matrix系统进行了一次大规模的系统热升级，在这次升级中，对整个系统的内存回收机制做出了根本上的修改，以使得系统更好的完成承载作为“电池”的人类思维的需求！
在影片中，从Merovingian、Architecture和Oracle的口中，直接间接的，我们可以知道，整个Matrix系统已经先后进行过六次这样的“升级”。这种开发模式，细细想来，与面向对象、敏捷开发中经常应用的迭代式的开发方式，有着异曲同工之处。
但是，一个系统要应用迭代开发方式，那么这个系统一定要满足特定的条件。在敏捷系统中，迭代开发需要满足
1、在项目开发早期需求可能有所变化
2、分析设计人员对应用领域很熟悉
3、高风险项目
4、用户可不同程度地参与整个项目的开发过程
5、使用面向对象的语言或统一建模语言
6、使用CASE工具
7、具有高素质的项目管理者和软件研发团队
其中，面向对象是使用迭代开发方式的必要条件之一。接下来，就来分析一下《The Matrix》世界对应到面向对象方式中的设计。
Mopheus第一次试图向Neo介绍the Matrix的真相时候告诉他，由于历史信息的缺失，对整个世界的构成和来由并不完全清楚，通过推测，才得以窥知这个世界的秘密，他称之为，the desert of the real。可以想见，如果到了一个人工智能发展如此强大的时代，编程语言的类别或许应该一定会超越目前人类的认知和理解。但是从矩阵世界中发生的故事中，我们其实还是可以管中窥豹，发现很多面向对象的蛛丝马迹。下面，就来一一剖析。
在PB Kruchten经典的论文中可以知道，我们可以通过4+1视对面向对象设计的架构进行描述。部署视图显示的是系统的实际部署情况，它是为了便于理解系统在一组处理节点上的物理分布。所以，这里就先从高层次上，与物理世界关系方面，对矩阵世界的部署视图进行尝试性的描述。
上图就是矩阵世界中地球上各个系统的部署视图，可以看出，地表上，机器城维护者作为能源的巨大的分布式电力供应系统，而这个电力系统以人供电，每个人又通过统一的接口连接在Matrix则个虚拟现实系统之中，为系统统一管理、控制。人类反抗组织以地下的Zion为基地，通过庞大的地下管道系统进行反抗机器城的活动，从电池组合Matrix中解救出有意愿脱离Matrix的人类。通过电影中可以得知，上面各个系统之间存在着繁复的交互关系，接下来就用4+1视图中逻辑视图中的类图，来形象的展示各个系统和各个部件之间的关系。
上图可见，Matrix系统由两部分组成，接入系统的人，以及保证系统正常运行的各类程序。Architecture与Oracle均是系统程序的一部分，拥有相对于其他程序高得多的权限，并且拥有与机器城交互的能力。每隔一定时间，系统中不稳定的人群中会由Oracle指引和选择出现The One，The One多重继承了程序和人的属性，帮助Matrix完成每一次的系统更新。
以上就是从The Matrix电影有限的信息中——the desert of real——中推测出来的，关于Matrix面向对象层次上抽象出来的设计描述。满足了这个前提条件，接下来，让我们回归正题，来讨论Matrix中对于“迭代式开发”技术的应用。
迭代开发，是面向对象以及目前流行的敏捷开发方式中最常用的一种开发方式。迭代，是快速原型与面向对象技术的结合。在迭代式开发方法中，整个开发工作被组织为一系列的短小的、固定长度的小项目，被称为一系列的迭代。每一次迭代都包括了需求分析、设计、实现与测试。采用这种方法，开发工作可以在需求被完整地确定之前启动，并在一次迭代中完成系统的一部分功能或业务逻辑的开发工作。再通过客户的反馈来细化需求，并开始新一轮的迭代。下面是迭代开发的流程示意图。
暂且不做说明，同时放上the Matrix中系统升级的时间线如下图。
图中可以很明显的看出，第一个有着完美世界架构的矩阵很快失败了，正如迭代开发中第一个原型，通常都是抛弃原型，是快速搭建起来的，以获取用户需求为目的的原型。通常，在使用过后，由于修改变更的可能性极大，所以通常会被抛弃而重新设计。而接下来的五次矩阵升级，都是同一版本的提升性升级，the One选择了重建Zion，新版本的矩阵仅仅是清楚了之前的Bug数据，便继续运行。但是Neo，第六次的选择，作出了不同的选择，选择相信爱，拯救Trinity和Zion。最终Neo牺牲了自己，挽救了矩阵世界和人类世界。同时，系统终于加入了决定性的变更——类似人类的情感——爱以及全新的内存回收机制。
多么典型的迭代开发流程！
由此可以发现，不论是标准的迭代开发的图示，抑或定义，抑或矩阵的升级流程图，FeedBack都起到了极其重要的作用。而Feedback的来源，正是每一任the One，从Zion，这些排斥现有矩阵的反抗者那里获取的。所以，其实一切的一切，不论是从矩阵中的逃脱、Zion中的自由还是最终Zion的获救，其实都是Machine City，为了更好地设计Matrix系统，进行的宿命办的迭代。
When some see coincidence, I see consequence. When other see chance, I see cost.
其实每次变更，都是一次选择，一次一次机遇，也一定会付出一定的代价。
在迭代中，代价由于快速迭代，所以只会影响这一次迭代的原型。
而同样的优点，在矩阵中，则被计算机所完美的利用。正如Architecture所说的，我们能够承受每一次系统迭代升级的代价，也随时准备承受这样的代价。
机器的精明，确实人类宿命的悲哀。
那么，再这样完善的系统精巧的设计中，人类是不是真的就没有机会完成复兴呢？其实人类反击的机遇也恰恰蕴含在这近乎完美的设计之中。
在这样一个世界中，Matrix的世界中，人类与机器其实遵循同样的定律生活在一个虚拟的空间中。也就是说，机器和人类的思维，其实共享着同样的接口。面向接口设计的完美实例！这也就意味着，如果信息隐藏做的足够好，那么，既然人无法分辨机器，那么反过来，在Matrix中，自然可以做到让机器无法分辨出人！这不正是人的机遇么~？！
在《The Matrix》的动画版《Animatrix》中，描述了这样一幕，人类反抗组织通过私有Matrix，成功的与机器交流，并“招安了”许多机器作为保护自己的助手。同样的枷锁，蕴含在面向对象中的精灵，原来同样可以加诸机器头上！
Architecture：You play a dangerous game!
Oracle: Change is always dangerous.
Architecture：Just how long, do you think this peace can last.
Oracle: As long as it can.
May so be the software system!
在完成Hadoop配置后，我们可以开始HBase的安装和配置了。
对于HBase，我只想说走对了路就成功了一半，选对了版本就省事好多。之前下载的是0.94版，按照官方的配置，连Standalone都跑不通，纠结了半天，放弃治疗，选用0.96版本，一切顺利。
1. 前提条件
MacOS 10.9
Java安装好
Standalone模式是单机的，基于Local FileSystem，不需要Hadoop，用于开发或测试。
[more…]
2. 下载HBase
hbase-0.96.1-hadoop2-bin.tar.gz
3. 解压安装
[shell]
$ tar xzvf hbase-0.96.1-hadoop2-bin.tar.gz
$ mv hbase-0.96.1-hadoop2 hbase
[/shell]
同样需要把hbase安装到Home目录下~/hbase, 对于我是/Users/lgrcyanny/hbase
4. 配置环境变量
[shell]
$ vim ~/.bashrc
#Config HBase
export HBASE_HOME=/Users/lgrcyanny/hbase
export PATH=$PATH:$HBASE_HOME/bin
[/shell]
5. 编辑hbase-site.xml
[shell]
$ cd ~/hbase
$ mkdir -p mydata/hbase
$ mkdir -p mydata/zookeeper
$ vim conf/hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///Users/lgrcyanny/hbase/mydata/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/Users/lgrcyanny/hbase/mydata/zookeeper</value>
</property>
</configuration>
[/shell]
6. 编辑hbase-env.sh
[shell]
$ cd ~/hbase
$ vim conf/hbase-env.sh
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents/Home/
export HBASE_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.security.krb5.conf=/dev/null"
[/shell]
7. 关于/etc/hosts
对于Ubuntu的用户，需要修改/etc/hosts，将127.0.1.1改为127.0.0.1，因为HBase的环回地址默认是127.0.0.1，但官方的quikstart中提到0.96以后的版本不需要修改，MacOS中确实不用修改，之前用0.94的版本，因为环回地址的问题总是报错也无法修复，这是0.94的bug。如果是Ubuntu用户，可以尝试不修改hosts看能不能跑通。
8. 启动并测试HBase
[shell]
$ start-hbase.sh
starting master, logging to /Users/lgrcyanny/hbase/logs/hbase-lgrcyanny-master-Cyanny-MacBook-Air.local.out
$ hbase shell
> status
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/lgrcyanny/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lgrcyanny/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2014-02-06 14:27:29,472 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
1 servers, 0 dead, 3.0000 average load
> create ‘test’, ‘cf’
> put ‘test’, ‘row1’, ‘cf:a’, ‘value1’
> list
TABLE
test
1 row(s) in 0.1060 seconds
=> ["test"]
> scan ‘test’
ROW COLUMN+CELL
row1 column=cf:a, timestamp=1391653468438, value=value1
1 row(s) in 0.0650 seconds
> exit
$ stop-hbase.sh
stopping hbase……………..
[/shell]
如果上面的步骤完成，HBase的Standalone模式就安装成功。
1. 修改hbase-site.xml
[shell]
$ cd ~/hbase
$ vim conf/hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/Users/lgrcyanny/hbase/mydata/zookeeper</value>
</property>
</configuration>
[/shell]
2. 启动Hadoop和HBase
[shell]
$ start-dfs.sh
$ start-hbase.sh
localhost: starting zookeeper, logging to /Users/lgrcyanny/hbase/logs/hbase-lgrcyanny-zookeeper-Cyanny-MacBook-Air.local.out
starting master, logging to /Users/lgrcyanny/hbase/logs/hbase-lgrcyanny-master-Cyanny-MacBook-Air.local.out
localhost: starting regionserver, logging to /Users/lgrcyanny/hbase/logs/hbase-lgrcyanny-regionserver-Cyanny-MacBook-Air.local.out
$ jps
11107 HQuorumPeer
11427 Jps
11180 HMaster
11276 HRegionServer
9155 SecondaryNameNode
9064 DataNode
8990 NameNode
[/shell]
不需要启动yarn, 启动HDFS即可，我们可以看到HBase的伪分布式模式中，启动了内嵌的Zookeeper，启动了Master和RegionServer。
3. 查看状态
查看Master Status： http://localhost:60010/master-status
查看Region Server Status： http://localhost:60030/rs-status
4. 测试HBase
可以按照Standalone模式中的第8步，打开hbase shell进行测试。
到这里，如果顺利那么就一切都配置好了，可能花费了你1~2个小时，还是要恭喜，我前后花费了2天。
Anyway，探索的过程还是很愉快，Good Luck！
过完春节，新年开始了，闲暇的时光弄了一下Hadoop和HBase，之前也配置过Hadoop，不过是1.x的版本和现在2.2的版本不一样了，HBase的官网推荐使用Hadoop2.x，配置HBase确实花费了点时间，网上的各种教程相似但各异，自己遇到的问题和方法也值得记下来，下次再配置时也可以查看一下。
Hadoop官方Guide的配置不够详细，需要参考各种博客，以下是我个人配置的方法：
1. 操作系统
我用的是MacOS 10.9， 尽量在配置时使用Linux系统如Ubuntu，用Windows需要下载安装Cygwin，可能会麻烦些。
[more…]
2. 下载Java
到Oracle下载J2SE，1.6版本以上，下载最新版1.7即可。
在Mac下安装后，Java的Home路径是：
“/Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents/Home/“
而不是很多博客中用的：
“/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK”
应该是MacOS10.8后有更新了。
3. 下载Hadoop 2.X
我用的版本是Hadoop2.2.0
点击下载 hadoop-2.2.0.tar.gz
[shell]
$ tar xzvf hadoop-2.2.0.tar.gz
$ mv hadoop-2.2.0 hadoop
[/shell]
我安装hadoop到路径/Users/lgrcyanny/hadoop，即当前Mac用户的Home目录下，为了方便没有创建新的用户hadoop，在配置时可以自己将“lgrcyanny”替换为自己的用户名。
4. 配置ssh localhost
Hadoop的Master和Slave的通信采用ssh，单机版的Hadoop也需要ssh。
Ubuntu用户可以”apt-get install ssh”， mac用户自带ssh
[shell]
$ ssh-keygen -t rsa # 如果之前配置过GitHub，rsa key是存在的，请不要覆盖即可
$ ssh lgrcyanny@localhost mkdir -p .ssh
$ cat .ssh/id_rsa.pub | ssh lgrcyanny@localhost ‘cat >> .ssh/authorized_keys’
$ ssh lgrcyanny@localhost "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
$ ssh localhost
Last login: Thu Feb 6 10:22:40 2014
[/shell]
5. 配置环境变量
[shell]
$ cd ~
$ vim .bashrc
#Java properties
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents/Home/
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=.:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
#Hadoop variables
export HADOOP_INSTALL=/Users/lgrcyanny/hadoop # Please change to your installation path
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HADOOP_YARN_HOME=$HADOOP_INSTALL
$ source .bashrc
[/shell]
6. 编辑hadoop-env.sh
[shell]
$ cd ~/hadoop/etc/hadoop
$vim hadoop-env.sh
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents/Home/
export HADOOP_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.security.krb5.conf=/dev/null"
[/shell]
7. 查看hadoop version
[shell]
$ hadoop version
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /Users/lgrcyanny/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar
[/shell]
8. 编辑core-site.xml
[shell]
$ cd ~/hadoop
$ mkdir -p mydata/tmp
$ vim etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/Users/lgrcyanny/hadoop/mydata/tmp</value>
</property>
</configuration>
[/shell]
9. 编辑yarn-site.xml
Hadoop 2.x的特点就是引入了YARN框架，这是和之前Hadoop 1.x不同的方面
[shell]
$ cd ~/hadoop
$ vim etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>10240</value>
<description>the amount of memory on the NodeManager in GB</description>
</property>
</configuration>
[/shell]
10. 编辑mapred-site.xml
[shell]
$ cd ~/hadoop
$ mkdir -p mydata/mapred/temp
$ mkdir -p mydata/mapred/local
$ mv etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
$ vim etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.cluster.temp.dir</name>
<value>/Users/lgrcyanny/hadoop/mydata/mapred/temp</value>
<description>The temp dir for map reduce</description>
<final>true</final>
</property>
<property>
<name>mapreduce.cluster.local.dir</name>
<value>/Users/lgrcyanny/hadoop/mydata/mapred/local</value>
<description>The local dir for map reduce</description>
<final>true</final>
</property>
</configuration>
[/shell]
11. 编辑hdfs-site.xml
[shell]
$ cd ~/hadoop
$ mkdir -p mydata/hdfs/namenode
$ mkdir -p mydata/hdfs/datanode
$ vim etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/Users/lgrcyanny/hadoop/mydata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/Users/lgrcyanny/hadoop/mydata/hdfs/datanode</value>
</property>
</configuration>
[/shell]
12. Format HDFS
[shell]
$ hadoop namenode -format
……
14/02/06 13:22:41 INFO namenode.NameNode: SHUTDOWN_MSG:
/****
SHUTDOWN_MSG: Shutting down NameNode at Cyanny-MacBook-Air.local/192.168.1.103
****/
[/shell]
13. 启动Hadoop
[shell]
$ start-dfs.sh
$ start-yarn.sh
$ jps # Java Virtual Machine Process Status Tool
10377 Jps
10224 NodeManager
10146 ResourceManager
9155 SecondaryNameNode
9064 DataNode
8990 NameNode
[/shell]
Hadoop 1.x采用start-all.sh启动，而Hadoop2.x拆分为start-dfs.sh, start-yarn.sh，我想是因为如果使用HBase时，只需要HDFS的服务，而一般不需要YARN的服务，这样就不会占用太多的内存。
14. Web查看Hadoop
查看HDFS的状态：http://localhost:50070
查看YARN的状态: http://localhost:8088
15. 测试Hadoop，使用Hadoop的WordCount example
[shell]
$ cd ~/hadoop
$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
14/02/06 13:28:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/06 13:28:40 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/06 13:28:41 INFO input.FileInputFormat: Total input paths to process : 2
14/02/06 13:28:41 INFO mapreduce.JobSubmitter: number of splits:2
………
[/shell]
你可以通过http://localhost:8088/cluster 查看MapReduce的运行状态
Congratulations, Hadoop 2.2.0安装完毕，接下来我们就要开始安装HBase了。
1. 当启动Hadoop，jps查看后没有datanode怎么办？
某一天我遇到这个问题，方法是将datanode和namenode, 以及tmp文件夹删除，重新format，再重启Hadoop。
[shell]
$ cd ~/hadoop
$ stop-dfs.sh
$ stop-yarn.sh
$ rm -r mydata/hdfs/datanode
$ rm -r mydata/hdfs/namenode
$ rm -r mydata/tmp
$ mkdir -p mydata/tmp
$ hadoop namenode -format
$ start-dfs.sh
$ start-yarn.sh
[/shell]
There are many node scaffoldings based on Mongoddb, but MySQL is rare. This is a simple scaffolding built on express and mysql.
node-express-mysql-scaffolding
[more…]
1. Register with fullname,username, email, passord, very simple
2. Login with passport-local strategy
3. Twitter Bootstrap Support
Note: I just want keep the scaffolding clean, no more complex function, and keep it flexible.
NOTE: You need to have node.js, MySQL Server installed
1. Clone the project
[shell]
$ git clone https://github.com/lgrcyanny/node-express-mysql-scaffolding.git
$ npm install
$ cp config/config.disk.js config/config.js
[/shell]
Please config your MySQL in the config.js;
2. Install MySQL server
3. Start MySQL service
4. Build the database
[shell]
$ mysql -u root -p
> create database scaffolding
> quit
$ mysql -u root -pyourpassword scaffolding < scaffolding.sql
[/shell]
5. Start Node.js Server
[shell]
$ npm start
[/shell]
6. Then visit http://localhost:3000/
Thanks to node-express-mongoose-demo, it’s a great scaffolding, but it still took me 2 days to migrate from MongoDB based scaffolding to MySQL scaffolding, and the node-express-mongoose-demo has too many Login Support which is too complicated.
-app/
|__controllers/
|__models/
|__mailer/
|__views/
-config/
|__routes.js
|__config.js
|__passport.js (auth config)
|__express.js (express.js configs)
|__middlewares/ (custom middlewares)
-public/
Tests are not shipped now, I will write tests later.
[shell]
$ npm test
[/shell]
(The MIT License)
Hadoop is a great framework for distributed large data computing. But Hadoop is not the silver bullet. Hadoop fits not very well in such cases as follow:
Applications that require real-time query, and low-latency access to data in tens of milliseconds will not work well with Hadoop.
Hadoop is not a substitute for a database. Database index records that will gains low-latency and fast response.
But if you really want to replace the database for real time needs, try HBase, which is a column-oriented database for random and real time read/write.[more…]
Hadoop is not fit for structured data with strong relationship. Hadoop works well for semi-structured and unstructured data. It stores data in files, doesn’t index them like RDBMS. Therefore, each ad hoc query for Hadoop is processed by MapReduce job which will bring the latency cost.
How big the data is big enough for Hadoop? The answer is TB or PB. When your analytics data is only tens of GB, Hadoop is heavy. Don’t follow the fashion and use Hadoop, just follow your requirements.
When there are too many small files, the NameNode will hit its memory limit where the block map and the metadata are hosted. And to handle the NameNode bottleneck, Hadoop introduces HDFS Federation.
HDFS is in write-once-and-read-many-times way. When there is too much files update needs, Hadoop won’t support that.
MapReduce is a simple programming model in parallel. But for MapReduce parallelism, you need to make sure each MR job and the data where the job runs on is independent from all the others. Every MR shouldn’t have dependencies.
But if you want to do some data sharing during MR, you can do like this:
Resources:
Part 0 Hadoop Overview
Part 1 Hadoop HDFS Review
Part 2 Hadoop HDFS Federation
Part 3 Hadoop HDFS High Availability(HA)
Part 4 Hadoop MapReduce Overview
Part 5 Hadoop MapReduce 1 Framework
Part 6 Hadoop MapReduce 2 (YARN)
Part 7 Hadoop isn’t Silver Bullet
For large clusters with more than 4000 nodes, the classic MapReduce framework hit the scalability problems.
Therefore, a group in Yahoo began to design the next generation MapReduce in 2010, and in 2013 Hadoop 2.x releases MapReduce 2, Yet Another Resource Negotiator (YARN) to remedy the sociability shortcoming.
The fundamental idea of YARN is to split up the two major functionalities of the JobTracker: resource management (job scheduling) and job monitoring into separate daemons. YARN has a global Resource Manager (RM) and per-application Application Master(AM).[more…]
Figure1 YARN High-level Overview
As shown in Figure1, the YARN involves more entities than classic MapReduce 1 :
Resource Manager consists of two components: Scheduler and ApplicationsManager.
Scheduler is in charge of allocating resources. The resource Container incorporates elements such as memory, cup, disk, network etc. Scheduler just has the resource allocation function, has no responsible for job status monitoring. And the scheduler is pluggable, can be replaced by other scheduler plugin-in.
The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific Application Master, and it provides restart service when the container fails.
The MapReduce job is just one type of application in YARN. Different application can run on the same cluster with YARN framework. That’s the beauty of YARN.
Figure2 MapReduce with YARN
As shown in Figure2, it is the MapReduce process with YARN, there are 11 steps, and we will explain it in 6 steps the same as the MapReduce 1 framework. They are Job Submission, Job Initialization, Task Assignment, Task Execution, Progress and Status Updates, and Job Completion.
Clients can submit jobs with the same API as MapReduce 1 in YARN. YARN implements its ClientProtocol, the submission process is similar to MapReduce 1.
When the Resource Manager(RM) receives the call submitApplication(), RM will hands off the job to its scheduler. The job initialization is as follows:
For small job, called uber job, which is the one has less than 10 mappers and only one reducer, or the input split size is smaller than a HDFS block, the Application Manager will run the job on its own JVM sequentially. This policy is different from MapReduce 1 which will ignore the small jobs on a single TaskTracker.
For large job, the Application Master will launches a new node with new NodeManager and new container, in which run the task. This can run job in parallel and gain more performance.
Application Master calls the job setup method to create the job’s output directory. That’s different from MapReduce 1, where the setup task is called by each task’s TaskTracker.
When the job is very large so that it can’t be run on the same node as the Application Master. The Application Master will make request to the Resource Manager to negotiate more resource container which is in piggybacked on heartbeat calls. The task assignment is as follows:
This way is more flexible than MapReduce 1. In MapReduce 1, the TaskTrackers have a fixed number of slots and each task runs in a slot. Each slot has fixed memory allowance which results in two problems. For small task, it will waste of memory, and for large task which need more memeory, it will lack of memory.
In YARN, the memory allocation is more fine-grained, which is also the beauty of YARE resides in.
After the task has been assigned the container by the Resource Manger’s scheduler, the Application Master will contact the NodeManger which will launch the task JVM.
The task execution is as follows:
When the job is running under YARN, the mapper or reducer will report its status and progress to its Application Master every 3 seconds over the umbilical interface. The Application Master will aggregate these status reports into a view of the task status and progress. While in MapReduce 1, the TaskTracker reports status to JobTracker which is responsible for aggregating status into a global view.
Moreover, the Node Manger will send heartbeats to the Resource Manager every few seconds. The Node Manager will monitoring the Application Master and the recourse container usage like cpu, memeory and network, and make reports to the Resource Manager. When the Node Manager fails and stops heartbeat the Resource Manager, the Resource Manager will remove the node from its available resource nodes pool.
The client pulls the status by calling getStatus() every 1 second to receive the progress updates, which are printed on the user console. User can also check the status from the web UI. The Resource Manager web UI will display all the running applications with links to the web UI where displays task status and progress in detail.
Figure 3 YARN Progress and Status Updates
Every 5 second the client will check the job completion over the HTTP ClientProtocol by calling waitForCompletion(). When the job is done, the Application Master and the task containers clean up their working state and the outputCommitter’s job cleanup method is called. And the job information is archived as history for later interrogation by user.
Resources:
Part 0 Hadoop Overview
Part 1 Hadoop HDFS Review
Part 2 Hadoop HDFS Federation
Part 3 Hadoop HDFS High Availability(HA)
Part 4 Hadoop MapReduce Overview
Part 5 Hadoop MapReduce 1 Framework
Part 6 Hadoop MapReduce 2 (YARN)
Part 7 Hadoop isn’t Silver Bullet
For MapReduce programming, a developer can run a MapReduce job by simply calling submit() or waitForCompletion() on a job object. This method abstracts the job processing details away from developer. But there is a great of job processing behind the scene that we will consider in this section.
Hadoop 2.x has released new MapReduce framework implementation called YARN or MapReduce 2, for traditional MapReduce is the classic framework which is also called MapReduce 1. YARN is compatible with MapReduce 1. [more…]
Figure1 Classic MapReduce Framework
As shown in Figure 1, there are four independent entities in the framework:
As the Figure 1 show, a MapReduce processing including 10 steps, and in short, that is:
When the client call submit() on job object. An internal JobSubmmitter Java Object is initiated and submitJobInternal() is called. If the clients calls the waiForCompletion(), the job progresss will begin and it will response to the client with process results to clients until the job completion.
JobSubmmiter do the following work:
When the JobTracker receives the call submitJob(), it will put the call into an internal queue from where the job scheduler will pick it up and initialize it. The initialization is done as follow:
TaskTrackers send heartbeat periodically to JobTracker Node to tell it if it is alive or ready to get a new task. The JobTracker will allocate a new task to the ready TaskTracker. Task assignment is as follows:
When the TaskTracker has been assigned a task. The task execution will be run as follows:
Figure 2 Classic MapReduce Framework Progress and Status Updates
After clients submit a job. The MapReduce job is a long time batching job. Hence the job progress report is important. What consists of the Hadoop task progress is as follows:
As shown in Figure 2, when a task is running, the TaskTracker will notify the JobTracker its task progress by heartbeat every 5 seconds.
And mapper and reducer on the child JVM will report to TaskTracker with it’s progress status every few seconds. The mapper or reducers will set a flag to indicate the status change that should be sent to the TaskTracker. The flag is checked in a separated thread every 3 seconds. If the flag sets, it will notify the TaskTracker of current task status.
The JobTracker combines all of the updates to produce a global view, and the Client can use getStatus() to get the job progress status.
When the JobTracker receives a report that the last task for a job is complete, it will change its status to successful. Then the JobTracker will send a HTTP notification to the client which calls the waitForCompletion(). The job statistics and the counter information will be printed to the client console. Finally the JobTracker and the TaskTracker will do clean up action for the job.
Resources:
Part 0 Hadoop Overview
Part 1 Hadoop HDFS Review
Part 2 Hadoop HDFS Federation
Part 3 Hadoop HDFS High Availability(HA)
Part 4 Hadoop MapReduce Overview
Part 5 Hadoop MapReduce 1 Framework
Part 6 Hadoop MapReduce 2 (YARN)
Part 7 Hadoop isn’t Silver Bullet
MapReduce is a parallel programming model and an associated implementation for processing and generating large data sets. The MapReduce model consists of two phrases: map and reduce. A map task is to process a key/value pair to generate a set of intermediate key/value pairs, and a reduce task is to merge all intermediate values associated with the same intermediated key.[more…]
Hadoop MapReduce is based on the MapReduce paper in 2006. This processing model is automatic parallelization and distribution. It provides a clean abstraction for programmers. MapReduce programs are usually written in java, and can be written in any scripting language like Ruby, Python, PHP using Hadoop Streaming, or in C++ using Hadoop Pipes. MapReduce abstracts all the ‘housekeeping’ away from the developer. Developer can concentrate simply on writing the Map and Reduce functions.
Figure1 MapReduce Data Flow
As shown in Figure1, A MapReduce process consists of two phrases: map phrase and the reduce phrase. Let’s consider them in detail.
A MapReduce job is a unit of work that the client wants to be run, including the input data, the MapReduce program and the configuration information. Hadoop runs the job by dividing it into tasks: map tasks and reduce tasks.
Hadoop divides the input data into fixed-size pieces call input splits or splits. Each map task runs on each split, which runs the user-defined map function. All of the map tasks runs in parallel. Usually, the split size is a HDFS block, 64MB or 128MB.
If the file is less than 64MB or 128MB, it will not be split. And the file will occupy one block, results in a waste of storage.
Map tasks usually runs on its local HDFS data, or the data near the node that runs the map task. Data Locality saves bandwidth and decreases dependencies.
The input value for map task is key/value pair. For example, in the WordCount example by Hadoop, the input value for map task: the key is the line offset whining the file, which we can ignore in our map function, the value is the line in the file.
In the map function, developers will process the value of each line, make sure the output is key/value pair, WorkCount again, the output for map function is like ‘<apple, 1>, <pear, 1>…’, key is the word, value is 1.
Map tasks output is written to the local disk, not to HDFS, then the reduce task will use these intermediate output to do merge work. Because storing these intermediate data to HDFS with replication would be overkill. And if the map task fails before the reduce task consume the output, Hadoop will automatically start another map task on another node that will re-create the output.
After map tasks done, the job tracker will start the reduce task. The reducer input is the intermediate mapper output.
Between the map task and the reduce task is the well known shuffle and sort. Hadoop will sort the intermediate map output by key. And each reduce task will run on map output with the same key. In WordCount example, the same key means the same word, like ‘<apple, 1> … <apple, 1>’, will be assigned to a reduce task, the reduce function just sum up the value and calculate the word count for ‘apple’.
Reduce tasks don’t have the advantage of data locality, the sorted map output have to be transferred across the network to the node where the reduce task is running.
Then the reduce task will merge the data with the user-defined reduce function. The reduce output is normally stored in HDFS for reliability. For each reduce output block, the first replica is stored on the local node, while the other two replicas are stored on off-rack nodes.
We can see that no reduce task can start until every map task has finished. Will the mapper become a bottleneck? Hadoop uses the ‘Speculative execution’ to mitigate against this:
Often, Mappers produce large amounts of intermediate data, which have to be transferred to Reducers that will result in a lot of network traffic.
To minimize the data transferred between Mapper and Reducer, Hadoop introduces the combiner function to be run on the map output, and combine the Mapper output and generate the Reducer Input.
Combiner is like a ‘Mini-Reducer’, runs locally on a the same node as Mapper. The output from the combiner is sent to the Reducers. Combiners decrease the amount of the network traffic required during the shuffle and sort phase.
Resources:
Part 0 Hadoop Overview
Part 1 Hadoop HDFS Review
Part 2 Hadoop HDFS Federation
Part 3 Hadoop HDFS High Availability(HA)
Part 4 Hadoop MapReduce Overview
Part 5 Hadoop MapReduce 1 Framework
Part 6 Hadoop MapReduce 2 (YARN)
Part 7 Hadoop isn’t Silver Bullet