HBase--通过idea搭建源码阅读环境
该篇文章主要是,搭建一个HBase的源码阅读环境。主要是通过github下载源码,编译导入到开发环境idea中,然后启动相关服务,做DeBug调试来阅读相关源码。
下载HBase源码
通过github下载相关源码(我下载到目录:/opt/dev/idea/hbase
)
- 注意:默认我们克隆时,会把所有的历史commit信息也会克隆下载。
对于一个非常活跃的开源项目来说这些历史信息非常占用空间,下载时就会很慢。
我们可以在git clone 后面加一个--depth 1
这样只克隆最新的一次提交(深度为1),就会快很多,如下:git clone https://github.com/apache/hbase.git –depth 1
编译
- 命令:mvn clean install -DskipTests
启动服务
这里启动服务主要是启动HBase单节点,HBase单节点只有一个HMaster线程,zookeeper以及HRegionServer都是以进程的形式存在,在后面的源码阅读中可以很好的发现。
配置HBase服务
- 在idea中添加一个配置(Configuration)- Application。
- 主要配置(注意自己的的项目路径):
Main Class
:org.apache.hadoop.hbase.master.HMasterVM Options
:-Dlog4j.configuration=file:/opt/dev/idea/hbase/conf/log4j.properties参数
:start工作目录
:/opt/dev/idea/hbase
- 如图所示:
配置HBase客户端
- 在idea中添加一个配置(Configuration)- Application。
- 主要配置(注意自己的的项目路径):
Main Class
:org.jruby.MainVM Options
:-Dhbase.ruby.sources=/opt/dev/idea/hbase/hbase-shell/src/main/ruby -Dlog4j.configuration=file:/opt/dev/idea/hbase/conf/log4j.properties参数
:/opt/dev/idea/hbase/bin/hirb.rb工作目录
:/opt/dev/idea/hbase
- 如图所示:
修改配置
- hbase-common 下面的resource,修改下面几个配置,如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38<property>
<name>hbase.tmp.dir</name>
<value>/opt/tmp/hbase-${user.name}</value>
<description>Temporary directory on the local filesystem.
Change this setting to point to a location more permanent
than '/tmp', the usual resolve for java.io.tmpdir, as the
'/tmp' directory is cleared on machine restart.
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>txd</value>
<description>Comma separated list of servers in the ZooKeeper ensemble
(This config. should have been named hbase.zookeeper.ensemble).
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper ensemble servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
this is the list of servers which hbase will start/stop ZooKeeper on as
part of cluster start/stop. Client-side, we will take this list of
ensemble members and put it together with the hbase.zookeeper.property.clientPort
config. and pass it into zookeeper constructor as the connectString
parameter.
</description>
</property>
<property>
<name>hbase.defaults.for.version.skip</name>
<value>true</value>
<description>Set to true to skip the 'hbase.defaults.for.version' check.
Setting this to true can be useful in contexts other than
the other side of a maven generation; i.e. running in an
IDE. You'll want to set this boolean to true to avoid
seeing the RuntimeException complaint: "hbase-default.xml file
seems to be for and old version of HBase (\${hbase.version}), this
version is X.X.X-SNAPSHOT"</description>
</property>
启动服务
- 服务启动,debug等调试操作
查看
- 通过zk查看
- 下载一个zookeeper,在
/etc/profile
或用户目录下配置好环境变量,如:export ZK_HOME=/opt/local/zookeeper-3.4.12
- 就可以运行zk的客户端了,查看hbase信息如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34txd:Downloads tangxuandong$ cd /opt/local/zookeeper-3.4.12/
txd:zookeeper-3.4.12 tangxuandong$ cd bin/
txd:bin tangxuandong$ zkCli.sh
Connecting to localhost:2181
2019-04-09 19:00:16,993 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2019-04-09 19:00:16,997 [myid:] - INFO [main:Environment@100] - Client environment:host.name=10.106.195.85
2019-04-09 19:00:16,997 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_161
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/local/zookeeper-3.4.12/bin/../build/classes:/opt/local/zookeeper-3.4.12/bin/../build/lib/*.jar:/opt/local/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/local/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/opt/local/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/opt/local/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/opt/local/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/opt/local/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/opt/local/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/opt/local/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/opt/local/zookeeper-3.4.12/bin/../conf:.:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/Users/tangxuandong/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/var/folders/yx/g41l6w1n62bgm721hhlr6b1c0000gn/T/
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Mac OS X
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=x86_64
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:os.version=10.14.4
2019-04-09 19:00:16,999 [myid:] - INFO [main:Environment@100] - Client environment:user.name=tangxuandong
2019-04-09 19:00:17,000 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/Users/tangxuandong
2019-04-09 19:00:17,000 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/local/zookeeper-3.4.12/bin
2019-04-09 19:00:17,001 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@255316f2
Welcome to ZooKeeper!
2019-04-09 19:00:17,028 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-09 19:00:17,094 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2019-04-09 19:00:17,104 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x16a01a1936b0007, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, hbase]
[zk: localhost:2181(CONNECTED) 1] ls /hbase
[meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, master, running, draining, hbaseid, table]
[zk: localhost:2181(CONNECTED) 2]
- 浏览器访问
访问:http://localhost:16010/