Twitter开源的Heron快速安装部署教程

2022-01-04 00:32:32

什么是Heron？

Twitter使用Storm实时分析海量数据已经有好几年了，并在2011年将其开源。该项目稍后开始在Apache基金会孵化，并在2015年秋天成为*项目。Storm以季度为发布周期，并且向着人们期望的稳定版前进。但一直以来，Twitter都在致力于开发替代方案Heron，因为Storm无法满足他们的实时处理需求。Twitter现在已经用Heron完全替换了Storm。前者现在每天处理“数10TB的数据，生成数10亿输出元组”，在一个标准的单词计数测试中，“吞吐量提升了6到14倍，元组延迟降低到了原来的五到十分之一”，硬件减少了2/3。当被问到Twitter是否会开源Heron时，Ramasamy说“在短时间内不会，但长期来看可能。”然而就在2016年5月25日，Twitter正式宣布Heron开源。Twitter工程经理Karthik Ramasamy在博客上宣布了这一消息。

快速安装Heron

一、使用安装脚本下载heron的二进制文件

安装脚本的下载地址是https://github.com/twitter/heron/releases。

博主用的是centos7，所以下载了centos版本的安装文件。

[heron@spark ~]$ ls

heron-client-install-0.14.4-centos.sh  heron-tools-install-0.14.4-centos.sh

[heron@spark ~]$

下载完后开始安装

[root@spark heron]# chmod +x heron-client-install-0.14.4-centos.sh

[root@spark heron]# ./heron-client-install-0.14.4-centos.sh

--warning=no-timestamp

Heron client installer

----------------------

Uncompressing..tar xfz /usr/local/heron/heron-client.tar.gz -C /usr/local/heron --warning=no-timestamp

....

Heron is now installed!

Make sure you have "/usr/local/bin" in your path.

See http://heronstreaming.io/docs/getting-started for how to use Heron.

heron.build.version : '0.14.4-1-g8de8003'

heron.build.time : Wed Oct 5 17:55:30 PDT 2016

heron.build.timestamp : 1475715348000

heron.build.host : tw-mbp-kramasamy

heron.build.user : kramasamy

heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f

heron.build.git.status : Clean

[root@spark heron]#

默认安装的路径是/usr/local/bin。

也可以指定安装路径 ./heron-client-install-VERSION-PLATFORM.sh --user

安装完后，配置一下 /etc/profile

vi /etc/profile

export PATH=$PATH:/usr/local/bin

然后source一下

source /etc/profile

安装另一个文件

[root@spark heron]# chmod +x heron-tools-install-0.14.4-centos.sh

[root@spark heron]# ./heron-tools-install-0.14.4-centos.sh

--warning=no-timestamp

Heron tools installer

---------------------

Uncompressing..tar xfz /usr/local/herontools/heron-tools.tar.gz -C /usr/local/herontools --warning=no-timestamp

....

Heron Tools is now installed!

Make sure you have "/usr/local/bin" in your path.

See http://heronstreaming.io/docs/getting-started for how to use Heron.

heron.build.version : '0.14.4-1-g8de8003'

heron.build.time : Wed Oct 5 17:55:30 PDT 2016

heron.build.timestamp : 1475715348000

heron.build.host : tw-mbp-kramasamy

heron.build.user : kramasamy

heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f

heron.build.git.status : Clean

[root@spark heron]#

最后确认一下是否安装完成

[root@spark local]# heron version

heron.build.version : '0.14.4-1-g8de8003'

heron.build.time : Wed Oct 5 17:55:30 PDT 2016

heron.build.timestamp : 1475715348000

heron.build.host : tw-mbp-kramasamy

heron.build.user : kramasamy

heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f

heron.build.git.status : Clean

[root@spark local]#

二、提交一个topology

[root@spark local]# heron submit local /usr/local/heron/examples/heron-examples.jar com.twitter.heron.examples.ExclamationTopology ExclamationTopology --deploy-deactivated

INFO: Using config file under /usr/local/heron/conf/local

INFO: Launching topology 'ExclamationTopology'

INFO: Topology 'ExclamationTopology' launched successfully

INFO: Elapsed time: 4.417s.

[root@spark local]#

提交了topology后，heron并不会激活它。

我们查看一下topology是否成功启动和工作目录

[root@spark local]# ls -al ~/.herondata/topologies/local/root/ExclamationTopology

container_1_exclaim1_1.pid  heron-executor-0.stdout       log-files                    metrics.json.metricsmgr-2.1

container_1_exclaim1_3.pid  heron-executor-1.pid          metrics.json.metricsmgr-0.0  metrics.json.metricsmgr-2.2

container_1_word_5.pid      heron-executor-1.stderr       metrics.json.metricsmgr-0.1  metrics.json.metricsmgr-2.3

container_2_exclaim1_2.pid  heron-executor-1.stdout       metrics.json.metricsmgr-0.2  metrics.json.metricsmgr-2.4

container_2_exclaim1_4.pid  heron-executor-2.pid          metrics.json.metricsmgr-0.3  metricsmgr-0.pid

container_2_word_6.pid      heron-executor-2.stderr       metrics.json.metricsmgr-0.4  metricsmgr-1.pid

ExcalamationTopology.defn   heron-executor-2.stdout       metrics.json.metricsmgr-1.0  metricsmgr-2.pid

heron-conf                  heron-shell-0.pid             metrics.json.metricsmgr-1.1  release.yaml

heron-core                  heron-shell-1.pid             metrics.json.metricsmgr-1.2  stmgr-1.pid

heron-examples.jar          heron-shell-2.pid             metrics.json.metricsmgr-1.3  stmgr-2.pid

heron-executor-0.pid        heron-tmaster.pid             metrics.json.metricsmgr-1.4

heron-executor-0.stderr     java-f6c01842-started.stderr  metrics.json.metricsmgr-2.0

[root@spark local]#

所有实例的日志文件都可以在工作目录下的日志文件中找到。

[root@spark local]# ls -al ~/.herondata/topologies/local/root/ExclamationTopology/log-files/

container_1_exclaim1_1.log.0      container_2_word_6.log.0                    heron-ExclamationTopology-scheduler.log.0.lck

container_1_exclaim1_1.log.0.lck  container_2_word_6.log.0.lck                heron-shell.log

container_1_exclaim1_3.log.0      gc.container_1_exclaim1_1.log               metricsmgr-0.log.0

container_1_exclaim1_3.log.0.lck  gc.container_1_exclaim1_3.log               metricsmgr-0.log.0.lck

container_1_word_5.log.0          gc.container_1_word_5.log                   metricsmgr-1.log.0

container_1_word_5.log.0.lck      gc.container_2_exclaim1_2.log               metricsmgr-1.log.0.lck

container_2_exclaim1_2.log.0      gc.container_2_exclaim1_4.log               metricsmgr-2.log.0

container_2_exclaim1_2.log.0.lck  gc.container_2_word_6.log                   metricsmgr-2.log.0.lck

container_2_exclaim1_4.log.0      gc.metricsmgr.log

container_2_exclaim1_4.log.0.lck  heron-ExclamationTopology-scheduler.log.0

三、启动 Heron Tracker

[root@spark local]# heron-tracker

INFO: 11/09/2016 09:55:17 +0000 Connecting to file state with rootpath: /root/.herondata/repository/state/local

INFO: 11/09/2016 09:55:17 +0000 State watch triggered for topologies.

INFO: 11/09/2016 09:55:17 +0000 Adding new topology: ExcalamationTopology, state_manager: local

INFO: 11/09/2016 09:55:17 +0000 Registering a watch with uid: 6d218788-1c8c-4f43-806a-a217c59ab4b5

INFO: 11/09/2016 09:55:17 +0000 No execution state found for: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology execution state: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Tracker has started

INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Running on port: 8888

INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology pplan: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Using config file: /usr/local/herontools/conf/heron_tracker.yaml

INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology tmaster: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology scheduler location: ExcalamationTopology

INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology

启动后可以通过浏览器来查看刚刚提交的topology是否成功，端口号是8888

了解更多的heron tracker，请参考Heron Tracker Rest API

四、启动 Heron UI

[root@spark ~]# heron-ui

INFO: 11/09/2016 10:02:25 +0000 Listening at http://0.0.0.0:8889

INFO: 11/09/2016 10:02:25 +0000 Using tracker url: http://localhost:8888

INFO: 11/09/2016 10:02:29 +0000 200 GET /topologies (192.168.25.57) 52.86ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/bootstrap.js (192.168.25.57) 37.41ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/jquery.min.js (192.168.25.57) 24.41ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/jquery-ui-1.8.23.custom.min.js (192.168.25.57) 16.54ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/moment.min.2.8.3.js (192.168.25.57) 15.74ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/underscore-min.1.6.0.js (192.168.25.57) 9.57ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/css/visstyle.css (192.168.25.57) 6.50ms

INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/common.js?v=aea0e809de29e66079cc8ee4c7ea16f7 (192.168.25.57) 2.54ms

INFO: 11/09/2016 10:02:30 +0000 200 GET /static/js/alltopologies.js?v=f6ba00d23089084d4cb32f8bfcf351a0 (192.168.25.57) 3.48ms

INFO: 11/09/2016 10:02:30 +0000 200 GET /static/img/logo54x54.png (192.168.25.57) 9.42ms

INFO: 11/09/2016 10:02:31 +0000 200 GET /static/icomoon/fonts/icomoon.woff (192.168.25.57) 21.88ms

INFO: 11/09/2016 10:02:31 +0000 200 GET /topologies/list.json?format=json (192.168.25.57) 6.98ms

启动后可以通过浏览器查看，端口号8889。（heron-tracker必须启动，不然报错）

五、topology的管理命令

在第二步中，我们已经提交了topology，现在我们可以用命令激活、停止、杀死等命令

heron activate local ExclamationTopology

heron deactivate local ExclamationTopology

heron kill local ExclamationTopology

博文地址：http://julyme.com/20161108/21.html

个人博客：http://julyme.com

码农公寓

相关文章