memcached演练(3) 使用apache+搭建双节点tomcat集群

2021-07-15 01:14:54

上篇文章，已经明确了tomcat集群的必要性，主要从高可用和高并发量考虑。一般使用tomcat集群搭建，都要使用前置设一个nginx或apache作个反向代理，将请求转发给后端tomcat。使用tomcat集群，必然会引来session数据共享问题。如何解决这个问题？可借助memcached实现。本章主要搭建一个tomcat双节点集群，算“准备”篇吧。

主要内容

  1. 测试单台tomcat的吞吐率

  2. 搭建apache+mod_jk+2个tomcat集群环境

环境准备

确认apache服务位置及版本号

1
2
3
4
5
6
[root@hadoop1 httpd]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Feb 22 2013 11:17:44
[root@hadoop1 httpd]# which httpd
/usr/sbin/httpd
#apache主目录:/etc/httpd

确认JDK 及tomcat服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[hadoop@hadoop1 tomcat-sets]$ ls /u01/tomcat-sets
apache-tomcat-node1
[hadoop@hadoop1 apache-tomcat-node1]$ cat /proc/meminfo 
MemTotal:        1937148 kB
...
[hadoop@hadoop1 /]$ /u01/tomcat-sets/apache-tomcat-node1/bin/version.sh 
Using CATALINA_BASE:   /u01/tomcat-sets/apache-tomcat-node1
Using CATALINA_HOME:   /u01/tomcat-sets/apache-tomcat-node1
Using CATALINA_TMPDIR: /u01/tomcat-sets/apache-tomcat-node1/temp
Using JRE_HOME:        /usr/local/java/jdk1.6.0_45
...
Server version: Apache Tomcat/7.0.33
Server built:   Nov 18 2012 04:15:21
Server number:  7.0.33.0
OS Name:        Linux
OS Version:     2.6.32-358.el6.i686
Architecture:   i386
JVM Version:    1.6.0_45-b06
JVM Vendor:     Sun Microsystems Inc.

1.测试单台tomcat的吞吐率

1.1准备一个静态测试资源（staticProtocal.html，大小50K）

资源可以通过http://192.168.163.146:8080/examples/staticProtocal.html访问

1
2
3
4
5
6
7
8
[hadoop@hadoop1 examples]$ ll /u01/tomcat-sets/apache-tomcat-node1/webapps/examples
total 72
-rw-r--r--.  1 hadoop hadoop  1179 Nov 19  2012 index.html
drwxr-xr-x. 22 hadoop hadoop  4096 Aug 10 18:48 jsp
drwxr-xr-x.  3 hadoop hadoop  4096 Aug 10 18:48 servlets
-rw-rw-r--.  1 hadoop hadoop 50663 Aug 12 21:09 staticProtocal.html
drwxr-xr-x.  7 hadoop hadoop  4096 Aug 10 18:48 WEB-INF
drwxr-xr-x.  2 hadoop hadoop  4096 Aug 10 18:48 websocket

1.2启动apache-tomcat-node1，端口为默认的8080

1.3 在另外一台主机使用apache ab命令测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
C:\Apache24\bin>ab  -n 10000 -c 4  http://192.168.163.146:8080/examples/staticProtocal.html
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        Apache-Coyote/1.1
Server Hostname:        192.168.163.146
Server Port:            8080
 
Document Path:          /examples/staticProtocal.html
Document Length:        50663 bytes
 
Concurrency Level:      4
Time taken for tests:   12.065 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      509120000 bytes
HTML transferred:       506630000 bytes
Requests per second:    828.81 [#/sec] (mean)
Time per request:       4.826 [ms] (mean)
Time per request:       1.207 [ms] (mean, across all concurrent requests)
Transfer rate:          41207.63 [Kbytes/sec] received
...

经过测试,单台tomcat的结果是：QPS:828.81

都有这样一个共识，apache处理静态文件的能力，比tomcat强大。是骡子是马，比比看。

1.4 简单编辑apache配置文件httpd.conf。(最好修改之前备份下)

1
2
3
4
5
$ cat /etc/httpd/conf/httpd.conf 
...
ServerName localhost:80
DocumentRoot "/var/www/html"
...

1.5 将刚才的staticProtocal.html文件，复制到DocumentRoot“/var/www/html”目录下。

1
2
$ ls /var/www/html/
staticProtocal.html

1.6 启动httpd

1
2
$  service httpd start
Starting httpd: [  OK  ]

这时候可以通过链接访问资源：http://192.168.163.146/staticProtocal.html

1.7 测试apache吞吐量

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
C:\Apache24\bin>ab  -n 10000 -c 4  http://192.168.163.146/staticProtocal.html
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Finished 10000 requests
 
 
Server Software:        Apache/2.2.15
Server Hostname:        192.168.163.146
Server Port:            80
 
Document Path:          /staticProtocal.html
Document Length:        50663 bytes
 
Concurrency Level:      4
Time taken for tests:   11.902 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      509350000 bytes
HTML transferred:       506630000 bytes
Requests per second:    840.17 [#/sec] (mean)
Time per request:       4.761 [ms] (mean)
Time per request:       1.190 [ms] (mean, across all concurrent requests)
Transfer rate:          41790.94 [Kbytes/sec] received

经过测试,单台apache的结果是：QPS:840.17

显而易见，apache处理静态文件能力略高于tomcat.

接下来，那就利用apache搭建一个双节点tomcat集群，顺带验证下QPS能达到多少。这是下一小节内容。

2.搭建apache+mod_jk+2个tomcat集群环境

2.1下载mod_jk并安装

要安装httpd-devel,否则会configure: error: Cannot find the WebServer

1
2
3
4
5
6
7
8
9
#  yum install httpd-devel
## 安装完httpd-devel之后，新添加目录/usr/sbin/apxs
##下载
# wget 
##解压
 # tar zxvf tomcat-connectors-1.2.41-src.tar.gz 
 # cd tomcat-connectors-1.2.41-src/native
 # ./buildconf.sh --with-apxs=/usr/sbin/apxs
 # make

到这里mod_jk.so已经生成了，最后将mod_jk.so放到apache的modules 目录下 

1
cp /u01/software/tomcat-connectors-1.2.41-src/native/apache-2.0/mod_jk.so  /etc/httpd/modules

2.2配置mod_jk

请参考：http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html

mod_jk：支持平台Linux, FreeBSD, AIX, HP-UX, MacOS X, Solaris，Apache 1.3 and/or 2.x

mod_jk: 支持Tomcat 3.2 to Tomcat 8.

redirector 使用协议：ajp12 and ajp13

ajp12 协议针对Tomcat 3.2.x and 3.3.x.

ajp13协议支持tomcat 4+

mod_jk:2个组成部分。

The Apache HTTP Server module。由平台决定，一般是mod_jk.so, mod_jk.nlm or MOD_JK.SRVPGM

workers.properties ：描述Tomcat 相关的host(s) and port(s)文件。

2.3启用jk_module模块

将下面内容片段，加入/etc/httpd/httpd.conf

1
2
3
4
5
6
# Load mod_jk module
LoadModule    jk_module  modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel  info
include "conf/mod_jk.conf"

2.4 配置后端tomcat 主机及端口映射,编辑或新建文件，/etc/httpd/workers.properties

配置说明参照：https://tomcat.apache.org/connectors-doc/reference/workers.html

1
2
3
4
5
worker.list=tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=127.0.0.1
worker.tomcat1.type=ajp13
#worker.tomcat1.lbfactor=1

现在达到的效果是apache反代1个tomcat。所以不需要配置额外的负载均衡等信息。

2.5 编辑conf/mod_jk.conf，增加资源映射

1
2
3
4
#LoadModule jk_module modules/mod_jk.so
JkMount  /*.jsp tomcat1
JkMount  /examples/* tomcat1
#JkWorkersFile conf/workers.properties

注释的两行，由于已经在httpd.conf中已经配置，所以不需要重复配置。

2.6 启动tomcat,及apache

笔者，在这一步经过测试，不能正常达到代理效果。

通过查看日志信息(/var/log/httpd/mod_jk.log)

[Fri Aug 12 22:49:23.051 2016] [9739:3078039376] [error] init_jk::mod_jk.c (3574): Initializing shm:/etc/httpd/logs/jk-runtime-status.9739 errno=13. Load balancing workers will not function properly.


通过参照网上解决方案：临时将selinux关闭

1
2
3
4
5
6
确认selinux状态
 /usr/sbin/sestatus -v 
临时关闭
  setenforce 0 
永久关闭
  修改/etc/selinux/config    设置为 SELINUX=disabled

访问测试，没有问题

http://192.168.163.146/examples/servlets/

http://192.168.163.146:8080/examples/servlets/


到了这里，完成了apache+单台tomcat的配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
C:\Apache24\bin>ab  -n 10000 -c 4  http://192.168.163.146/examples/staticProtocal.html
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        Apache/2.2.15
Server Hostname:        192.168.163.146
Server Port:            80
 
Document Path:          /examples/staticProtocal.html
Document Length:        50663 bytes
 
Concurrency Level:      4
Time taken for tests:   13.555 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      509320000 bytes
HTML transferred:       506630000 bytes
Requests per second:    737.73 [#/sec] (mean)
Time per request:       5.422 [ms] (mean)
Time per request:       1.356 [ms] (mean, across all concurrent requests)
Transfer rate:          36693.36 [Kbytes/sec] received

很遗憾,apache+单台tomcat组合的QPS:737.73

接下来，开始搭建apache+双节点环境。

3.搭建apache+mod_jk+2个tomcat集群环境

3.1复制tomcat环境为node2

1
2
3
4
5
6
7
8
9
[root@hadoop1 ~]# cd /u01/tomcat-sets/
[root@hadoop1 tomcat-sets]# ls
apache-tomcat-node1
[root@hadoop1 tomcat-sets]# cp -R  apache-tomcat-node1/ apache-tomcat-node2
[root@hadoop1 bin]# chown -R hadoop:hadoop /u01/tomcat-sets 
[root@hadoop1 tomcat-sets]# ll
total 8
drwxr-xr-x. 9 hadoop hadoop 4096 Aug 10 18:48 apache-tomcat-node1
drwxr-xr-x  9 hadoop hadoop 4096 Aug 13 08:36 apache-tomcat-node2

3.2 修改apache-tomcat-node2节点的相关端口配置

<Server port="8005" shutdown="SHUTDOWN">
<Server port="8006" shutdown="SHUTDOWN">
   

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
  <Connector port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8445" />

<Connector port="8009" protocol="AJP/1.3" 

redirectPort="8445" />

<Connector port="8019" protocol="AJP/1.3" 

redirectPort="8445" />



配置好了简单测试下

1
2
3
4
5
6
7
8
访问以下均正常
 
http://192.168.163.146:8080/examples/staticProtocal.html 
 
ps -ef |grep tomcat
[root@hadoop1 apache-tomcat-node2]# ps -ef |grep tomcat
root      2797     1  0 08:31 pts/0    00:00:03 /usr/bin/java -Djava.util.logging.config.file=/u01/tomcat-sets/apache-tomcat-node1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/u01/tomcat-sets/apache-tomcat-node1/endorsed -classpath /u01/tomcat-sets/apache-tomcat-node1/bin/bootstrap.jar:/u01/tomcat-sets/apache-tomcat-node1/bin/tomcat-juli.jar -Dcatalina.base=/u01/tomcat-sets/apache-tomcat-node1 -Dcatalina.home=/u01/tomcat-sets/apache-tomcat-node1 -Djava.io.tmpdir=/u01/tomcat-sets/apache-tomcat-node1/temp org.apache.catalina.startup.Bootstrap start
root      2993     1  7 08:47 pts/0    00:00:00 /usr/bin/java -Djava.util.logging.config.file=/u01/tomcat-sets/apache-tomcat-node2/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/u01/tomcat-sets/apache-tomcat-node2/endorsed -classpath /u01/tomcat-sets/apache-tomcat-node2/bin/bootstrap.jar:/u01/tomcat-sets/apache-tomcat-node2/bin/tomcat-juli.jar -Dcatalina.base=/u01/tomcat-sets/apache-tomcat-node2 -Dcatalina.home=/u01/tomcat-sets/apache-tomcat-node2 -Djava.io.tmpdir=/u01/tomcat-sets/apache-tomcat-node2/temp org.apache.catalina.startup.Bootstrap start

关闭一个tomcat节点，再测。（以前遇到过两个节点shutdown端口一样，导致关闭异常。）

3.3 配置后端tomcat 主机及端口映射,编辑或新建文件，/etc/httpd/workers.properties。

与2.4步骤一样，但要求要做到支持多节点和负载均衡。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
worker.list=tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=127.0.0.1
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
 
worker.tomcat2.port=8019
worker.tomcat2.host=127.0.0.1
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1
 
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.loadbalancer.sticky_session=1

3.4 再次编辑节点服务器配置(server.xml)

node1节点server.xml最终信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
 
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
 
    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->
 
    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">   
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
 
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
 
      </Host>
    </Engine>
  </Service>
</Server>


node2节点配置信息(server.xml)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
<?xml version='1.0' encoding='utf-8'?>
 
<Server port="8006" shutdown="SHUTDOWN">
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
 
  <Service name="Catalina">
    <Connector port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8445" />
 
    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8019" protocol="AJP/1.3" redirectPort="8445" />
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">
      
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
 
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
 
      </Host>
    </Engine>
  </Service>
</Server>

值得一提的是：搭建过程中遇到一个问题。查看tomcat错误信息

java.net.BindException: Cannot assign requested address; No faulty members identified.


按照网上解决方案：主机配置错误。这才想起来，曾经切换过虚拟机的Network Adapter。

参照网址http://blog.csdn.net/ligang2585116/article/details/43796371。
解决方式，修改/etc/hosts中的IP，修改为现在的IP即可。

[root@hadoop1 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:20:2B:84  
          inet addr:192.168.163.146  Bcast:192.168.163.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe20:2b84/64 Scope:Link

[root@hadoop1 ~]# cat /etc/hosts  
192.168.163.146 hadoop1

3.5测试切换效果。

为了区分不同tomcat下的应用，将staticProtocal.html的内容作了下调整。

结果

memcached演练(3) 使用apache+搭建双节点tomcat集群

为了排除浏览器缓存干扰，最好使用强制刷新。（CTRL+F5）,这时候可以验证下权重。

3.6 测试下双节点集群的吞吐量。

C:\Apache24\bin>ab  -n 10000 -c 4  http://192.168.163.146/examples/staticProtocal.html
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Finished 10000 requests

Server Software:        Apache/2.2.15
Server Hostname:        192.168.163.146
Server Port:            80

Document Path:          /examples/staticProtocal.html
Document Length:        49598 bytes

Concurrency Level:      4
Time taken for tests:   13.996 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      498670000 bytes
HTML transferred:       495980000 bytes
Requests per second:    714.51 [#/sec] (mean)
Time per request:       5.598 [ms] (mean)
Time per request:       1.400 [ms] (mean, across all concurrent requests)
Transfer rate:          34795.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      1       5
Processing:     0    5   9.1      3     162
Waiting:        0    3   4.9      2     125
Total:          0    6   9.1      3     162
ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

吞吐量最差：714.51

为什么？

到了这里，还没有讲到session和memcached的影子。

结论（以1个50K的静态文本）

环境
 吞吐量
1.单台tomcat
 828.81
2.单个apache
 840.17
3.apache+单tomcat节点
 737.73
apache+双tomcat节点 714.51

通过对比，很直观得出结论。

  1. apache处理静态文件的能力，比tomcat的能力高一些.

  2. apache+单tomcat节点情况处理静态文件，纯属胡搞做实验，没有任何实际意义，很显然多了一层代理，当然吞吐量下来了。有点脱了裤子放屁的意思。

  3. 至于双节点tomcat的吞吐量下降的这么厉害，应该和我的环境有关系，我的CPU是单核，跑2个节点，想想发生了资源争用。

  4. 最后，这个实验不能掩盖tomcat的威力，它的强大在处理动态处理上，而这恰恰是apache做不了的。

    ------------------------

    The End,下一篇，重点验证下集群间的session的共享，和memcached在集群中的应用。



本文转自 randy_shandong 51CTO博客，原文链接：http://blog.51cto.com/dba10g/1837468，如需转载请自行联系原作者
上一篇：【百度地图API】除夕夜，大家一起来赶走“夕”——删除标注功能


下一篇：舒适美观的mac终端, iTerm2+zsh+powerlevel9k+vim+virtualenv