puppet 横向扩展(一)

2021-09-06 13:27:00

概述

横向扩展实验之一 – 扩展puppet master 的个数.

实验环境

master 和 node 都是 debian 7.7 i686 系统

2个 puppet master 在一台机器上, 都是 apache 虚拟主机

实验步骤

创建puppetmaster的rack环境

cd /usr/share/puppet/rack

mkdir -p puppetmasterd_18140/{public,tmp}

cp puppetmasterd/config.ru puppetmasterd_18140/

chown puppet puppetmasterd_18140/config.ru

mkdir -p puppetmasterd_18141/{public,tmp}

cp puppetmasterd/config.ru puppetmasterd_18141/

chown puppet puppetmasterd_18141/config.ru

配置文件设置

  • passenger.conf : passenger 配置信息

    放在 /etc/apache2/mods-available 中， 并在 /etc/apache2/mods-enabled中建立软连接
  • puppetmaster_proxy.conf

    关闭 SSL， 重新请求头部， 为后端进程做负载均衡，放在 /etc/apache2/site-available 中， 并在 /etc/apache2/site-enabled中建立软连接
  • puppetmaster_worker_1.conf

    虚拟主机1， 指向处理puppet请求的Rac目录，放在 /etc/apache2/site-available 中， 并在 /etc/apache2/site-enabled中建立软连接
  • puppetmaster_worker_2.conf

    虚拟主机2， 指向处理puppet请求的Rac目录，放在 /etc/apache2/site-available 中， 并在 /etc/apache2/site-enabled中建立软连接

各个配置文件的详细内容如下:

$ cat passenger.conf

<IfModule mod_passenger.c>

  PassengerRoot /usr

  PassengerRuby /usr/bin/ruby

  # And the passenger performance tuning settings

  PassengerHighPerformance On

  # Set this to about 1.5 times the number of CPU cores in your master:

  PassengerMaxPoolSize 2

  # Recycle master processes after they service 1000 requests

  PassengerMaxRequests 1000

  # Stop processes if they sit idle for 10 minutes

  PassengerPoolIdleTime 600

</IfModule>

$ cat puppetmaster_proxy.conf

# Available back-end worker virtual hosts

# NOTE the use of cleartext unencrypted HTTP.

<Proxy balancer://puppetmaster>

  BalancerMember http://127.0.0.1:18140

  BalancerMember http://127.0.0.1:18141

</Proxy>

Listen 8140

<VirtualHost *:8140>

  SSLEngine on

  # SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA

  SSLProtocol ALL +SSLv3 +TLSv1

  SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

  #SSLProtocol ALL -SSLv2

  #SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP

  # Puppet master should generate initial CA certificate.

  # ensure certs are located in /var/lib/puppet/ssl

  SSLCertificateFile /var/lib/puppet/ssl/certs/master-1.puppet.com.pem

  SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/master-1.puppet.com.pem

  SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem

  SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem

  SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem

  # optional to all CSR request, required if certificates distributed to client during provisioning.

  SSLVerifyClient optional

  SSLVerifyDepth 1

  SSLOptions +StdEnvVars

  # The following client headers record authentication information for downstream workers.

  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

  <Location />

    SetHandler balancer-manager

    Order allow,deny

    Allow from all

  </Location>

  ProxyPass / balancer://puppetmaster/

  ProxyPassReverse / balancer://puppetmaster/

  ProxyPreserveHost On

  # log settings

  ErrorLog /var/log/apache2/balancer_error.log

  CustomLog /var/log/apache2/balancer_access.log combined

  CustomLog /var/log/apache2/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

$ cat puppetmaster_worker_1.conf

Listen 18140

<VirtualHost 127.0.0.1:18140>

  SSLEngine off

  # Obtain Authentication Information from Client Request Headers

  SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1

  SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

  PassengerEnabled On

  DocumentRoot /usr/share/puppet/rack/puppetmasterd_18140/public

  <Directory /usr/share/puppet/rack/puppetmasterd_18140>

    Options None

    AllowOverride None

    Order allow,deny

    allow from all

  </Directory>

  # log settings

  ErrorLog /var/log/apache2/puppetmaster_worker_error_1.log

  CustomLog /var/log/apache2/puppetmaster_worker_access_1.log combined

</VirtualHost>

$ cat puppetmaster_worker_2.conf

Listen 18141

<VirtualHost 127.0.0.1:18141>

  SSLEngine off

  # Obtain Authentication Information from Client Request Headers

  SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1

  SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

  PassengerEnabled On

  DocumentRoot /usr/share/puppet/rack/puppetmasterd_18141/public

  <Directory /usr/share/puppet/rack/puppetmasterd_18141>

    Options None

    AllowOverride None

    Order allow,deny

    allow from all

  </Directory>

  # log settings

  ErrorLog /var/log/apache2/puppetmaster_worker_error_2.log

  CustomLog /var/log/apache2/puppetmaster_worker_access_2.log combined

</VirtualHost>

补充说明

apache默认没有加载 proxy 和 proxy_balancer 和 proxy_http 模块，需要补上

cd /etc/apache2/mods-enabled

ln -s ../mods-available/proxy.conf proxy.conf

ln -s ../mods-available/proxy.load proxy.load

ln -s ../mods-available/proxy_balancer.conf proxy_balancer.conf

ln -s ../mods-available/proxy_balancer.load proxy_balancer.load

ln -s ../mods-available/proxy_http.load proxy_http.load     # 没有这个模块， agent 会有 503错误

cd /etc/apache2/sites-enabled

ln -s ../sites-available/puppetmaster_proxy.conf puppetmaster_proxy

ln -s ../sites-available/puppetmaster_worker_1.conf puppetmaster_worker_1

ln -s ../sites-available/puppetmaster_worker_2.conf puppetmaster_worker_2

默认的 8140 端口关闭

rm /etc/apache2/sites-enabled/puppetmaster

service apache2 restart

配置完成后如果有类似如下 403 权限不足的错误

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /certificate_revocation_list/ca [find] at :119

那么, 将 master-1 上的 /etc/puppet/puppet.conf 文件中 [master] 下 如下2行注释掉。（估计SSL由代理服务器来完成）

#ssl_client_header = SSL_CLIENT_S_DN

#ssl_client_verify_header = SSL_CLIENT_VERIFY

测试配置结果

默认的负载均衡

# puppet master 上执行

root@master-1:/var/log/apache2# service apache2 restart

# 开始时, 负载均衡的log都是空的

root@master-1:/var/log/apache2# ll /var/log/apache2/

total 4

-rw-r--r-- 1 root root   0 Jan  7 09:58 access.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 balancer_access.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 balancer_error.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 balancer_ssl_requests.log

-rw-r--r-- 1 root root 597 Jan  7 16:43 error.log

-rw-r--r-- 1 root root   0 Jan  7 09:58 other_vhosts_access.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_access_1.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_access_2.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_error_1.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_error_2.log

# agent 上执行

puppet agent -t

# master 上查看log

root@master-1:/var/log/apache2# ll /var/log/apache2/

total 20

-rw-r--r-- 1 root root   0 Jan  7 09:58 access.log

-rw-r--r-- 1 root root 821 Jan  7 16:52 balancer_access.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 balancer_error.log

-rw-r--r-- 1 root root 903 Jan  7 16:52 balancer_ssl_requests.log

-rw-r--r-- 1 root root 597 Jan  7 16:43 error.log

-rw-r--r-- 1 root root   0 Jan  7 09:58 other_vhosts_access.log

-rw-r--r-- 1 root root 489 Jan  7 16:52 puppetmaster_worker_access_1.log

-rw-r--r-- 1 root root 311 Jan  7 16:52 puppetmaster_worker_access_2.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_error_1.log

-rw-r--r-- 1 root root   0 Jan  7 16:43 puppetmaster_worker_error_2.log

root@master-1:/var/log/apache2# cat balancer_access.log

192.168.1.120 - - [07/Jan/2015:16:52:37 +0800] "GET /production/node/node-1.puppet.com?transaction_uuid=7998c4d3-ba8e-4ffd-8f7f-8d37f3de42ec&fail_on_404=true HTTP/1.1" 200 10464 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:16:52:39 +0800] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 569 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:16:52:40 +0800] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 569 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:16:52:40 +0800] "POST /production/catalog/node-1.puppet.com HTTP/1.1" 200 869 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:16:52:41 +0800] "PUT /production/report/node-1.puppet.com HTTP/1.1" 200 298 "-" "Ruby"

root@master-1:/var/log/apache2# cat puppetmaster_worker_access_1.log

127.0.0.1 - - [07/Jan/2015:16:52:37 +0800] "GET /production/node/node-1.puppet.com?transaction_uuid=7998c4d3-ba8e-4ffd-8f7f-8d37f3de42ec&fail_on_404=true HTTP/1.1" 200 5120 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:16:52:40 +0800] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 566 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:16:52:41 +0800] "PUT /production/report/node-1.puppet.com HTTP/1.1" 200 295 "-" "Ruby"

root@master-1:/var/log/apache2# cat puppetmaster_worker_access_2.log

127.0.0.1 - - [07/Jan/2015:16:52:39 +0800] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 567 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:16:52:40 +0800] "POST /production/catalog/node-1.puppet.com HTTP/1.1" 200 866 "-" "Ruby"

从上面的log可以看出 agent 上执行的 puppet agent -t 一共访问了 5 次 master.

其中3次由 worker_1 处理了, 2次由 worker_2 处理了. <== 这就是负载均衡的效果

负载均衡情况下, 一台 puppet master 挂了的情况

# master 上执行, 清空log, 删除 puppetmaster_worker_1 的配置, 重启apache2 服务

root@master-1:~# rm /var/log/apache2/* -rf

root@master-1:~# rm /etc/apache2/sites-enabled/puppetmaster_worker_1

rm: remove symbolic link `/etc/apache2/sites-enabled/puppetmaster_worker_1'? y

root@master-1:~# service apache2 restart

[ ok ] Restarting web server: apache2 ... waiting .

root@master-1:~# ll /var/log/apache2/

total 4

-rw-r--r-- 1 root root   0 Jan  7 17:53 access.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 balancer_access.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 balancer_error.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 balancer_ssl_requests.log

-rw-r--r-- 1 root root 155 Jan  7 17:53 error.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 other_vhosts_access.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 puppetmaster_worker_access_2.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 puppetmaster_worker_error_2.log

# agent 上执行

root@node-1:~# puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for node-1.puppet.com

Info: Applying configuration version '1420626127'

Notice: Finished catalog run in 0.03 seconds

# master 上查看执行结果

root@master-1:~# ll /var/log/apache2/

total 20

-rw-r--r-- 1 root root   0 Jan  7 17:53 access.log

-rw-r--r-- 1 root root 821 Jan  7 18:22 balancer_access.log

-rw-r--r-- 1 root root 223 Jan  7 18:22 balancer_error.log

-rw-r--r-- 1 root root 903 Jan  7 18:22 balancer_ssl_requests.log

-rw-r--r-- 1 root root 155 Jan  7 17:53 error.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 other_vhosts_access.log

-rw-r--r-- 1 root root 800 Jan  7 18:22 puppetmaster_worker_access_2.log

-rw-r--r-- 1 root root   0 Jan  7 17:53 puppetmaster_worker_error_2.log

root@master-1:~# cat /var/log/apache2/balancer_access.log

192.168.1.120 - - [07/Jan/2015:18:22:05 +0800] "GET /production/node/node-1.puppet.com?transaction_uuid=29b0a3a5-8749-4647-92a4-a6da66c25c64&fail_on_404=true HTTP/1.1" 200 10466 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:18:22:06 +0800] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 569 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:18:22:06 +0800] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 569 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:18:22:07 +0800] "POST /production/catalog/node-1.puppet.com HTTP/1.1" 200 869 "-" "Ruby"

192.168.1.120 - - [07/Jan/2015:18:22:07 +0800] "PUT /production/report/node-1.puppet.com HTTP/1.1" 200 298 "-" "Ruby"

root@master-1:~# cat /var/log/apache2/puppetmaster_worker_access_2.log

127.0.0.1 - - [07/Jan/2015:18:22:05 +0800] "GET /production/node/node-1.puppet.com?transaction_uuid=29b0a3a5-8749-4647-92a4-a6da66c25c64&fail_on_404=true HTTP/1.1" 200 5122 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:18:22:06 +0800] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 566 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:18:22:06 +0800] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 566 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:18:22:07 +0800] "POST /production/catalog/node-1.puppet.com HTTP/1.1" 200 866 "-" "Ruby"

127.0.0.1 - - [07/Jan/2015:18:22:07 +0800] "PUT /production/report/node-1.puppet.com HTTP/1.1" 200 295 "-" "Ruby"

从上面的log可以看出, 没有生成 worker_1 的log, 全部处理都是由 worker_2 完成的. agent 也没有出错.

上一篇：Maven中-DskipTests和-Dmaven.test.skip=true的区别


下一篇：c#转义字符串中的所有正则特殊字符