Docker+Jenkins持续集成

2021-07-21 15:53:09

Docker+Jenkins持续集成

使用etcd+confd实现容器服务注册与发现

 

前面我们已经通过jenkins+docker搭建了基本的持续集成环境，实现了服务的自动构建和部署，但是，我们遇到一个问题，jenkins构建出来的镜像部署后，需要通过ip:port去访问，有什么更好的方法吗？肯定是通过域名啊！前提是你注册一个域名，或者修改机器hosts文件。
本文介绍通过引入etcd+confd实现部署服务的自动注册，自动生成nginx配置文件，实现每个服务独立域名访问。

配置域名

假设你的域名是： example.com,那么我们可以规划

  • dev.$servicename.example.com作为开发环境，
  • test.$servicename.example.comz作为服务的测试环境。

配置步骤:

  • 首先将*.example.com 指向一台nginx服务器
  • 增加vhost配置文件，假设86.6,86.8,86.11 是docker swarm集群中的机器，服务的名称为allinoneservice，那么我们可以增加一个配置文件nginx_vhosts/service.conf：
    upstream test_service_allinoneservice {

        server 192.168.86.11:10091;

        server 192.168.86.6:10091;

        server 192.168.86.8:10091;

    }

    server {

        listen       80;

        server_name test.allinoneservice.example.com;

        location / {

            proxy_redirect off;

            proxy_set_header Host $host;

            proxy_read_timeout 300;

            proxy_set_header X-Real-IP $http_x_forwarded_for;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_pass http://test_service_allinoneservice;

        }

    }

  • 修改nginx配置文件nginx.conf,在最后一个大括号前，将刚新建的配置文件包含进去：

    include nginx_vhosts/*.conf;

  • 重启nginx，就可以通过test.allinoneservice.example.com访问服务了

通过服务注册自动生成配置文件

第一步里，我们需要手动编写配置文件，有更好的方式吗？答案是通过服务注册+confd，自动生成配置文件。

docker 安装etcd集群

首先，docker安装etcd作为注册中心，我们安装一个包含3个实例的集群，编写docker-compose.yml:

version: '3'

services:

  etcd0:

    image: 192.168.86.8:5000/etcd

    ports:

      - "2379:2379"

    volumes:

      - etcd0:/etcd_data

    command:

      - /usr/local/bin/etcd

      - -name

      - etcd0

      - --data-dir

      - /etcd_data

      - -advertise-client-urls

      - http://etcd0:2379

      - -listen-client-urls

      - http://0.0.0.0:2379

      - -initial-advertise-peer-urls

      - http://etcd0:2380

      - -listen-peer-urls

      - http://0.0.0.0:2380

      - -initial-cluster

      - etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380

  etcd1:

    image: 192.168.86.8:5000/etcd

    ports:

      - "2380:2379"

    volumes:

      - etcd1:/etcd_data

    command:

      - /usr/local/bin/etcd

      - -name

      - etcd1

      - --data-dir

      - /etcd_data

      - -advertise-client-urls

      - http://etcd1:2379

      - -listen-client-urls

      - http://0.0.0.0:2379

      - -initial-advertise-peer-urls

      - http://etcd1:2380

      - -listen-peer-urls

      - http://0.0.0.0:2380

      - -initial-cluster

      - etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380

  etcd2:

    image: 192.168.86.8:5000/etcd

    ports:

      - "2381:2379"

    volumes:

      - etcd2:/etcd_data

    command:

      - /usr/local/bin/etcd

      - -name

      - etcd2

      - --data-dir

      - /etcd_data

      - -advertise-client-urls

      - http://etcd2:2379

      - -listen-client-urls

      - http://0.0.0.0:2379

      - -initial-advertise-peer-urls

      - http://etcd2:2380

      - -listen-peer-urls

      - http://0.0.0.0:2380

      - -initial-cluster

      - etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380

volumes:

  etcd0:

  etcd1:

  etcd2:

注意，上面的image: 192.168.86.8:5000/etcd 是用的私有仓库，大家可以使用官方版本quay.io/coreos/etcd

然后启动etcd:

docker stack deploy -c docker-compose.yml etcd

服务注册

etcd注册就简单了，可以通过etcd的rest api，例如：

curl http://192.168.86.11:2379/v2/keys/services/test/allinoneservice/service1 -XPUT -d value="192.168.86.8:10091"

所以，我们修改一下jenkins里的docker部署脚本，服务部署后自动向etcd注册，由于是swarm集群，因此我们可以注册多个ip。

echo "start remove old service"

docker service rm  ${service_name}-${env}

echo "start create new service with latest builded image"

docker service create --name ${service_name}-${env} --replicas ${replicas} --publish ${service_port}:${docker_expose_port} 192.168.86.8:5000/${service_name}-${env}

echo "publish service to nginx"

curl http://192.168.86.11:2379/v2/keys/services/${env}/${service_name}/service1 -XPUT -d value="192.168.86.8:${service_port}"

curl http://192.168.86.11:2379/v2/keys/services/${env}/${service_name}/service2 -XPUT -d value="192.168.86.11:${service_port}"

curl http://192.168.86.11:2379/v2/keys/services/${env}/${service_name}/service3 -XPUT -d value="192.168.86.6:${service_port}"

注意，上面的service_name是jenkins参数化构建里定义的参数：

Docker+Jenkins持续集成

通过confd生成nginx配置文件

confd 是一个配置文件生成工具，可以从etcd、consul等注册中心读取数据根据模板生成配置文件，并在配置发生变化后自动更新配置文件，还能自动重启服务，是服务自动发现的居家必备良药。

首先讲下怎么安装，根据官方文档：

wget https://github.com/kelseyhightower/confd/releases/download/v0.14.0/confd-0.14.0-linux-amd64

mkdir -p /opt/confd/bin

mv confd-0.14.0-linux-amd64 /opt/confd/bin/confd

chmod +x /opt/confd/bin/confd

export PATH="$PATH:/opt/confd/bin"

为了方便使用，最好修改下/etc/profile,加入export PATH="$PATH:/opt/confd/bin"，然后source /etc/profile让配置生效。

然后编写confd配置文件/etc/confd/conf.d/myapp-nginx.toml：

[template]

src = "nginx.conf.tmpl"

dest = "/opt/third_party/nginx_vhosts/service.conf"

keys = [

    "/services/dev",

    "/services/test",

]

reload_cmd = "/opt/third_party/sbin/nginx -s reload"

上面配置了，会读取的keys，以及dest目标配置文件地址，和配置文件更新后的reload_cmd，用于重启nginx

接着编写模板文件

{{range $dir := lsdir "/services/test"}}

upstream test_service_{{base $dir}} {

    {{$custdir := printf "/services/test/%s/*" $dir}}{{range getvs $custdir}}

    server {{.}};

    {{end}}

}

server {

    listen       80;

    server_name test.{{base $dir}}.iflyresearch.com;

    location / {

        proxy_redirect off;

        proxy_set_header Host $host;

        proxy_read_timeout 300;

        proxy_set_header X-Real-IP $http_x_forwarded_for;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://test_service_{{base $dir}};

    }

}

{{end}}

上面的模板比较简单，通过lsdir指令读取服务列表，然后通过range getvs获取服务对应的负载地址。

然后启动confd，需要指定etcd的地址：

nohup confd -backend etcd -node http://192.168.86.11:2379 &

搞定！

集成PMD、FindBugs、Checkstyle静态代码检查工具并邮件发送检查结果

 

为了规范代码，我们一般会集成静态代码检测工具，比如PMD、FindBugs、Checkstyle，那么Jenkins如何集成这些检查工具，并把检查结果放到构建邮件里呢？

今天做了调研和实现，过程如下

首先看，最终效果：

Docker+Jenkins持续集成

1.pom.xml

build。plugins 增加：

<plugin>

                <artifactId>maven-assembly-plugin</artifactId>

                <executions>

                    <execution>

                        <phase>package</phase>

                        <goals>

                            <goal>single</goal>

                        </goals>

                        <configuration>

                            <finalName>${project.artifactId}-${project.version}</finalName>

                            <appendAssemblyId>false</appendAssemblyId>

                            <descriptors>

                                <descriptor>src/assembly/assembly-descriptor.xml</descriptor>

                            </descriptors>

                        </configuration>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.codehaus.mojo</groupId>

                <artifactId>findbugs-maven-plugin</artifactId>

                <version>3.0.5</version>

                <configuration>

                    <threshold>High</threshold>

                    <effort>Default</effort>

                    <findbugsXmlOutput>true</findbugsXmlOutput>

                    <findbugsXmlWithMessages>true</findbugsXmlWithMessages>

                    <xmlOutput>true</xmlOutput>

                    <formats><format>html</format></formats>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-pmd-plugin</artifactId>

                <version>3.8</version>

            </plugin>

reporting 增加：

    <reporting>

        <plugins>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-checkstyle-plugin</artifactId>

                <version>3.0.0</version>

                <reportSets>

                    <reportSet>

                        <reports>

                            <report>checkstyle</report>

                        </reports>

                    </reportSet>

                </reportSets>

            </plugin>

            <!--pmd-->

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-pmd-plugin</artifactId>

                <configuration>

                    <linkXref>true</linkXref>

                    <sourceEncoding>utf-8</sourceEncoding>

                    <minimumTokens>100</minimumTokens>

                    <targetJdk>1.8</targetJdk>

                    <excludes>

                        <exclude>**/*Bean.java</exclude>

                        <exclude>**/generated/*.java</exclude>

                    </excludes>

                    <excludeRoots>

                        <excludeRoot>target/generated-sources/stubs</excludeRoot>

                    </excludeRoots>

                </configuration>

                <version>3.8</version>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-jxr-plugin</artifactId>

                <version>2.5</version>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-surefire-report-plugin</artifactId>

                <version>2.14.1</version>

            </plugin>

            <plugin>

                <groupId>org.codehaus.mojo</groupId>

                <artifactId>cobertura-maven-plugin</artifactId>

            </plugin>

        </plugins>

    </reporting>

2 jenkins设置

2.1 安装相关插件

直接搜索安装：

  • Checkstyle Plug-in
  • PMD Plug-in
  • FindBugs Plug-in
  • Static Analysis Collector Plug-in

邮件插件安装：

  • Email Extension Plugin
  • Email Extension Template Plugin

2.2 项目配置

maven构建Goals设置为：
pmd:pmd checkstyle:checkstyle findbugs:findbugs package -DskipTests

在构建设置里，勾上

  • Publish Checkstyle analysis results
  • Publish FindBugs analysis results
  • Publish PMD analysis results

构建后操作，添加Publish combined static analysis results，默认设置即可。

2.3 邮件配置

首先在系统设置里，配置Extended E-mail Notification部分

设置：

  • Default Subject ： 自动构建通知：$PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS!
  • Default Content： ${JELLY_SCRIPT, template="analysis.jelly"}

剩下的自己配置下SMTP和收件人。

这里邮件内容使用analysis.jelly，使用jelly script，系统没有这个模板，我们需要配置一下：

打开系统管理-Managed files，增加一个Extended Email Publisher Jelly Template

模板文件如下，该模板修改自官方的模板，做了一定的本地化和样式调整：

<?jelly escape-by-default='true'?>

<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define">

  <html>

    <head>

      <title>${project.name}</title>

      <style>

        body table, td, th, p, h1, h2 {

        margin:0;

        font:normal normal

        100% Georgia, Serif;

        }

        h1, h2 {

        border-bottom:dotted 1px #999999;

        padding:5px;

        margin-bottom:10px;

        color: #000000;

        font: normal bold 130%

        Georgia,Serif;

        background-color:#f0f0f0;

        }

        tr.gray {

        background-color:#f0f0f0;

        }

        h2 {

        padding:5px;

        margin-top:5px;

        margin-bottom:5px;

        font: italic bold 110% Georgia,Serif;

        }

        .bg2 {

        color:black;

        background-color:#E0E0E0;

        font-size:110%

        }

        th {

        font-weight: bold;

        }

        tr, td, th {

        padding:2px;

        }

        td.test_passed {

        color:blue;

        }

        td.test_failed {

        color:red;

        }

        td.center {

          text-align: center;

        }

        td.test_skipped {

        color:grey;

        }

        .console {

        font: normal normal 90% Courier New,

        monotype;

        padding:0px;

        margin:0px;

        }

        div.content, div.header {

        background: #ffffff;

        border: dotted

        1px #666;

        margin: 2px;

        content:

        2px;

        padding: 2px;

        }

        table.border, th.border, td.border {

        border:

        1px solid black;

        border-collapse:collapse;

        }

        .round_border{margin-bottom:5px;-webkit-border-radius:6px;-moz-border-radius:6px;border-radius:6px;margin-top:0;font-size:14px;padding:6px;border:1px solid #ccc}

.status{background-color:<j:choose><j:when test="${build.result=='SUCCESS'}">green</j:when><j:otherwise>red</j:otherwise></j:choose>;font-size:28px;font-weight:bold;color:white;height:52px;margin-bottom:18px;text-align:center;vertical-align:middle;border-collapse:collapse;background-repeat:no-repeat}

.status .info{color:white!important;text-shadow:0 -1px 0 rgba(0,0,0,0.3);font-size:32px;line-height:36px;padding:8px 0}

      </style>

    </head>

    <body>

      <div class="status">

          <p class="info">${project.name}构建<j:choose><j:when test="${build.result=='SUCCESS'}">成功</j:when><j:otherwise>失败</j:otherwise></j:choose></p>

      </div>

      <div class="header round_border">

        <j:set var="spc" value="  " />

        <h1>基本信息</h1>

        <table>

          <tr>

            <td>构建地址</td>

            <td>

              <a href="${rooturl}${build.url}">${rooturl}${build.url}</a>

            </td>

          </tr>

          <tr>

            <td>项目:</td>

            <td>${project.name}</td>

          </tr>

          <tr>

            <td>构建时间:</td>

            <td>${it.timestampString}</td>

          </tr>

          <tr>

            <td>构建耗时:</td>

            <td>${build.durationString}</td>

          </tr>

          <tr>

            <td>构建原因:</td>

            <td>

              <j:forEach var="cause" items="${build.causes}">${cause.shortDescription}

              </j:forEach>

            </td>

          </tr>

          <tr>

            <td>构建描述:</td>

            <td>${build.description}</td>

          </tr>

          <tr>

            <td>构建服务器:</td>

            <td>

              <j:choose>

                <j:when test="${build.builtOnStr!=''}">${build.builtOnStr}</j:when>

                <j:otherwise>master</j:otherwise>

              </j:choose>

            </td>

          </tr>

        </table>

      </div>

      <!-- HEALTH TEMPLATE -->

      <div class="content round_border">

        <j:set var="healthIconSize" value="16x16" />

        <j:set var="healthReports" value="${project.buildHealthReports}" />

        <j:if test="${healthReports!=null}">

          <h1>构建健康报告</h1>

          <table>

            <tr>

              <th>W</th>

              <th>Description</th>

              <th>Score</th>

            </tr>

            <j:forEach var="healthReport" items="${healthReports}">

              <tr>

                <td>

                  <img

                    src="${rooturl}${healthReport.getIconUrl(healthIconSize)}" />

                </td>

                <td>${healthReport.description}</td>

                <td>${healthReport.score}</td>

              </tr>

            </j:forEach>

          </table>

          <br />

        </j:if>

      </div>

      <!-- CHANGE SET -->

      <div class="content round_border">

        <j:set var="changeSet" value="${build.changeSet}" />

        <j:if test="${changeSet!=null}">

          <j:set var="hadChanges" value="false" />

          <a href="${rooturl}${build.url}/changes">

            <h1>代码变更记录</h1>

          </a>

          <j:forEach var="cs" items="${changeSet.logs}"

            varStatus="loop">

            <j:set var="hadChanges" value="true" />

            <h2>${cs.msgAnnotated}</h2>

            <p>

              by <em>${cs.author}</em>

            </p>

            <table>

              <j:forEach var="p" items="${cs.affectedFiles}">

                <tr>

                  <td width="10%">${spc}${p.editType.name}</td>

                  <td>

                    <tt>${p.path}</tt>

                  </td>

                </tr>

              </j:forEach>

            </table>

          </j:forEach>

          <j:if test="${!hadChanges}">

            <p>无变更</p>

          </j:if>

          <br />

        </j:if>

      </div>

      <!-- ARTIFACTS -->

      <j:set var="artifacts" value="${build.artifacts}" />

      <j:if test="${artifacts!=null and artifacts.size()&gt;0}">

        <div class="content round_border">

          <h1>构建产物</h1>

          <ul>

            <j:forEach var="f" items="${artifacts}">

              <li>

                <a href="${rooturl}${build.url}artifact/${f}">${f}</a>

              </li>

            </j:forEach>

          </ul>

        </div>

      </j:if>

      <!-- MAVEN ARTIFACTS -->

      <j:set var="mbuilds" value="${build.moduleBuilds}" />

      <j:if test="${mbuilds!=null}">

        <div class="content round_border">

          <h1>MAVEN 构建产物</h1>

          <j:forEach var="m" items="${mbuilds}">

            <h2>${m.key.displayName}</h2>

            <j:forEach var="mvnbld" items="${m.value}">

              <j:set var="artifacts" value="${mvnbld.artifacts}" />

              <j:if test="${artifacts!=null and artifacts.size()&gt;0}">

                <ul>

                  <j:forEach var="f" items="${artifacts}">

                    <li>

                      <a href="${rooturl}${mvnbld.url}artifact/${f}">${f}</a>

                    </li>

                  </j:forEach>

                </ul>

              </j:if>

            </j:forEach>

          </j:forEach>

          <br />

        </div>

      </j:if>

      <!-- JUnit TEMPLATE -->

      <j:set var="junitResultList" value="${it.JUnitTestResult}" />

      <j:if test="${junitResultList.isEmpty()!=true}">

        <div class="content round_border">

          <a href="${rooturl}${build.url}/testReport">

            <h1>JUnit Tests</h1>

          </a>

          <table class="border">

            <tr>

              <th class="border">Package</th>

              <th class="border">Failed</th>

              <th class="border">Passed</th>

              <th class="border">Skipped</th>

              <th class="border">Total</th>

            </tr>

            <j:forEach var="junitResult" items="${it.JUnitTestResult}">

              <j:forEach var="packageResult" items="${junitResult.getChildren()}">

                <tr>

                  <td class="border">

                    <tt>${packageResult.getName()}</tt>

                  </td>

                  <td class="border test_failed">${packageResult.getFailCount()}</td>

                  <td class="border test_passed">${packageResult.getPassCount()}</td>

                  <td class="border test_skipped">${packageResult.getSkipCount()}</td>

                  <td class="border">

                    <b>${packageResult.getPassCount()+packageResult.getFailCount()+packageResult.getSkipCount()}

                    </b>

                  </td>

                </tr>

                <j:forEach var="failed_test"

                  items="${packageResult.getFailedTests()}">

                  <tr>

                    <td class="test_failed" colspan="5">

                      <tt>${failed_test.getFullName()}</tt>

                    </td>

                  </tr>

                </j:forEach>

              </j:forEach>

            </j:forEach>

          </table>

          <br />

        </div>

      </j:if>

      <!-- Static Analysis -->

      <j:set var="actions" value="${it.staticAnalysisActions}" />

      <j:if test="${!actions.isEmpty()}">

        <div class="content round_border">

          <h1>代码静态检查结果</h1>

          <table>

            <tr>

              <th>名称</th>

              <th>检查结果</th>

              <th>总数</th>

              <th>High</th>

              <th>Normal</th>

              <th>Low</th>

            </tr>

            <j:forEach var="action" items="${actions}">

              <tr>

                <td>

                  <a href="${rooturl}${build.url}/${action.urlName}">${action.displayName}</a>

                </td>

                <td class="center">

                  <j:choose>

                    <j:when test="${action.result.pluginResult=='SUCCESS'}">

                      <img src="${rooturl}static/e59dfe28/images/16x16/blue.gif" />

                    </j:when>

                    <j:when test="${action.result.pluginResult=='FAILURE'}">

                      <img src="${rooturl}static/e59dfe28/images/16x16/red.gif" />

                    </j:when>

                    <j:otherwise>

                      <img src="${rooturl}static/e59dfe28/images/16x16/yellow.gif" />

                    </j:otherwise>

                  </j:choose>

                </td>

                <td class="center">${action.result.numberOfAnnotations} </td>

                <td class="center">${action.result.getNumberOfAnnotations('HIGH')} </td>

                <td class="center">${action.result.getNumberOfAnnotations('NORMAL')} </td>

                <td class="center">${action.result.getNumberOfAnnotations('LOW')} </td>

              </tr>

            </j:forEach>

          </table>

        </div>

      </j:if>

      <div class="content round_border">

        <!-- CONSOLE OUTPUT -->

        <a href="${rooturl}${build.url}/console">

          <h1>Console 输出结果（后50行）</h1>

        </a>

        <table class="console">

          <j:forEach var="line" items="${build.getLog(50)}">

            <tr>

              <td>

                <tt>${line}</tt>

              </td>

            </tr>

          </j:forEach>

        </table>

        <br />

      </div>

    </body>

  </html>

</j:jelly>

保存。

然后配置系统管理——Editable Email Notification Templates，增加一个默认模板，名称为默认，直接保存。

回到项目配置， 在构建环境里，
勾上Provide Configuration files
选择File为刚新增的jelly文件，target填写/var/jenkins_home/email-templates
Variable填写analysis.jelly

然后，在构建后操作里增加Editable Email Notification Templates，选择默认模板。

这样就完成了配置。

延伸阅读

Jenkins+Docker 搭建持续集成环境：

作者：Jadepeng
出处：jqpeng的技术记事本--http://www.cnblogs.com/xiaoqi

上一篇：Jekens 配置多项目SCM GitLab+Jenkins持续集成环境


下一篇：Spring Security 之方法级的安全管控