【新功能发布】事件监控升级-支持自动化处理云产品异常

2021-07-02 09:25:31

背景

系统事件监控为用户提供各类云产品产生的系统事件的统一统计和查询入口，使得用户明确知晓云产品的使用状态，让云更透明。

本月的新版本，支持在这些产品的系统事件发生时，分发给您的消息服务队列、函数计算，以便后续自动化处理这些异常。

原理解读

事件发生时，云产品会将事件推送给云监控，云监控收到事件后，检查您是否配置了报警规则，如果有相关配置，会根据报警规则的配置将事件分发到相应渠道。

最佳实践分享

如何在ECS宕机时，通过函数计算将EIP迁移到另一台机器上。并且发送事件到消息服务的队列中。

创建函数计算

登录函数计算控制台：https://fc.console.aliyun.com/overview/cn-shanghai
新建服务
新建函数

选择空白函数，不创建触发器，以Java8运行环境为例

上传一段EIP从机器A解绑后，绑定到机器B的代码逻辑，jar包中的代码见文章结尾分享。

函数入口为com.aliyun.cms.fc.driver.FCDriver::handleRequest

无需其他权限设置，点击创建后保存函数

创建消息服务的队列

登录消息服务控制台：https://mns.console.aliyun.com/#/list/cn-hangzhou
创建队列

创建云监控系统事件报警

登录云监控事件监控控制台：https://cloudmonitor.console.aliyun.com/#/eventmonitoring/alarmrules
创建事件报警规则

点击按钮进入创建报警规则页面

首次使用先点击授权，允许云监控向您的队列和函数发送事件。

授权后，选择上一步创建好的队列和函数，保存设置。

调试事件

您可以使用系统事件的调试功能，模拟系统事件的发生，以便验证报警规则中设置的消息服务队列是否能正常接收时间、函数计算的函数是否能正常被触发。

点击报警规则列表的调试操作，进入调试页面
选择需要调试的事件，内容中会显示相应的事件内容，可以根据实际情况修改内容中的实例id等字段。
点击确定按钮，将根据内容发送一个事件，触发函数计算的逻辑、向消息服务队列中写一个事件。

查看结果

查看消息服务的队列信息
在消息服务队列的列表页点击接收消息，可以看到队列里收到了调试发出的事件。

在弹性公网IP控制台查看IP已经从一台实例解绑后，重新绑定到另一台实例。
在函数计算的代码执行模块，可以查询执行结果。

示例代码

maven依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.aliyun.cms</groupId>
    <artifactId>drive-fc</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>com.aliyun.fc.runtime</groupId>
            <artifactId>fc-java-core</artifactId>
            <version>1.0.0</version>
        </dependency>
        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>aliyun-java-sdk-ecs</artifactId>
            <version>4.9.4</version>
        </dependency>
        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>aliyun-java-sdk-slb</artifactId>
            <version>3.2.6</version>
        </dependency>
        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>aliyun-java-sdk-core</artifactId>
            <version>4.0.3</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.48</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <appendAssemblyId>false</appendAssemblyId> <!-- this is used for not append id to the jar name -->
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase> <!-- bind to the packaging phase -->
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Java 代码

示例包含EIP的解绑、绑定，SLB后端ESC实例的绑定、解绑，磁盘的挂载。试用时请将实例id、AK、SK等信息替换成您的具体信息，示例仅供演示参考，具体事件发生时应做的处理，需要根据您的实际业务决定。

package com.aliyun.cms.fc.driver;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.aliyun.fc.runtime.Context;
import com.aliyun.fc.runtime.FunctionComputeLogger;
import com.aliyun.fc.runtime.StreamRequestHandler;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.ecs.model.v20140526.*;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.profile.DefaultProfile;
import com.aliyuncs.profile.IClientProfile;
import com.aliyuncs.slb.model.v20140515.AddBackendServersRequest;
import com.aliyuncs.slb.model.v20140515.AddBackendServersResponse;
import com.aliyuncs.slb.model.v20140515.RemoveBackendServersRequest;
import com.aliyuncs.slb.model.v20140515.RemoveBackendServersResponse;
import org.apache.commons.io.IOUtils;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.TimeUnit;
public class FCDriver implements StreamRequestHandler 
   private static final String ak = "*";
   private static final String aks = "*";

   /**
    *
    * @param inputStream
    * @param outputStream
    * @param context
    * @throws IOException
    */
   @Override
   public void handleRequest(
           InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
       String input = IOUtils.toString(inputStream, StandardCharsets.UTF_8.name());
       JSONObject event = JSON.parseObject(input, JSONObject.class);
       context.getLogger().info("receive event message: " + event);
       String product = event.getString("product");
       if ("ECS1".equalsIgnoreCase(product)) {
           handleECS(context.getLogger(), event);
       }
       outputStream.write("ok".getBytes());
   }

   public void handleECS(FunctionComputeLogger logger, JSONObject event) {
       JSONObject content = event.getJSONObject("content");
       String ecsInstanceId = content.getString("ecsInstanceId");
       String regionId = event.getString("regionId");
       String name = event.getString("name");
       String eipId = "eip-*";
       if ("Disk:Stalled:Executing".equalsIgnoreCase(name)){
           unassociateEipAddress(regionId, ecsInstanceId, eipId, logger);
           try {
               TimeUnit.SECONDS.sleep(30);
           } catch (InterruptedException e) {
           }
           associateEipAddress(regionId, "i-****", eipId, logger);
       }else if ("Disk:Stalled:Executed".equalsIgnoreCase(name)){
           unassociateEipAddress(regionId, "i-****", eipId, logger);
           try {
               TimeUnit.SECONDS.sleep(30);
           } catch (InterruptedException e) {
           }
           associateEipAddress(regionId, ecsInstanceId, eipId, logger);
       }else if ("Instance:SystemFailure.Reboot:Executing".equalsIgnoreCase(name)){
           unassociateEipAddress(regionId, ecsInstanceId, eipId, logger);
           try {
               TimeUnit.SECONDS.sleep(30);
           } catch (InterruptedException e) {
           }
           associateEipAddress(regionId, "i-****", eipId, logger);
       }else if ("Instance:SystemFailure.Reboot:Executed".equalsIgnoreCase(name)){
           unassociateEipAddress(regionId, "i-****", eipId, logger);
           try {
               TimeUnit.SECONDS.sleep(30);
           } catch (InterruptedException e) {
           }
           associateEipAddress(regionId, ecsInstanceId, eipId, logger);
       }
   }

   private void associateEipAddress(String regionId, String ecsInstanceId, String eipId, FunctionComputeLogger logger ) {
       DefaultProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);

       try {
           for (int count=0;count<5;count++) {
               DescribeEipAddressesRequest describeEipAddressesRequest = new DescribeEipAddressesRequest();
               describeEipAddressesRequest.setAllocationId(eipId);
               describeEipAddressesRequest.setRegionId(regionId);
               DescribeEipAddressesResponse describeEipAddressesResponse = client.getAcsResponse(describeEipAddressesRequest);
               if (describeEipAddressesResponse!=null&&describeEipAddressesResponse.getEipAddresses()!=null&&"vailable".equalsIgnoreCase(describeEipAddressesResponse.getEipAddresses().get(0).getStatus())){
                   AssociateEipAddressRequest request = new AssociateEipAddressRequest();
                   request.setInstanceId(ecsInstanceId);
                   request.setAllocationId(eipId);
                   try {
                       AssociateEipAddressResponse response = client.getAcsResponse(request);
                       logger.info(String.format("Associate eip %s to ecs %s, requestId:%s, ", eipId, ecsInstanceId, response.getRequestId()));
                   } catch (ClientException e) {
                       logger.info(String.format("Failure of associate eip %s to ecs %s, errorMsg:%s", eipId, ecsInstanceId, e.getLocalizedMessage()));
                   }
                   return;
               }
               TimeUnit.SECONDS.sleep(5);
           }
       } catch (InterruptedException | ClientException e) {
           logger.info(String.format("Failure of describe eip %s, errorMsg:%s", eipId, e.getLocalizedMessage()));
       }
   }

   private void unassociateEipAddress(String regionId, String ecsInstanceId, String eipId, FunctionComputeLogger logger) {
       DefaultProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);

       UnassociateEipAddressRequest request = new UnassociateEipAddressRequest();
       request.setInstanceId(ecsInstanceId);
       request.setAllocationId(eipId);
       try {
           UnassociateEipAddressResponse response = client.getAcsResponse(request);
           logger.info(String.format("Unassociate eip %s to ecs %s, requestId:%s, ", eipId, ecsInstanceId, response.getRequestId()));
       } catch (ClientException e) {
           logger.info(String.format("Failure of unassociate eip %s to ecs %s, errorMsg:%s", eipId, ecsInstanceId, e.getLocalizedMessage()));
       }
   }

   private void addBackendServer(String regionId, String ecsInstanceId, FunctionComputeLogger logger) {
       DefaultProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);
       String lb = "lb-****";
       AddBackendServersRequest request = new AddBackendServersRequest();
       request.setBackendServers(String.format("[{\"ServerId\":\"%s\",\"Weight\":50}]",ecsInstanceId));
       request.setLoadBalancerId(lb);
       try {
           AddBackendServersResponse response = client.getAcsResponse(request);
           logger.info(String.format("add backend server %s to lb %s, requestId:%s, ", ecsInstanceId, lb, response.getRequestId()));
       } catch (ClientException e) {
           logger.info(String.format("failure of add backend server %s to lb %s, errorMsg:%s", ecsInstanceId, lb, e.getLocalizedMessage()));
       }
   }

   private void removeBackendServer(String regionId, String ecsInstanceId, FunctionComputeLogger logger) {
       DefaultProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);
       String lb = "lb-****";
       RemoveBackendServersRequest request = new RemoveBackendServersRequest();
       request.setBackendServers(String.format("[\"%s\"]",ecsInstanceId));
       request.setLoadBalancerId(lb);
       try {
           RemoveBackendServersResponse response = client.getAcsResponse(request);
           logger.info(String.format("remove backend server %s to lb %s, requestId:%s, ", ecsInstanceId, lb, response.getRequestId()));
       } catch (ClientException e) {
           logger.info(String.format("failure of remove backend server %s to lb %s, errorMsg:%s", ecsInstanceId, lb, e.getLocalizedMessage()));
       }
   }

   private void mountDisk(String regionId, String ecsInstanceId, String diskId, FunctionComputeLogger logger) {
       logger.info("prepare mount diskId:"+diskId);
       AttachDiskRequest request = new AttachDiskRequest();
       request.setDiskId(diskId);
       request.setInstanceId(ecsInstanceId);
       IClientProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);
       try {
           AttachDiskResponse response = client.getAcsResponse(request);
           logger.info("mount disk result:" + JSON.toJSONString(response));
       } catch (ClientException e) {
           logger.error("mount failure, diskId:" + diskId + e.getMessage());
       }

   }

   private void unmountDisk(String regionId, String ecsInstanceId, String diskId, FunctionComputeLogger logger) {
       logger.info("prepare mount diskId:"+diskId);
       DetachDiskRequest request = new DetachDiskRequest();
       request.setDiskId(diskId);
       request.setInstanceId(ecsInstanceId);
       IClientProfile profile = DefaultProfile.getProfile(regionId, ak, aks);
       IAcsClient client = new DefaultAcsClient(profile);
       try {
           DetachDiskResponse response = client.getAcsResponse(request);
           logger.info("unmount disk result:" + JSON.toJSONString(response));
       } catch (ClientException e) {
           logger.error("unmount failure, diskId:" + diskId);
       }
   }
}

欢迎扫描下方二维码，加入云监控用户支持群。

码农公寓

背景