Kubernetes监控体系(1)-cAdvisor介绍

1. cAdvisor简介
​ cAdvisor对Node机器上的资源及容器进行实时监控和性能数据采集,包括CPU使用情况、内存使用情况、网络吞吐量及文件系统使用情况,cAdvisor集成在Kubelet中,当kubelet启动时会自动启动cAdvisor,即一个cAdvisor仅对一台Node机器进行监控。kubelet的启动参数–cadvisor-port可以定义cAdvisor对外提供服务的端口,默认为4194。可以通过浏览器访问。项目主页:http://github.com/google/cadvisor 。

2. cAdvisor结构图

Kubernetes监控体系(1)-cAdvisor介绍
3. Metrics
分类    字段    描述

1.cAvisor简介:
cAdvisor是Google开源的容器资源监控和性能分析工具,它是专门为容器而生,在Kubernetes中,我们不需要单独去安装,cAdvisor作为kubelet内置的一部分程序可以直接使用,也就是我们可以直接使用cadvisor采集数据,可以采集到和容器运行相关的所有指标,单独安装cAdvisor时的数据路径为/api/v1/nodes/[节点名称]/proxy/metrics/cadvisor,如果cadvisor集成到kubelet,采集数据的路径是https://127.0.0.1:10250/metrics/cadvisor

2.查看cadvisor监控指标,在k8s-master节点操作
kubectl create ns monitor-sa  #创建一个monitor-sa的名称空间
kubectl create serviceaccount monitor -n monitor-sa  #创建一个sa账号
kubectl create clusterrolebinding monitor-clusterrolebinding -n monitor-sa --clusterrole=cluster-admin  --serviceaccount=monitor-sa:monitor  
#把sa账号monitor通过clusterrolebing绑定到clusterrole上

kubectl get secret -n monitor-sa   #查看monitor-sa名称空间下的secret密钥
kubectl describe secret  monitor-token-j4jwf -n monitor-sa    #可看到token相关的内容如下所示
eyJhbGciOiJSUzI1NiIsImtpZCI6IkV5VUZuUmlPa0pMSF9sSFdUYktjdWdGVk9CR3owMlZhUDg4UzdVQWtveEEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tajRqd2YiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImQ5NzJiNDA1LWEzZTYtNDJiYS04YzU3LTA2MjE2YmE3Nzk1MCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.U0fMb34xlIcMrC5g_v3jeTMwxg3L3VkAD6lUa84Ke3kor3aB9tT092PM4N5_8cVPRJkHkh5UXx3A7mWOErjftgux41azA2N1Zkuqt-7VXkvvBCOBAmv-95mRz9FPEbzbR9gG5EudcCFeJypYOO3n7Oipr1MS4YxGLYVjUTQ46f5GIMJli9Uw6MYkij9HwuoD8qbLulAq6W540qvJfK4Bd20kvjqzZQveD2Ej-hmUlHR2cqshgD64VgBOIAJJir4bQ04JthLqgpC9peTTYo2hJ8XK-Y5OCx2v419syb0xPC2jrzwcZabvTBG_QCB4Ly8BRAxjEDB4ox3R6EMw8Ie68A

通过下面命令可以获取到cadvisor采集的指标数据
curl https://127.0.0.1:10250/metrics/cadvisor -k -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkV5VUZuUmlPa0pMSF9sSFdUYktjdWdGVk9CR3owMlZhUDg4UzdVQWtveEEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tajRqd2YiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImQ5NzJiNDA1LWEzZTYtNDJiYS04YzU3LTA2MjE2YmE3Nzk1MCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.U0fMb34xlIcMrC5g_v3jeTMwxg3L3VkAD6lUa84Ke3kor3aB9tT092PM4N5_8cVPRJkHkh5UXx3A7mWOErjftgux41azA2N1Zkuqt-7VXkvvBCOBAmv-95mRz9FPEbzbR9gG5EudcCFeJypYOO3n7Oipr1MS4YxGLYVjUTQ46f5GIMJli9Uw6MYkij9HwuoD8qbLulAq6W540qvJfK4Bd20kvjqzZQveD2Ej-hmUlHR2cqshgD64VgBOIAJJir4bQ04JthLqgpC9peTTYo2hJ8XK-Y5OCx2v419syb0xPC2jrzwcZabvTBG_QCB4Ly8BRAxjEDB4ox3R6EMw8Ie68A"


3.cadvisor中获取到的典型监控指标如下:

指标名称                                                  类型              含义
container_cpu_load_average_10s           gauge           过去10秒容器CPU的平均负载
container_cpu_usage_seconds_total       counter         容器在每个CPU内核上的累积占用时间 (单位:秒)
container_cpu_system_seconds_total      counter         System CPU累积占用时间(单位:秒)
container_cpu_user_seconds_total          counter          User CPU累积占用时间(单位:秒)
container_fs_usage_bytes                         gauge           容器中文件系统的使用量(单位:字节)
container_fs_limit_bytes                           gauge           容器可以使用的文件系统总量(单位:字节)
container_fs_reads_bytes_total                 counter         容器累积读取数据的总量(单位:字节)
container_fs_writes_bytes_total                counter         容器累积写入数据的总量(单位:字节)
container_memory_max_usage_bytes      gauge           容器的最大内存使用量(单位:字节)
container_memory_usage_bytes               gauge           容器当前的内存使用量(单位:字节
container_spec_memory_limit_bytes        gauge            容器的内存使用量限制
machine_memory_bytes                              gauge            当前主机的内存总量
container_network_receive_bytes_total     counter           容器网络累积接收数据总量(单位:字节)
container_network_transmit_bytes_total   counter           容器网络累积传输数据总量(单位:字节)


4.当能够正常采集到cAdvisor的样本数据后,可以通过以下表达式计算容器的CPU使用率:

(1)sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu)
容器CPU使用率

(2)container_memory_usage_bytes{image!=""}
查询容器内存使用量(单位:字节):

(3)sum(rate(container_network_receive_bytes_total{image!=""}[1m])) without (interface)
查询容器网络接收量(速率)(单位:字节/秒):

(4)sum(rate(container_network_transmit_bytes_total{image!=""}[1m])) without (interface)
容器网络传输量 字节/秒

(5)sum(rate(container_fs_reads_bytes_total{image!=""}[1m])) without (device)
容器文件系统读取速率 字节/秒

(6)sum(rate(container_fs_writes_bytes_total{image!=""}[1m])) without (device)
容器文件系统写入速率 字节/秒



5.cadvisor 常用容器监控指标
(1)网络流量
sum(rate(container_network_receive_bytes_total{name=~".+"}[1m])) by (name)
 ##容器网络接收的字节数(1分钟内),根据名称查询 name=~".+"

sum(rate(container_network_transmit_bytes_total{name=~".+"}[1m])) by (name)
 ##容器网络传输的字节数(1分钟内),根据名称查询 name=~".+"

 
(2)容器 CPU相关
sum(rate(container_cpu_system_seconds_total[1m]))
###所用容器system cpu的累计使用时间(1min钟内)

sum(irate(container_cpu_system_seconds_total{image!=""}[1m])) without (cpu)
 ###每个容器system cpu的使用时间(1min钟内)


sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name) * 100
#每个容器的cpu使用率


sum(sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name) * 100)
#总容器的cpu使用率




4. cAdvisor源码
4.1. cAdvisor入口函数
cadvisor.go

func main() {
    defer glog.Flush()
    flag.Parse()
    if *versionFlag {
        fmt.Printf("cAdvisor version %s (%s)\n", version.Info["version"], version.Info["revision"])
        os.Exit(0)
    }
    setMaxProcs()
    memoryStorage, err := NewMemoryStorage()
    if err != nil {
        glog.Fatalf("Failed to initialize storage driver: %s", err)
    }
    sysFs, err := sysfs.NewRealSysFs()
    if err != nil {
        glog.Fatalf("Failed to create a system interface: %s", err)
    }
    collectorHttpClient := createCollectorHttpClient(*collectorCert, *collectorKey)
    containerManager, err := manager.New(memoryStorage, sysFs, *maxHousekeepingInterval, *allowDynamicHousekeeping, ignoreMetrics.MetricSet, &collectorHttpClient)
    if err != nil {
        glog.Fatalf("Failed to create a Container Manager: %s", err)
    }
    mux := http.NewServeMux()
    if *enableProfiling {
        mux.HandleFunc("/debug/pprof/", pprof.Index)
        mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
        mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
        mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    }
    // Register all HTTP handlers.
    err = cadvisorhttp.RegisterHandlers(mux, containerManager, *httpAuthFile, *httpAuthRealm, *httpDigestFile, *httpDigestRealm)
    if err != nil {
        glog.Fatalf("Failed to register HTTP handlers: %v", err)
    }
    cadvisorhttp.RegisterPrometheusHandler(mux, containerManager, *prometheusEndpoint, nil)
    // Start the manager.
    if err := containerManager.Start(); err != nil {
        glog.Fatalf("Failed to start container manager: %v", err)
    }
    // Install signal handler.
    installSignalHandler(containerManager)
    glog.Infof("Starting cAdvisor version: %s-%s on port %d", version.Info["version"], version.Info["revision"], *argPort)
    addr := fmt.Sprintf("%s:%d", *argIp, *argPort)
    glog.Fatal(http.ListenAndServe(addr, mux))
}
核心代码:

memoryStorage, err := NewMemoryStorage()
sysFs, err := sysfs.NewRealSysFs()
#创建containerManager
containerManager, err := manager.New(memoryStorage, sysFs, *maxHousekeepingInterval, *allowDynamicHousekeeping, ignoreMetrics.MetricSet, &collectorHttpClient)
#启动containerManager
err := containerManager.Start()

4.2. cAdvisor Client的使用
import "github.com/google/cadvisor/client"
func main(){
    client, err := client.NewClient("http://192.168.19.30:4194/")   //http://<host-ip>:<port>/
}

4.2.1 client定义
cadvisor/client/client.go

// Client represents the base URL for a cAdvisor client.
type Client struct {
    baseUrl string
}
// NewClient returns a new v1.3 client with the specified base URL.
func NewClient(url string) (*Client, error) {
    if !strings.HasSuffix(url, "/") {
        url += "/"
    }
    return &Client{
        baseUrl: fmt.Sprintf("%sapi/v1.3/", url),
    }, nil
}

4.2.2. client方法
1)MachineInfo

// MachineInfo returns the JSON machine information for this client.
// A non-nil error result indicates a problem with obtaining
// the JSON machine information data.
func (self *Client) MachineInfo() (minfo *v1.MachineInfo, err error) {
       u := self.machineInfoUrl()
       ret := new(v1.MachineInfo)
       if err = self.httpGetJsonData(ret, nil, u, "machine info"); err != nil {
              return
       }
       minfo = ret
       return
}

2)ContainerInfo

// ContainerInfo returns the JSON container information for the specified
// container and request.
func (self *Client) ContainerInfo(name string, query *v1.ContainerInfoRequest) (cinfo *v1.ContainerInfo, err error) {
       u := self.containerInfoUrl(name)
       ret := new(v1.ContainerInfo)
       if err = self.httpGetJsonData(ret, query, u, fmt.Sprintf("container info for %q", name)); err != nil {
              return
       }
       cinfo = ret
       return
}

3)DockerContainer

// Returns the JSON container information for the specified
// Docker container and request.
func (self *Client) DockerContainer(name string, query *v1.ContainerInfoRequest) (cinfo v1.ContainerInfo, err error) {
       u := self.dockerInfoUrl(name)
       ret := make(map[string]v1.ContainerInfo)
       if err = self.httpGetJsonData(&ret, query, u, fmt.Sprintf("Docker container info for %q", name)); err != nil {
              return
       }
       if len(ret) != 1 {
              err = fmt.Errorf("expected to only receive 1 Docker container: %+v", ret)
              return
       }
       for _, cont := range ret {
              cinfo = cont
       }
       return
}

4)AllDockerContainers

// Returns the JSON container information for all Docker containers.
func (self *Client) AllDockerContainers(query *v1.ContainerInfoRequest) (cinfo []v1.ContainerInfo, err error) {
       u := self.dockerInfoUrl("/")
       ret := make(map[string]v1.ContainerInfo)
       if err = self.httpGetJsonData(&ret, query, u, "all Docker containers info"); err != nil {
              return
       }
       cinfo = make([]v1.ContainerInfo, 0, len(ret))
       for _, cont := range ret {
              cinfo = append(cinfo, cont)
       }
       return
}

文章参考:http://blog.opskumu.com/cadvisor.html

上一篇:prometheus cAdvisor 监控docker CPU利用率 教程Prometheus监控docker容器


下一篇:LINQ检索使用