kubernetes 部署容器监控方案

2022-03-06 02:13:06

===============================================

2021/2/9_第1次修改 ccb_warlock

===============================================

在很多年前整理过的容器监控方案（https://www.cnblogs.com/straycats/p/9281889.html）中，曾经采用在docker swarm中运行cAdvisor、Prometheus、Grafana来实现对容器与宿主机的监控。因为懂docker，上个月又被当作运维要求通过在kubernetes上实现监控系统的解决方案，于是我需要实现在kubernetes上运行这套解决方案。

在使用grafana的demo时，了解到监控k8s资源有个比cAdvisor更好用的服务Kube-state-metrics。

cAdvisor：

Kube-state-metrics：

Prometheus：

Grafana：

Alertmanager：

一、部署kubernetes

centos7可以参考：https://www.cnblogs.com/straycats/p/14322995.html

写教程时部署的版本是v1.20.1

二、部署cAdvisor

因为在kubernetes上运行，而kubelet已经集成了cAdvisor，所以不需要额外安装，直接使用kubelet即可。

三、部署Kube-state-metrics

3.1 创建编排脚本

# 创建目录

mkdir -p /opt/yaml

# 创建编排脚本

vi /opt/yaml/kube-state-metrics.yaml

将下面的内容保存到kube-state-metrics.yaml中，wq保存。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v2.0.0-beta
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - list
  - watch

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v2.0.0-beta
  name: kube-state-metrics
  namespace: monit

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v2.0.0-beta
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monit

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: ‘true‘
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v2.0.0-beta
  name: kube-state-metrics
  namespace: monit
spec:
  type: NodePort
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    #nodePort: 30001
  - name: telemetry
    port: 8081
    targetPort: telemetry
    #nodePort: 30002
  selector:
    app.kubernetes.io/name: kube-state-metrics

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v2.0.0-beta
  name: kube-state-metrics
  namespace: monit
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v2.0.0-beta
    spec:
      containers:
      - image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-beta
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http-metrics
        - containerPort: 8081
          name: telemetry
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
      nodeSelector:
        beta.kubernetes.io/os: linux
      serviceAccountName: kube-state-metrics

PS.如何获取镜像k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-beta，我会另外补一篇文章来描述操作。

3.2 部署

# 执行编排脚本

cd /opt/yaml
kubectl apply -f kube-state-metrics.yaml

四、部署Prometheus

4.1 创建数据持久化目录

mkdir -p /opt/vol/prometheus/data

4.2 创建编排脚本