kubernetes资源监控
1. 查看集群资源状况
·k8s集群的master节点一般不会跑业务容器·
kubectl get cs #查看master资源状态
kubectl get node #查看节点
kubectl cluster-info #查看集群状态
kubectl describe pods [pod名] #查看各类资源状态
kubectl get pods -o wide #查看更新信息
2.监控集群资源利用率【metrics-server安装使用】
#可以通过kubectl 来查看资源利用率，但是该命令需要 "heapster" 调用资源才行，如果没有提供资源则会报错如下：
[root@k8s-master1 ~]# kubectl top node k8s-node1
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
#heapster组件已经被弃用,现在使用的是： metrics-server + cAdvisor聚合器来提供资源。
#cAdvisor 已经内置于kubelet组件中。所以需要安装metrics-server
metrics-server架构示意图：
wget https://www.chenleilei.net/soft/k8s/metrics-server.zip
#安装metrics-server 或 上传相关包
git clone https://github.com/kubernetes-incubator/metrics-server
cd metrics-server/
vim metrics-server-deployment.yaml
31,32行改为：
- name: metrics-server
image: lizhenliang/metrics-server-amd64:v0.3.1
33行 - --secure-port=4443 下方加入以下两行内容[从github中下载需要添加这两处，从博客下载这里就不用改了已经改好了]：
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP [这里意思是讲链接方式改为内部IP链接]
[注意：metrics-server是通过主机名来区分主机的,所以说必须要配置 host解析，metrics-server才能正确的采集到目标。]
修改配置文件 vim /etc/kubernetes/manifests/kube-apiserver.yaml
大约20行 - --enable-admission-plugins=NodeRestriction的下方添加1行代码：
- --enable-aggregator-routing=true
root@k8s-master1 metrics-server]# ll
otal 28
rw-r--r-- 1 root root 397 Mar 15 21:04 aggregated-metrics-reader.yaml
rw-r--r-- 1 root root 303 Mar 15 21:04 auth-delegator.yaml
rw-r--r-- 1 root root 324 Mar 15 21:04 auth-reader.yaml
rw-r--r-- 1 root root 298 Mar 15 21:04 metrics-apiservice.yaml #将merics-server注册到k8s的api中
rw-r--r-- 1 root root 1277 Mar 27 15:57 metrics-server-deployment.yaml #部署metrics-server
rw-r--r-- 1 root root 297 Mar 15 21:04 metrics-server-service.yaml
rw-r--r-- 1 root root 532 Mar 15 21:04 resource-reader.yaml
修改完毕后直接安装：
[root@k8s-master1 metrics-server]# kubectl apply -f .
提示：
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
# 重启 kubelet：
systemctl restart kubelet
检查安装：
[root@k8s-master1 metrics-server]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-kvbbh 1/1 Running 0 6d4h
coredns-7ff77c879f-lqk9q 1/1 Running 5 6d4h
etcd-k8s-master1 1/1 Running 2 8d
kube-apiserver-k8s-master1 1/1 Running 2 8d
kube-controller-manager-k8s-master1 1/1 Running 2 8d
kube-flannel-ds-amd64-8gssm 1/1 Running 5 8d
kube-flannel-ds-amd64-gtpwc 1/1 Running 2 8d
kube-flannel-ds-amd64-mx4jx 1/1 Running 4 8d
kube-proxy-kzwft 1/1 Running 5 8d
kube-proxy-rgjmf 1/1 Running 2 8d
kube-proxy-vhdpp 1/1 Running 4 8d
kube-scheduler-k8s-master1 1/1 Running 2 8d
metrics-server-5667498b7d-lmbtr 1/1 Running 0 56s <------metrics-server安装完毕。
查看：
[root@k8s-master1 metrics-server]# kubectl get apiservice
看到这行：
v1beta1.metrics.k8s.io kube-system/metrics-server True 17s #启动成功
验证配置kubectl top配置：
[root@k8s-master1 metrics-server]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
nginx-f89759699-6jfdp 0m 2Mi
#最开始执行这条命令报错，现在正常：
[root@k8s-master1 metrics-server]# kubectl top node k8s-node1
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% #<---- 成功
k8s-node1 42m 2% 383Mi 13%
[root@k8s-master1 metrics-server]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master1 94m 4% 809Mi 27% #<---- 成功
k8s-node1 48m 2% 385Mi 13%
k8s-node2 26m 1% 386Mi 13%
#资源利用率排序
[root@k8s-master1 metrics-server]# kubectl top pods -l app=nginx --sort-by=memory
NAME CPU(cores) MEMORY(bytes)
nginx-f89759699-6jfdp 0m 2Mi
3. 问题排查
[root@k8s-master1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
如果kubectl get cs查询失败，可能是apiserver出现了问题。
#apiserver 正常可以通过该命令查看状态
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://10.0.0.63:6443
KubeDNS is running at https://10.0.0.63:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump‘.
# 通过dump.log查看集群问题
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://10.0.0.63:6443
KubeDNS is running at https://10.0.0.63:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump‘.
#状态信息写入到文件：
kubectl cluster-info dump >a.txt
#kubectl describe pod [pod名]
#实时观察pod动态
kubectl get pods -w [删除创建pod，这条命令里都会输出出来，并可以显示整理流程]