ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard

 Ceph Dashboard介绍   Ceph 的监控可视化界面方案很多----grafana、Kraken。但是从Luminous开始,Ceph 提供了原生的Dashboard功能,通过Dashboard可以获取Ceph集群的各种基本状态信息。  (nautilus版) 需要安装 ceph-mgr-dashboard ceph nautilus dashboard 已经具有一些管理功能了,可以在WEB界面创建一些内容   1.    Install the appropriate package from the below link # ceph -v ceph version 14.2.9 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) # yum install ceph-mgr-dashboard Note: it will ask for few dependencies, install with yum/apt package manager and then execute the above command. 注:所有mgr节点都安装上ceph-mgr-dashboard包, 2.    Enable the ceph mgr dashboard # ceph mgr module enable dashboard --force  # ceph mgr module ls | grep dashboard   3.    Create a user for Dashboard Example: [ceph dashboard ac-user-create (username) (password) administrator] # ceph dashboard ac-user-create admin admin administrator {"username": "admin", "lastUpdate": 1581346477, "name": null, "roles": ["administrator"], "password": "$2b$12$55BR7an1KU9Rp3pBcc7HiOWNmI45kNzZlAJVK1c/Tk7krnGEEfY4S", "email": null}   ------------------------------------------------------------------------ 注:新点的版本比如14.2.19,创建用户的命令已经更改,操作如下 echo '123456' > password.txt ceph dashboard ac-user-create admin  administrator -i password.txt {"username": "admin", "lastUpdate": 1619334864, "name": null, "roles": ["administrator"], "password": "$2b$12$3XoNkgAD19mGLgjX28s4Z.9os8kMJoRglLuMWejXVINGxNiMNQA2W", "email": null} ------------------------------------------------------------------------   [root@ceph-host-01 mgr-dashboard]# ceph mgr services {     "dashboard": "https://ceph-host-01:8443/" } Note: Here you can access with IP address of ceph-mgr node, instead of hostname. 5.    Make sure firewall port is open # firewall-cmd --add-port=8443/tcp --permanent  # firewall-cmd --reload  6.    Open the dashboard url in any browser https://ceph-host-01:8443/or https://10.30.1.221:8443 7.    Enter the username: cent and password: password 配置好之后就可以登录,页面如下: ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboardceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard     注:当mgr主机重启后,https://10.30.1.221:8443无法打开页面来了,解决办法是在admin节点重新把那个mgr节点加入进来,或者查看到现在的active的mgr节点是哪个,如果active的mgr节点有安装ceph-mgr-dashboard,则直接使用active的mgr节点IP登陆就好 [root@ceph-host-01 ceph-cluster]# ceph -s   cluster:     id:     272905d2-fd66-4ef6-a772-9cd73a274683     health: HEALTH_WARN             application not enabled on 1 pool(s)             8 daemons have recently crashed   services:     mon: 3 daemons, quorum ceph-host-01,ceph-host-02,ceph-host-03 (age 14m)     mgr: ceph-host-03(active, since 15m), standbys: ceph-host-02, ceph-host-01     mds: nova:1 {0=ceph-host-02=up:active} 1 up:standby     osd: 20 osds: 20 up (since 13m), 20 in (since 19h)   data:     pools:   6 pools, 512 pgs     objects: 1.10k objects, 7.6 GiB     usage:   35 GiB used, 1.5 TiB / 1.5 TiB avail     pgs:     512 active+clean [root@ceph-host-01 ceph-cluster]# ceph-deploy mgr create ceph-host-01     总结ceph nautilus开启dashboard的步骤 ----------------------------------------------------------------------------   mgr节点操作:   yum -y install ceph-mgr-dashboard pip uninstall urllib3 yum install python-urllib3 -y   ceph管理节点操作: ceph mgr module enable dashboard ceph config set mgr mgr/dashboard/server_addr 0.0.0.0 ceph config set mgr mgr/dashboard/server_port 7000 ceph config set mgr mgr/dashboard/ssl false ceph dashboard ac-user-create admin password administrator   修改默认配置命令#   指定集群dashboard的访问端口 # ceph config set mgr mgr/dashboard/server_port 7000 指定集群 dashboard的访问IP # ceph config set mgr mgr/dashboard/server_addr 0.0.0.0 关闭ssl # ceph config set mgr mgr/dashboard/ssl false   开启Object Gateway管理功能# 1、创建rgw用户 radosgw-admin user create --uid=dashboard --display-name='dashboard rgw'   --system # radosgw-admin user info --uid=dashboard 2、提供Dashboard证书 # ceph dashboard set-rgw-api-access-key $access_key # ceph dashboard set-rgw-api-secret-key $secret_key 3、配置rgw主机名和端口 # ceph dashboard set-rgw-api-host 0.0.0.0 # ceph dashboard set-rgw-api-port 7480 4、刷新web页面   ----------------------------------------------------------------------------------- 使用 Prometheus 监控 Ceph   ceph的可视化方案很多,本篇介绍的是比较简单的一种方式,并且对包都进行了二次封装,所以能够在极短的时间内构建出一个可视化的监控系统 本系统组件如下: * ceph-nautilus版本 * ceph_exporter的nautilus版本 * prometheus的2.20.1版本 * grafana的grafana-7.1.3版本 * Ceph grafana的插件- Clusterby Cristian Calin 适配的系统为centos7 资源如下: https://github.com/digitalocean/ceph_exporter(国内源https://github.com.cnpmjs.org/digitalocean/ceph_exporter) https://nodejs.org/dist/v10.13.0/node-v10.13.0-linux-x64.tar.xz http://static.zybuluo.com/zphj1987/1nu2k4cpcery94q2re3u6s1t/ceph-cluster_rev1.json https://github.com/prometheus/prometheus/releases/download/v2.24.1/prometheus-2.24.1.linux-amd64.tar.gz https://dl.grafana.com/oss/release/grafana-8.0.2-1.x86_64.rpm   注:一些包的来源地址 grafana下载源 https://github.com/grafana/grafana/tags https://grafana.com/grafana/download/8.0.2 prometheus下载源 https://github.com/prometheus/prometheus/tags ceph_exporter下载源 https://github.com/digitalocean/ceph_exporter/tags     以上资源均可以直接用wget进行下载,然后直接安装 监控的架构介绍 通过ceph_exporter抓取的ceph相关的数据并且在本地监听端口9128端口 prometheus抓取ceph_exporter的9128的端口的数据存储在本地的/usr/local/prometheus/目录下面 grafana抓取prometheus的数据进行渲染成web页面 页面的模板就是使用的grafana的ceph模板插件 那么我们就根据上面的架构去一步步的把系统配置起来 配置监控系统 安装ceph_exporter 方法1: https://docs.ceph.com/docs/master/mgr/prometheus/ ceph mgr module enable prometheus 端口 9283   方法2:     1)安装需要的软件golang       #yum install golang git librados2-devel librbd1-devel -y     2)设置go的环境变量       # /etc/profile.d/go.sh         export GOROOT=/usr/lib/golang         export GOBIN=$GOROOT/bin         export GOPATH=/home/golang         export PATH=$PATH:$GOROOT/bin:$GOPATH/bin       #source /etc/profile.d/go.sh     3)安装ceph_exporter       #cd /home/golang       #go get -u github.com/digitalocean/ceph_exporter    (由于被墙我们只能放弃这个办法而改用下面的命令)                     # mkdir -pv src/github.com/digitalocean/                     # cd src/github.com/digitalocean/ceph_exporter/                     # git clone https://github.com.cnpmjs.org/digitalocean/ceph_exporter                     # go build     4)运行ceph_exporter       #cd /usr/lib/golang/bin/       #nohup ./ceph_exporter &     5)检验结果       #curl 127.0.0.1:9128 可以看到端口起来了就是安装成功了,这个ceph_exporter建议是安装在管理节点上,也就是能够执行出ceph -s的节点上面的   安装prometheus admin节点上部署prometheus wget https://github.com/prometheus/prometheus/releases/download/v2.24.1/prometheus-2.24.1.linux-amd64.tar.gz tar -xvf prometheus-2.24.1.linux-amd64.tar.gz cd prometheus-2.24.1.linux-amd64/ cp prometheus /usr/bin/    cp promtool /usr/bin/     # vim /lib/systemd/system/prometheus.service [Unit] Description=Prometheus: the monitoring system Documentation=http://prometheus.io/docs/   [Service] ExecStart=/usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090 --web.external-url= Restart=always StartLimitInterval=0 RestartSec=10   [Install] WantedBy=multi-user.target mkdir /etc/prometheus/ cp -R consoles console_libraries prometheus.yml /etc/prometheus/ mkdir /var/lib/prometheus/   systemctl daemon-reload systemctl enable prometheus.service systemctl start prometheus.service   vim /etc/prometheus/prometheus.yml # my global config global:   scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.   # scrape_timeout is set to the global default (10s).   # Alertmanager configuration alerting:   alertmanagers:   - static_configs:     - targets:       # - alertmanager:9093   # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files:   # - "first_rules.yml"   # - "second_rules.yml"   # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs:   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.   - job_name: 'prometheus'       # metrics_path defaults to '/metrics'     # scheme defaults to 'http'.       static_configs:     - targets: ['localhost:9090']     - job_name: 'ceph'     static_configs:     - targets: ['10.2.36.196:9283'] systemctl restart prometheus.service   这个地方默认是认为prometheus和ceph_exporter在一台机器上面,所以配置文件的/etc/ceph/prometheus.yml里面的targets写的是127.0.0.1,根据需要修改成ceph_exporter的ip地址即可   prometheus的默认监听端口为9090,到这个时候直接去web 上面就可以看到prometheus的抓取的数据了   到这里是数据到prometheus的已经完成了,下面就去做跟grafana相关的配置了 安装grafana wget https://dl.grafana.com/oss/release/grafana-8.0.2-1.x86_64.rpm yum install grafana-8.0.2-1.x86_64.rpm systemctl enable grafana-server.service systemctl start grafana-server.service   grafana默认监听的3000的端口 默认登陆的用户名密码为admin admin,登陆成功后会强制修改密码 配置grafana   首先增加数据源 ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboardceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard   这里如果能上网就直接输入id 917 ,如果不能上网就把上面的ceph-cluster_rev1.json文件弄到本地去,导入进去即可 ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard   ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard 把ceph-cluster_rev1.json内容粘贴进去就行     到这里就完成了配置了 ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard

 

 

  --------------------------------------------------------------------------------------------------------------------------------------------------   扩展:编译N版ceph rpm包支持中文dashboard 一、前言 N版ceph dashboard已经支持中文了,但是用默认是英语,对于有中文需求的人可以参考以下步骤,重新编译ceph RPM包,支持dashboard中文显示。 二,环境准备 需要一个大内存和多核的主机可以是虚拟机,磁盘空间要足够大(80GB),安装centos7.6 三、下载ceph源码包 用git clone方式下载ceph源码需要下载ceph的子模块,这个过程要连接国外的网站,下载速度非常的慢,因此下载ceph的tar源码包,下载地址:https://download.ceph.com/tarballs/。 四、编译dashboard angular ceph的dashboard使用的angular框架,默认情况下只编译了英文包,因此需要自己重新编译自己需要的语言包 1、解压 下载了ceph 源码包后,解压 wget https://download.ceph.com/tarballs/ceph_14.2.19.orig.tar.gz tar -xvf ceph_14.2.19.orig.tar.gz   2、创建临时文件 TEMP_DIR=`mktemp -d` 3、创建python虚拟机环境 cd ceph-14.2.19/ ./src/tools/setup-virtualenv.sh --python=python2.7 $TEMP_DIR 4、安装包 $TEMP_DIR/bin/pip install nodeenv $TEMP_DIR/bin/nodeenv -p --node=10.13.0 5、进入dashboard目录 cd  ./src/pybind/mgr/dashboard/frontend 6、激活python虚拟环境   . $TEMP_DIR/bin/activate 7、安装js相关包 npm ci   8、编译angular npx npm-run-all --print-label --parallel --max-parallel 2 "build:*  - - -- --prod"     #编译支持的所有语言也可以用:zh-CN编译中文   ------------------------------------------------------------------------ 报错:Error: Cannot find module 'node-sass' 解决办法: npm install -g cnpm --registry=https://registry.npm.taobao.org cnpm install node-sass   报错:Error:Node Sass version 6.0.0 is incompatible with ^4.0.0 卸载 node-sass  npm uninstall node-sass  然后安装最新版本(5.0之前)  npm install node-sass@4.14.1 ---------------------------------------------------------------------------   编译成功后查看dist目录下是否有对应的语言文件 编译成功后退出python虚拟环境 deactivate 9、重新打ceph tar包 tar czvf ceph_14.2.19.tar.gz ceph-14.2.19/ 五、编译ceph rpm包 1、安装ceph编译依赖包 执行ceph源码目录下的./install-deps.sh 2、安装gcc 7.3版本 yum install centos-release-scl scl-utils-build yum list all --enablerepo='centos-sclo-rh' 列出scl有哪些源可以用 yum install devtoolset-7-gcc.x86_64 devtoolset-7-gcc-c++.x86_64 devtoolset-7-gcc-gdb-plugin.x86_64     安装gcc-7.3编译工具链 scl enable devtoolset-7 bash     切换到gcc-7.3编译环境 3、设置编译环境 yum install rpm-build rpmdevtools 安装rpm包编译工具 rpmdev-setuptree 设置编译环境 cp ceph_14.2.19.tar.gz rpmbuild/SOURCES     将ceph 源码压缩包拷贝到rpmbuild目录下 tar --strip-components=1 -C ~/rpmbuild/SPECS/ --no-anchored -xvzf ~/rpmbuild/SOURCES/ceph_14.2.19.tar.gz "ceph.spec" "ceph.spec" 提取ceph.sepc文件 修改文件rpmbuild/SPECS/ceph.spec  编辑Source0:        http://ceph.com/download/ceph-<version>行, 将ceph-<version>改为rpmbuild/SOURCES下的文件名 4、编译ceph rpm包 执行  rpmbuild -ba ~/rpmbuild/SPECS/ceph.spec编译ceph rpm包   注:编译的过程会报大量错,主要是缺失一些包导致的 解决办法: 把所有报错中提到的缺失的包都安装上 yum install -y gperf cmake3 cryptsetup fuse-devel devtoolset-8-gcc-c++ gperftools-develleveldb-devellibaio-devel libblkid-devel libcurl-devel libcap-ng-devel libudev-devel libnl3-devel liboath-devel libtool libxml2-devel ncurses-devel snappy-devel valgrind-devel xfsprogs-devel xmlstarlet yasm librabbitmq-devel librdkafka-devel boost-random nss-devel keyutils-libs-devel libibverbs-devel librdmacm-devel openldap-devel openssl-devel CUnit-devel redhat-lsb-core python2-Cython python36-Cython python-prettytable python-sphinx lz4-devel lttng-ust-devel libbabeltrace-devel expat-devel junit gperftools-devel   查看结果 # find /root/rpmbuild/ -name "*.rpm" /root/rpmbuild/SRPMS/ceph-14.2.19-0.el7.src.rpm /root/rpmbuild/RPMS/x86_64/python3-rbd-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-test-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librados2-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-mon-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python3-rgw-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-mgr-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librgw-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librbd1-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-radosgw-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libcephfs_jni1-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-rgw-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-common-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-osd-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/rados-objclass-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librgw2-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-rados-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/cephfs-java-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/rbd-mirror-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-rbd-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-cephfs-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/rbd-nbd-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-base-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python3-rados-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-ceph-argparse-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python3-ceph-argparse-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-mds-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libcephfs2-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python-ceph-compat-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/rbd-fuse-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/python3-cephfs-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libcephfs-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-selinux-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-fuse-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libradosstriper1-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libcephfs_jni-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librbd-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libradospp-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/librados-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/libradosstriper-devel-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-resource-agents-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/x86_64/ceph-debuginfo-14.2.19-0.el7.x86_64.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-rook-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-local-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-ssh-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-cloud-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-k8sevents-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm /root/rpmbuild/RPMS/noarch/ceph-grafana-dashboards-14.2.19-0.el7.noarch.rpm 六、遇到的问题 1、编译rpm包过程中可能会报需要python3 yum install python36u -y yum install python36u-devel -y 2、编译rpm包过程中出现以下错误   在ceph.spec文件中加入 %define _binaries_in_noarch_packages_terminate_build 0   七、安装方法 rpm -Uvh  ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm --force   我们再次打开dashboard后发现支持中文了 ceph高可用分布式存储集群08-ceph nautilus 14.2.19安装 ceph mgr dashboard       作者:Dexter_Wang   工作岗位:某互联网公司资深云计算与存储工程师  联系邮箱:993852246@qq.com  
上一篇:ceph总结复习


下一篇:调整ceph的pg数(pg_num, pgp_num)