ceph集群安装

1、环境节点准备:

角色      IP
master 192.168.77.10   系统盘:/dev/sda   数据盘:/dev/sdb
node1  192.168.77.11   系统盘:/dev/sda   数据盘:/dev/sdb
node2  192.168.77.12   系统盘:/dev/sda   数据盘:/dev/sdb

2、给虚拟机增加数据盘:

ceph集群安装

 

 

 

ceph集群安装

 

 

 

然后应用保存,其他两台虚拟机也这样操作,然后将三台虚拟机进行重启,reboot:

重启后然后查询数据盘:

[root@master ~]# ansible k8s -m shell -a "lsblk|grep sdb"
192.168.77.10 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk 
192.168.77.11 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk 
192.168.77.12 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk

  

数据盘添加完成

3、环境初始化:

主机名设置,并设置hosts文件,并配置ssh互信,因为已经配置了ansible,所以这一块都是已完成:

[root@master ~]# ansible k8s -m shell -a "hostname"
192.168.77.10 | CHANGED | rc=0 >>
master
192.168.77.11 | CHANGED | rc=0 >>
node1
192.168.77.12 | CHANGED | rc=0 >>
node2

  

关闭防火墙、关闭selinux:

[root@master ~]# ansible k8s -m shell -a "systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld"

  

[root@master ~]# ansible k8s -m shell -a "sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config"
[root@master ~]# ansible k8s -m shell -a "setenforce 0"

  

注意:三个节点一定要时间同步

4、配置ceph源:

[root@master ~]# vim /etc/yum.repos.d/cephcentos7.repo
[root@master ~]# ansible k8s -m copy -a "src=/etc/yum.repos.d/cephcentos7.repo dest=/etc/yum.repos.d/cephcentos7.repo"

  

5、部署安装ceph集群,采用将这些rpm下载到本地再安装,以后离线也可以使用:

[root@master ~]# mkdir /root/ceph_deploy/
[root@master ~]# yum -y install --downloadonly --downloaddir=/root/ceph_deploy/ ceph-deploy python-pip
[root@master ~]# cd ceph_deploy/
[root@master ceph_deploy]# ll
total 2000
-rw-r--r--. 1 root root  292428 Apr 10  2020 ceph-deploy-2.0.1-0.noarch.rpm
-rw-r--r--. 1 root root 1752855 Sep  3  2020 python2-pip-8.1.2-14.el7.noarch.rpm
[root@master ceph_deploy]# yum localinstall *.rpm -y

  

安装ceph ceph-radosgw:

[root@master ceph_deploy]# ansible k8s -m shell -a "mkdir -p /root/ceph_deploy/pkg"
[root@master ceph_deploy]# ansible k8s -m shell -a "yum -y install --downloadonly --downloaddir=/root/ceph_deploy/pkg ceph  ceph-radosgw"
[root@master pkg]# ansible k8s -m shell -a "cd /root/ceph_deploy/pkg;yum  localinstall *.rpm -y"

  

开始创建ceph cluster:

[root@master pkg]# mkdir /root/ceph_deploy/cephcluster
[root@master pkg]# cd /root/ceph_deploy/cephcluster

  

[root@master cephcluster]# ceph-deploy new master node1 node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new master node1 node2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f6904d2ade8>
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f6904d27f80>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['master', 'node1', 'node2']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[master][DEBUG ] find the location of an executable
[master][INFO  ] Running command: /usr/sbin/ip link show
Exception KeyError: (7,) in <function remove at 0x7f690419d7d0> ignored
[master][INFO  ] Running command: /usr/sbin/ip addr show
[master][DEBUG ] IP addresses found: [u'192.168.77.10']
[ceph_deploy.new][DEBUG ] Resolving host master
[ceph_deploy.new][DEBUG ] Monitor master at 192.168.77.10
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[node1][DEBUG ] connected to host: master 
[node1][INFO  ] Running command: ssh -CT -o BatchMode=yes node1
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[node1][INFO  ] Running command: /usr/sbin/ip link show
[node1][INFO  ] Running command: /usr/sbin/ip addr show
[node1][DEBUG ] IP addresses found: [u'192.168.77.11']
[ceph_deploy.new][DEBUG ] Resolving host node1
[ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.77.11
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[node2][DEBUG ] connected to host: master 
[node2][INFO  ] Running command: ssh -CT -o BatchMode=yes node2
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[node2][INFO  ] Running command: /usr/sbin/ip link show
[node2][INFO  ] Running command: /usr/sbin/ip addr show
[node2][DEBUG ] IP addresses found: [u'192.168.77.12']
[ceph_deploy.new][DEBUG ] Resolving host node2
[ceph_deploy.new][DEBUG ] Monitor node2 at 192.168.77.12
[ceph_deploy.new][DEBUG ] Monitor initial members are ['master', 'node1', 'node2']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.77.10', '192.168.77.11', '192.168.77.12']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[root@master cephcluster]# ll
total 16
-rw-r--r--. 1 root root  239 Oct 24 05:42 ceph.conf
-rw-r--r--. 1 root root 4876 Oct 24 05:42 ceph-deploy-ceph.log
-rw-------. 1 root root   73 Oct 24 05:42 ceph.mon.keyring

  

查看配置文件:

[root@master cephcluster]# cat ceph.conf 
[global]
fsid = 4d95bda1-1894-4b87-be64-ca65a66376c8
mon_initial_members = master, node1, node2
mon_host = 192.168.77.10,192.168.77.11,192.168.77.12
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

修改ceph.conf 新增网络配置

[root@master cephcluster]# cat ceph.conf 
[global]
fsid = 4d95bda1-1894-4b87-be64-ca65a66376c8
mon_initial_members = master, node1, node2
mon_host = 192.168.77.10,192.168.77.11,192.168.77.12
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public network = 192.168.77.0/24
cluster network = 192.168.77.0/24

集群配置初始化,生成所有密钥,并启动monitor组件:

[root@master cephcluster]# ceph-deploy mon create-initial
输出结果:
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[master][DEBUG ] get remote short hostname
[master][DEBUG ] fetch remote file
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.master.asok mon_status
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-master/keyring auth get client.admin
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-master/keyring auth get client.bootstrap-mds
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-master/keyring auth get client.bootstrap-mgr
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-master/keyring auth get client.bootstrap-osd
[master][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-master/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpJXF3xy

生成的密钥文件:

[root@master cephcluster]# ls -l
total 72
-rw-------. 1 root root   113 Oct 24 05:47 ceph.bootstrap-mds.keyring
-rw-------. 1 root root   113 Oct 24 05:47 ceph.bootstrap-mgr.keyring
-rw-------. 1 root root   113 Oct 24 05:47 ceph.bootstrap-osd.keyring
-rw-------. 1 root root   113 Oct 24 05:47 ceph.bootstrap-rgw.keyring
-rw-------. 1 root root   151 Oct 24 05:47 ceph.client.admin.keyring
-rw-r--r--. 1 root root   305 Oct 24 05:45 ceph.conf
-rw-r--r--. 1 root root 42977 Oct 24 05:47 ceph-deploy-ceph.log
-rw-------. 1 root root    73 Oct 24 05:42 ceph.mon.keyring

配置的ceph-mon服务:

systemctl status ceph-mon@master
systemctl status ceph-mon@node1
systemctl status ceph-mon@node2
查看节点选举状态:
ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.master.asok mon_status|grep state
ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status|grep state
ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status|grep state

角色如下:

[root@master cephcluster]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.master.asok mon_status|grep state
    "state": "leader",
[root@node1 pkg]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status|grep state
    "state": "peon",
[root@node2 ~]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status|grep state
    "state": "peon",

没问题后,将ceph集群配置文件ceph.conf推送到三个节点:

[root@master cephcluster]# ceph-deploy admin master node1 node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin master node1 node2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc78aa97830>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['master', 'node1', 'node2']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7fc78b7ab230>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to master
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[master][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[root@master cephcluster]# ll /etc/ceph/
total 12
-rw-------. 1 root root 151 Oct 24 05:53 ceph.client.admin.keyring
-rw-r--r--. 1 root root 305 Oct 24 05:53 ceph.conf
-rw-r--r--. 1 root root  92 Jun 29 18:36 rbdmap
-rw-------. 1 root root   0 Oct 24 05:46 tmpnL8vKD

以上配置操作没问题后,检查ceph集群状态:

[root@master cephcluster]# ceph -s|grep -i health
    health: HEALTH_WARN

发现集群状态是warn,报错信息:mons are allowing insecure global_id reclaim

解决办法:禁用不安全模式

[root@master cephcluster]# ceph config set mon auth_allow_insecure_global_id_reclaim false
[root@master cephcluster]# ceph -s
  cluster:
    id:     4d95bda1-1894-4b87-be64-ca65a66376c8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum master,node1,node2 (age 8m)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

至此ceph集群目前搭建ok,集群状态正常

6、配置osd,之前已经在三个节点挂在了数据盘,没有做初始化操作:

[root@master cephcluster]# ansible k8s -m shell -a "lsblk|grep sdb"
192.168.77.10 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk 
192.168.77.12 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk 
192.168.77.11 | CHANGED | rc=0 >>
sdb               8:16   0   10G  0 disk

目前三个节点的数据盘都是10G的大小,现在将数据盘进行配置osd添加到ceph集群:

ceph-deploy disk zap master /dev/sdb
ceph-deploy osd create master --data /dev/sdb
ceph-deploy disk zap node1 /dev/sdb
ceph-deploy osd create node1 --data /dev/sdb
ceph-deploy disk zap node2 /dev/sdb
ceph-deploy osd create node2 --data /dev/sdb

具体执行信息如下:

[root@master cephcluster]# ceph-deploy disk zap master /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy disk zap master /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f37573fc6c8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : master
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f3757435938>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdb']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdb on master
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[master][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[master][DEBUG ] zeroing last few blocks of device
[master][DEBUG ] find the location of an executable
[master][INFO  ] Running command: /usr/sbin/ceph-volume lvm zap /dev/sdb
[master][WARNIN] --> Zapping: /dev/sdb
[master][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[master][WARNIN] Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync
[master][WARNIN]  stderr: 10+0 records in
[master][WARNIN] 10+0 records out
[master][WARNIN] 10485760 bytes (10 MB) copied
[master][WARNIN]  stderr: , 0.0272198 s, 385 MB/s
[master][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdb>
[root@master cephcluster]# ceph-deploy osd create master --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy osd create master --data /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f15e47b47e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  journal                       : None
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  host                          : master
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x7f15e47e78c0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.cli][INFO  ]  data                          : /dev/sdb
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[master][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to master
[master][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[master][WARNIN] osd keyring does not exist yet, creating one
[master][DEBUG ] create a keyring file
[master][DEBUG ] find the location of an executable
[master][INFO  ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[master][WARNIN] Running command: /usr/bin/ceph-authtool --gen-print-key
[master][WARNIN] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 155a0972-b125-4cf9-9324-a6852e4ec426
[master][WARNIN] Running command: /usr/sbin/vgcreate --force --yes ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07 /dev/sdb
[master][WARNIN]  stdout: Physical volume "/dev/sdb" successfully created.
[master][WARNIN]  stdout: Volume group "ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07" successfully created
[master][WARNIN] Running command: /usr/sbin/lvcreate --yes -l 2559 -n osd-block-155a0972-b125-4cf9-9324-a6852e4ec426 ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07
[master][WARNIN]  stdout: Logical volume "osd-block-155a0972-b125-4cf9-9324-a6852e4ec426" created.
[master][WARNIN] Running command: /usr/bin/ceph-authtool --gen-print-key
[master][WARNIN] Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
[master][WARNIN] Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-0
[master][WARNIN] Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07/osd-block-155a0972-b125-4cf9-9324-a6852e4ec426
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
[master][WARNIN] Running command: /usr/bin/ln -s /dev/ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07/osd-block-155a0972-b125-4cf9-9324-a6852e4ec426 /var/lib/ceph/osd/ceph-0/block
[master][WARNIN] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
[master][WARNIN]  stderr: 2021-10-24 05:58:25.289 7f6aca610700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
[master][WARNIN] 2021-10-24 05:58:25.289 7f6aca610700 -1 AuthRegistry(0x7f6ac40662f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[master][WARNIN]  stderr: got monmap epoch 2
[master][WARNIN] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQBALnVhzwPSEBAANhdbMRBKBY9aLA2sstsFLw==
[master][WARNIN]  stdout: creating /var/lib/ceph/osd/ceph-0/keyring
[master][WARNIN] added entity osd.0 auth(key=AQBALnVhzwPSEBAANhdbMRBKBY9aLA2sstsFLw==)
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
[master][WARNIN] Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 155a0972-b125-4cf9-9324-a6852e4ec426 --setuser ceph --setgroup ceph
[master][WARNIN]  stderr: 2021-10-24 05:58:25.818 7fb9720daa80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
[master][WARNIN] --> ceph-volume lvm prepare successful for: /dev/sdb
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[master][WARNIN] Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07/osd-block-155a0972-b125-4cf9-9324-a6852e4ec426 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
[master][WARNIN] Running command: /usr/bin/ln -snf /dev/ceph-38fea9c0-9680-453d-9af4-8d3dbfb1ed07/osd-block-155a0972-b125-4cf9-9324-a6852e4ec426 /var/lib/ceph/osd/ceph-0/block
[master][WARNIN] Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
[master][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[master][WARNIN] Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-155a0972-b125-4cf9-9324-a6852e4ec426
[master][WARNIN]  stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-155a0972-b125-4cf9-9324-a6852e4ec426.service to /usr/lib/systemd/system/ceph-volume@.service.
[master][WARNIN] Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
[master][WARNIN]  stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service to /usr/lib/systemd/system/ceph-osd@.service.
[master][WARNIN] Running command: /usr/bin/systemctl start ceph-osd@0
[master][WARNIN] --> ceph-volume lvm activate successful for osd ID: 0
[master][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[master][INFO  ] checking OSD status...
[master][DEBUG ] find the location of an executable
[master][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host master is now ready for osd use.
[root@master cephcluster]# ceph-deploy disk zap node1 /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy disk zap node1 /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fb0905006c8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : node1
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7fb090539938>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdb']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdb on node1
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[node1][DEBUG ] zeroing last few blocks of device
[node1][DEBUG ] find the location of an executable
[node1][INFO  ] Running command: /usr/sbin/ceph-volume lvm zap /dev/sdb
[node1][WARNIN] --> Zapping: /dev/sdb
[node1][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[node1][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync
[node1][WARNIN]  stderr: 10+0 records in
[node1][WARNIN] 10+0 records out
[node1][WARNIN] 10485760 bytes (10 MB) copied
[node1][WARNIN]  stderr: , 0.0342769 s, 306 MB/s
[node1][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdb>
[root@master cephcluster]# ceph-deploy osd create node1 --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy osd create node1 --data /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fb87c37e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  journal                       : None
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  host                          : node1
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x7f4fb87f68c0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.cli][INFO  ]  data                          : /dev/sdb
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][WARNIN] osd keyring does not exist yet, creating one
[node1][DEBUG ] create a keyring file
[node1][DEBUG ] find the location of an executable
[node1][INFO  ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[node1][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[node1][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4044bc47-2945-4748-8464-a76ca6d8c3c1
[node1][WARNIN] Running command: /usr/sbin/vgcreate --force --yes ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d /dev/sdb
[node1][WARNIN]  stdout: Physical volume "/dev/sdb" successfully created.
[node1][WARNIN]  stdout: Volume group "ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d" successfully created
[node1][WARNIN] Running command: /usr/sbin/lvcreate --yes -l 2559 -n osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1 ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d
[node1][WARNIN]  stdout: Logical volume "osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1" created.
[node1][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[node1][WARNIN] Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
[node1][WARNIN] Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-1
[node1][WARNIN] Running command: /bin/chown -h ceph:ceph /dev/ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d/osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-2
[node1][WARNIN] Running command: /bin/ln -s /dev/ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d/osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1 /var/lib/ceph/osd/ceph-1/block
[node1][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap
[node1][WARNIN]  stderr: 2021-10-24 05:58:37.698 7f9bfb6f0700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
[node1][WARNIN] 2021-10-24 05:58:37.698 7f9bfb6f0700 -1 AuthRegistry(0x7f9bf40662f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[node1][WARNIN]  stderr: got monmap epoch 2
[node1][WARNIN] Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-1/keyring --create-keyring --name osd.1 --add-key AQBMLnVhEGs5KRAA3tzQQNeZYJL07OaYV+Yagw==
[node1][WARNIN]  stdout: creating /var/lib/ceph/osd/ceph-1/keyring
[node1][WARNIN] added entity osd.1 auth(key=AQBMLnVhEGs5KRAA3tzQQNeZYJL07OaYV+Yagw==)
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
[node1][WARNIN] Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid 4044bc47-2945-4748-8464-a76ca6d8c3c1 --setuser ceph --setgroup ceph
[node1][WARNIN]  stderr: 2021-10-24 05:58:38.230 7fb8c1628a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
[node1][WARNIN] --> ceph-volume lvm prepare successful for: /dev/sdb
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1
[node1][WARNIN] Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d/osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1 --path /var/lib/ceph/osd/ceph-1 --no-mon-config
[node1][WARNIN] Running command: /bin/ln -snf /dev/ceph-7306e20f-d3f1-4fe6-9a02-c65b9f4f543d/osd-block-4044bc47-2945-4748-8464-a76ca6d8c3c1 /var/lib/ceph/osd/ceph-1/block
[node1][WARNIN] Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-2
[node1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1
[node1][WARNIN] Running command: /bin/systemctl enable ceph-volume@lvm-1-4044bc47-2945-4748-8464-a76ca6d8c3c1
[node1][WARNIN]  stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-1-4044bc47-2945-4748-8464-a76ca6d8c3c1.service to /usr/lib/systemd/system/ceph-volume@.service.
[node1][WARNIN] Running command: /bin/systemctl enable --runtime ceph-osd@1
[node1][WARNIN]  stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@1.service to /usr/lib/systemd/system/ceph-osd@.service.
[node1][WARNIN] Running command: /bin/systemctl start ceph-osd@1
[node1][WARNIN] --> ceph-volume lvm activate successful for osd ID: 1
[node1][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[node1][INFO  ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.
[root@master cephcluster]# ceph-deploy disk zap node2 /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy disk zap node2 /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f065e4ff6c8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : node2
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f065e538938>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdb']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdb on node2
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[node2][DEBUG ] zeroing last few blocks of device
[node2][DEBUG ] find the location of an executable
[node2][INFO  ] Running command: /usr/sbin/ceph-volume lvm zap /dev/sdb
[node2][WARNIN] --> Zapping: /dev/sdb
[node2][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[node2][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync
[node2][WARNIN]  stderr: 10+0 records in
[node2][WARNIN] 10+0 records out
[node2][WARNIN] 10485760 bytes (10 MB) copied
[node2][WARNIN]  stderr: , 0.0299158 s, 351 MB/s
[node2][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdb>
[root@master cephcluster]# ceph-deploy osd create node2 --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy osd create node2 --data /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f2c0fa327e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  journal                       : None
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  host                          : node2
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x7f2c0fa658c0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.cli][INFO  ]  data                          : /dev/sdb
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][WARNIN] osd keyring does not exist yet, creating one
[node2][DEBUG ] create a keyring file
[node2][DEBUG ] find the location of an executable
[node2][INFO  ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[node2][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[node2][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 496ca4ff-307e-419b-923c-dddf6304eefa
[node2][WARNIN] Running command: /usr/sbin/vgcreate --force --yes ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14 /dev/sdb
[node2][WARNIN]  stdout: Physical volume "/dev/sdb" successfully created.
[node2][WARNIN]  stdout: Volume group "ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14" successfully created
[node2][WARNIN] Running command: /usr/sbin/lvcreate --yes -l 2559 -n osd-block-496ca4ff-307e-419b-923c-dddf6304eefa ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14
[node2][WARNIN]  stdout: Logical volume "osd-block-496ca4ff-307e-419b-923c-dddf6304eefa" created.
[node2][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[node2][WARNIN] Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2
[node2][WARNIN] Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-2
[node2][WARNIN] Running command: /bin/chown -h ceph:ceph /dev/ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14/osd-block-496ca4ff-307e-419b-923c-dddf6304eefa
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-2
[node2][WARNIN] Running command: /bin/ln -s /dev/ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14/osd-block-496ca4ff-307e-419b-923c-dddf6304eefa /var/lib/ceph/osd/ceph-2/block
[node2][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap
[node2][WARNIN]  stderr: 2021-10-24 05:58:52.238 7f2b352f5700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
[node2][WARNIN] 2021-10-24 05:58:52.238 7f2b352f5700 -1 AuthRegistry(0x7f2b300662f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[node2][WARNIN]  stderr: got monmap epoch 2
[node2][WARNIN] Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQBbLnVhcWMnDRAApxlMO02H7rIOcgEo3qYbIw==
[node2][WARNIN]  stdout: creating /var/lib/ceph/osd/ceph-2/keyring
[node2][WARNIN] added entity osd.2 auth(key=AQBbLnVhcWMnDRAApxlMO02H7rIOcgEo3qYbIw==)
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/
[node2][WARNIN] Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 496ca4ff-307e-419b-923c-dddf6304eefa --setuser ceph --setgroup ceph
[node2][WARNIN]  stderr: 2021-10-24 05:58:52.768 7f290d041a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid
[node2][WARNIN] --> ceph-volume lvm prepare successful for: /dev/sdb
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
[node2][WARNIN] Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14/osd-block-496ca4ff-307e-419b-923c-dddf6304eefa --path /var/lib/ceph/osd/ceph-2 --no-mon-config
[node2][WARNIN] Running command: /bin/ln -snf /dev/ceph-bc7a7134-8143-4905-9f6c-bd24e0d1ce14/osd-block-496ca4ff-307e-419b-923c-dddf6304eefa /var/lib/ceph/osd/ceph-2/block
[node2][WARNIN] Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-2
[node2][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
[node2][WARNIN] Running command: /bin/systemctl enable ceph-volume@lvm-2-496ca4ff-307e-419b-923c-dddf6304eefa
[node2][WARNIN]  stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-496ca4ff-307e-419b-923c-dddf6304eefa.service to /usr/lib/systemd/system/ceph-volume@.service.
[node2][WARNIN] Running command: /bin/systemctl enable --runtime ceph-osd@2
[node2][WARNIN]  stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@2.service to /usr/lib/systemd/system/ceph-osd@.service.
[node2][WARNIN] Running command: /bin/systemctl start ceph-osd@2
[node2][WARNIN] --> ceph-volume lvm activate successful for osd ID: 2
[node2][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[node2][INFO  ] checking OSD status...
[node2][DEBUG ] find the location of an executable
[node2][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node2 is now ready for osd use.

配置成功后,查询ceph集群是否已经支持osd:

ceph集群安装

 

 

这里三个节点,每个节点一块数据盘,正常配置

7、部署mgr:

[root@master cephcluster]# ceph-deploy mgr create master node1 node2

执行信息如下:

[root@master cephcluster]# ceph-deploy mgr create master node1 node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mgr create master node1 node2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  mgr                           : [('master', 'master'), ('node1', 'node1'), ('node2', 'node2')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f3a32993a70>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f3a33274140>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts master:master node1:node1 node2:node2
[master][DEBUG ] connected to host: master 
[master][DEBUG ] detect platform information from remote host
[master][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to master
[master][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[master][WARNIN] mgr keyring does not exist yet, creating one
[master][DEBUG ] create a keyring file
[master][DEBUG ] create path recursively if it doesn't exist
[master][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.master mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-master/keyring
[master][INFO  ] Running command: systemctl enable ceph-mgr@master
[master][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@master.service to /usr/lib/systemd/system/ceph-mgr@.service.
[master][INFO  ] Running command: systemctl start ceph-mgr@master
[master][INFO  ] Running command: systemctl enable ceph.target
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][WARNIN] mgr keyring does not exist yet, creating one
[node1][DEBUG ] create a keyring file
[node1][DEBUG ] create path recursively if it doesn't exist
[node1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.node1 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-node1/keyring
[node1][INFO  ] Running command: systemctl enable ceph-mgr@node1
[node1][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@node1.service to /usr/lib/systemd/system/ceph-mgr@.service.
[node1][INFO  ] Running command: systemctl start ceph-mgr@node1
[node1][INFO  ] Running command: systemctl enable ceph.target
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][WARNIN] mgr keyring does not exist yet, creating one
[node2][DEBUG ] create a keyring file
[node2][DEBUG ] create path recursively if it doesn't exist
[node2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.node2 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-node2/keyring
[node2][INFO  ] Running command: systemctl enable ceph-mgr@node2
[node2][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@node2.service to /usr/lib/systemd/system/ceph-mgr@.service.
[node2][INFO  ] Running command: systemctl start ceph-mgr@node2
[node2][INFO  ] Running command: systemctl enable ceph.target

mgr服务:

systemctl status ceph-mgr@master
systemctl status ceph-mgr@node1
systemctl status ceph-mgr@node2

检查ceph集群安装mgr状态:

ceph集群安装

 

 

至此monitor、osd、mgr都已安装成功,现在部署mgr-dashboard:

[root@master cephmgrdashboard]# ansible k8s -m shell -a "mkdir -p /root/ceph_deploy/pkg/cephmgrdashboard"

下载rpm包,然后安装

[root@master cephmgrdashboard]# ansible k8s -m shell -a "yum -y install --downloadonly --downloaddir=/root/ceph_deploy/pkg/cephmgrdashboard ceph-mgr-dashboard"
[root@master cephmgrdashboard]# ansible k8s -m shell -a "cd /root/ceph_deploy/pkg/cephmgrdashboard;yum localinstall *.rpm -y"

ceph集群mgr服务启用dashboard模块:

[root@master cephmgrdashboard]# ceph mgr module enable dashboard
[root@master cephmgrdashboard]# ceph dashboard create-self-signed-cert
Self-signed certificate created
[root@master cephmgrdashboard]# echo "admin" > password
[root@master cephmgrdashboard]# ceph dashboard set-login-credentials admin -i password 
******************************************************************
***          WARNING: this command is deprecated.              ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************
Username and password updated
[root@master cephmgrdashboard]# ceph mgr services
{
    "dashboard": "https://master:8443/"
}

上面-i password是需要以文件形式进行配置,然后浏览器进行访问:

ceph集群安装

 

 

至此ceph集群安装完毕

上一篇:1.2 C程序变量


下一篇:Python3逐行分析代码运行时间