一： CephX 认证机制
Ceph 使用 cephx 协议对客户端进行身份认证，cephx 用于对 ceph 保存的数据进行认证访问和授权，用于对访问 ceph 的请求进行认证和授权检测，与 mon 通信的请求都要经过 ceph 认证通过，但是也可以在 mon 节点关闭 cephx认证，但是关闭认证之后任何访问都将被允许，因此无法保证数据的安全性。
1.1 授权流程
每个 mon 节点都可以对客户端进行身份认证并分发秘钥，因此多个 mon 节点就不存在单点故障和认证性能瓶颈。
mon 节点会返回用于身份认证的数据结构，其中包含获取 ceph 服务时用到的 session key,session key 通 过 客 户 端 秘 钥 进 行 加 密 ， 秘 钥 是 在 客 户 端 提 前 配 置 好 的 ，/etc/ceph/ceph.client.admin.keyring
1.2 访问流程
无论 ceph 客户端是哪种类型，例如块设备、对象存储、文件系统，ceph 都会在存储池中将所有数据存储为对象:
ceph 用户需要拥有存储池访问权限，才能读取和写入数据
ceph 用户必须拥有执行权限才能使用 ceph 的管理命令
1.3 ceph 用户
用户是指个人(ceph 管理者)或系统参与者(MON/OSD/MDS)。
通过创建用户，可以控制用户或哪个参与者能够访问 ceph 存储集群、以及可访问的存储池及存储池中的数据。
ceph 支持多种类型的用户，但可管理的用户都属于 client 类型
区分用户类型的原因在于，MON/OSD/MDS 等系统组件特使用 cephx 协议，但是它们为非客户端。
#node节点 test@ceph-node1:~$ cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQD55h9h5ICUJBAAfk/2gBzkwU+G8bfqY023Yg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
1.4 ceph 授权和使能
ceph 基于使能/能力(Capabilities，简称 caps )来描述用户可针对 MON/OSD 或 MDS 使用的授权范围或级别。
能力一览表： r：向用户授予读取权限。访问监视器(mon)以检索 CRUSH 运行图时需具有此能力。 w：向用户授予针对对象的写入权限。 x：授予用户调用类方法（包括读取和写入）的能力，以及在监视器中执行auth 操作的能力。 *：授予用户对特定守护进程/存储池的读取、写入和执行权限，以及执行管理命令的能力 class-read：授予用户调用类读取方法的能力，属于是 x 能力的子集。 class-write：授予用户调用类写入方法的能力，属于是 x 能力的子集。
MON 能力：
包括 r/w/x 和 allow profile cap(ceph 的运行图)
例如： mon ‘allow rwx‘ mon ‘allow profile osd
OSD 能力:
包括 r、w、x、class-read、class-write(类读取)）和 profile osd(类写入)，另外 OSD 能力还允 许进行存储池和名称空间设置。
MDS 能力：
只需要 allow 或空都表示允许。 mds ‘allow
1.5 列出指定用户
#deploy节点
test@ceph-deploy:~/ceph-cluster$ ceph auth ls #列出所以用户
test@ceph-deploy:~/ceph-cluster$ ceph auth get osd.10 [osd.10] key = AQDNBilhkPDRKRAABW8mMaGrYMwYHVVVjtOU0g== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" exported keyring for osd.10 test@ceph-deploy:~/ceph-cluster$ ceph auth get osd #write keyring file with requested key test@ceph-deploy:~/ceph-cluster$ ceph auth get osd.10 [osd.10] key = AQDNBilhkPDRKRAABW8mMaGrYMwYHVVVjtOU0g== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" exported keyring for osd.10 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.admin [client.admin] key = AQD55h9h5ICUJBAAfk/2gBzkwU+G8bfqY023Yg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" exported keyring for client.admin
1.6 ceph用户管理
用户管理功能可让 Ceph 集群管理员能够直接在 Ceph 集群中创建、更新和删除用户。在 Ceph 集群中创建或删除用户时，可能需要将密钥分发到客户端，以便将密钥添加到密钥环文件中/etc/ceph/ceph.client.admin.keyring，此文件中可以包含一个或者多个用户认证信息，凡是拥有此文件的节点，将具备访问 ceph 的权限，而且可以使用其中任何一个账户的权限
1.61 列出用户
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph auth ls mds.ceph-mgr1 key: AQA5UyhhXsY/MBAAgv/L+/cKMPx4fy+V2Cm+vg== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx osd.0 key: AQAswSBh2jDUERAA+jfMZKocn+OjdFYZf7lrbg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQBjwSBhhYroNRAAO5+aqRxoaYGiMnI8FZegZw== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.10 key: AQDNBilhkPDRKRAABW8mMaGrYMwYHVVVjtOU0g== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.11 key: AQDfBilhKPzvGBAAVx7+GDBZlXkdRdLQM/qypw== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow *
1.62 用户管理
添加一个用户会创建用户名、密钥，以及包含在命令中用于创建该用户的所有能力,用户可使用其密钥向Ceph存储集群进行身份验证。用户的能力授予该用户在 Cephmonitor (mon)、Ceph OSD (osd) 或Ceph 元数据服务器 (mds) 上进行读取、写入或执行的能力
1.62.1添加用户
添加用户的规范方法：它会创建用户、生成密钥，并添加所有指定的能力
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph auth -h auth add <entity> [<caps>...] #添加认证 key test@ceph-deploy:~/ceph-cluster$ ceph auth add client.tom mon ‘allow r‘ osd ‘allow rwx pool=mypool‘ 0added key for client.tom #验证key test@ceph-deploy:~/ceph-cluster$ ceph auth get client.tom [client.tom] key = AQBvVipheB/5DhAAaABVJGZbBlneBJUNoWfowg== caps mon = "allow r" caps osd = "allow rwx pool=mypool" exported keyring for client.tom
1.62.2 ceph auth get-or-create
ceph auth get-or-create 此命令是创建用户较为常见的方式之一，它会返回包含用户名和密钥的密钥文，如果该用户已存在，此命令只以密钥文件格式返回用户名和密钥，还可以使用 -o 指定文件名选项将输出保存到某个文件
#创建用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create client.test mon ‘allow r‘ osd ‘allow rwx pool=mypool‘ [client.test] key = AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g== #验证用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.test [client.test] key = AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g== caps mon = "allow r" caps osd = "allow rwx pool=mypool" exported keyring for client.test #再次创建用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create client.test mon ‘allow r‘ osd ‘allow rwx pool=mypool‘ [client.test] key = AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g==
1.62.3 ceph auth get-or-create-key
此命令是创建用户并仅返回用户密钥，对于只需要密钥的客户端（例如 libvirt），此命令非常有用。如果该用户已存在，此命令只返回密钥。您可以使用 -o 文件名选项将输出保存到某个文件。
创建客户端用户时，可以创建不具有能力的用户。不具有能力的用户可以进行身份验证，但不能执行其他操作，此类客户端无法从监视器检索集群地图，但是，如果希望稍后再添加能力，可以使用 ceph auth caps 命令创建一个不具有能力的用户。
典型的用户至少对 Ceph monitor 具有读取功能，并对 Ceph OSD 具有读取和写入功能。此外，用户的 OSD 权限通常限制为只能访问特定的存储池
#用户有 key 就显示没有就创建 test@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create-key client.test mon ‘allow r‘ osd ‘allow rwx pool=mypool‘ AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g==
1.62.4 ceph auth print-key
#获取单个指定用户的key test@ceph-deploy:~/ceph-cluster$ ceph auth print-key client.test AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g==test
1.62.5 修改用户能力
使用 ceph auth caps 命令可以指定用户以及更改该用户的能力，设置新能力会完全覆盖当前的能力，因此要加上之前的用户已经拥有的能和新的能力，如果看当前能力，可以运行 cephauth get USERTYPE.USERID
#查看用户当前权限 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.test [client.test] key = AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g== caps mon = "allow r" caps osd = "allow rwx pool=mypool" exported keyring for client.test #修改权限 test@ceph-deploy:~/ceph-cluster$ ceph auth caps client.test mon ‘allow r‘ osd ‘allow rw pool=mypool‘ updated caps for client.test #验证权限 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.test [client.test] key = AQAYVyphyzZdGxAAYZlScsmbAf3mK9zyuaod6g== caps mon = "allow r" caps osd = "allow rw pool=mypool" exported keyring for client.test
1.62.6 删除用户
要删除用户使用 ceph auth del TYPE.ID，其中 TYPE 是 client、osd、mon 或 mds 之一，ID 是用户名或守护进程的 ID
test@ceph-deploy:~/ceph-cluster$ ceph auth del client.tom updated
1.7 秘钥环管理
ceph 的秘钥环是一个保存了 secrets、keys、certificates 并且能够让客户端通认证访问 ceph的 keyring file(集合文件)，一个 keyring file 可以保存一个或者多个认证信息，每一个 key 都有一个实体名称加权限，类型为：
{client、mon、mds、osd}.name
1.71 通过秘钥环文件备份与恢复用户
使用 ceph auth add 等命令添加的用户还需要额外使用 ceph-authtool 命令为其创建用户秘钥环文件
创建 keyring 文件命令格式：
ceph-authtool --create-keyring FILE
1.71.1 导出用户认证信息至 keyring 文件
将用户信息导出至 keyring 文件，对用户信息进行备份。
#deploy节点 #创建用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create client.user1 mon ‘allow r‘ osd ‘allow * pool=mypool‘ [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== #验证用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.user1 [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== caps mon = "allow r" caps osd = "allow * pool=mypool" exported keyring for client.user1 #创建keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph-authtool --create-keyring ceph.client.user1.keyring creating ceph.client.user1.keyring #验证 keyring 文件 test@ceph-deploy:~/ceph-cluster$ cat ceph.client.user1.keyring test@ceph-deploy:~/ceph-cluster$ file ceph.client.user1.keyring ceph.client.user1.keyring: empty #空文件 #导出 keyring 至指定文件 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.user1 -o ceph.client.user1.keyring exported keyring for client.user1 #验证指定用户的 keyring 文件 test@ceph-deploy:~/ceph-cluster$ cat ceph.client.user1.keyring [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== caps mon = "allow r" caps osd = "allow * pool=mypool"
1.71.2 从 keyring 文件恢复用户认证信息
可以使用 ceph auth import -i 指定 keyring 文件并导入到 ceph，起到用户备份和恢复的作用
#验证用户 test@ceph-deploy:~/ceph-cluster$ cat ceph.client.user1.keyring [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== caps mon = "allow r" caps osd = "allow * pool=mypool" #模拟误删用户 test@ceph-deploy:~/ceph-cluster$ ceph auth del client.user1 updated #验证用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.user1 Error ENOENT: failed to find client.user1 in keyring #导入用户 keyring test@ceph-deploy:~/ceph-cluster$ ceph auth import -i ceph.client.user1.keyring imported keyring #验证用户 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.user1 [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== caps mon = "allow r" caps osd = "allow * pool=mypool" exported keyring for client.user1
1.72 秘钥环文件多用户
一个 keyring 文件中可以包含多个不同用户的认证文件
1.72.1 将多用户导出至秘钥环
#创建空的keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph-authtool --create-keyring ceph.client.user.keyring creating ceph.client.user.keyring #把指定的 admin 用户的 keyring 文件内容导入到 user 用户的 keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph-authtool ./ceph.client.user.keyring --import-keyring ./ceph.client.admin.keyring importing contents of ./ceph.client.admin.keyring into ./ceph.client.user.keyring #验证 keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph-authtool -l ./ceph.client.user.keyring [client.admin] key = AQD55h9h5ICUJBAAfk/2gBzkwU+G8bfqY023Yg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" #再导入一个其他用户的 keyring test@ceph-deploy:~/ceph-cluster$ ceph-authtool ./ceph.client.user.keyring --import-keyring ./ceph.client.user1.keyring importing contents of ./ceph.client.user1.keyring into ./ceph.client.user.keyring #验证 keyring 文件是否包含多个用户的认证信息 test@ceph-deploy:~/ceph-cluster$ ceph-authtool -l ./ceph.client.user.keyring [client.admin] key = AQD55h9h5ICUJBAAfk/2gBzkwU+G8bfqY023Yg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.user1] key = AQB6WiphsylPERAALnVZ0wMPapQ0lb3ehDdrVA== caps mon = "allow r" caps osd = "allow * pool=mypool"
二： Ceph RBD 使用
2.1 RBD简介
Ceph 可以同时提供对象存储 RADOSGW、块存储 RBD、文件系统存储 Ceph FS,RBD 即 RADOSBlock Device 的简称，RBD 块存储是常用的存储类型之一，RBD 块设备类似磁盘可以被挂载，RBD 块设备具有快照、多副本、克隆和一致性等特性，数据以条带化的方式存储在 Ceph 集群的多个 OSD 中。
2.2 创建存储池
#deploy节点 #创建存储池 test@ceph-deploy:~/ceph-cluster$ ceph osd pool create rbd-data1 32 32 pool ‘rbd-data1‘ created #验证存储池 test@ceph-deploy:~/ceph-cluster$ ceph osd pool ls device_health_metrics mypool myrbd1 .rgw.root default.rgw.log default.rgw.control default.rgw.meta cephfs-metadata cephfs-data rbd-data1 #在存储池启用 rbd test@ceph-deploy:~/ceph-cluster$ ceph osd pool application enable -h osd pool application enable <pool> <app> [--yes-i-really-mean-it] test@ceph-deploy:~/ceph-cluster$ ceph osd pool application enable rbd-data1 rbd enabled application ‘rbd‘ on pool ‘rbd-data1‘ #初始化RBD test@ceph-deploy:~/ceph-cluster$ rbd pool init -p rbd-data1
2.3 创建 img 镜像
rbd 存储池并不能直接用于块设备，而是需要事先在其中按需创建映像（image），并把映像文件作为块设备使用。rbd 命令可用于创建、查看及删除块设备相在的映像（image），以及克隆映像、创建快照、将映像回滚到快照和查看快照等管理操作。
2.31 创建镜像
#deploy节点 #创建2个镜像 test@ceph-deploy:~/ceph-cluster$ rbd create data-img1 --size 3G --pool rbd-data1 --image-format 2 --image-feature layering test@ceph-deploy:~/ceph-cluster$ rbd create data-img2 --size 5G --pool rbd-data1 --image-format 2 --image-feature layering #验证镜像 test@ceph-deploy:~/ceph-cluster$ rbd ls --pool rbd-data1 data-img1 data-img2 #查看镜像信息 test@ceph-deploy:~/ceph-cluster$ rbd ls --pool rbd-data1 -l NAME SIZE PARENT FMT PROT LOCK data-img1 3 GiB 2 data-img2 5 GiB 2
2.32 查看镜像详细信息
#deploy节点 #查看data-img2的详细信息 test@ceph-deploy:~/ceph-cluster$ rbd --image data-img2 --pool rbd-data1 info rbd image ‘data-img2‘: size 5 GiB in 1280 objects order 22 (4 MiB objects) snapshot_count: 0 id: 12468e5b9a04b block_name_prefix: rbd_data.12468e5b9a04b format: 2 features: layering op_features: flags: create_timestamp: Sun Aug 29 00:08:51 2021 access_timestamp: Sun Aug 29 00:08:51 2021 modify_timestamp: Sun Aug 29 00:08:51 2021 #查看data-img1的详细信息 test@ceph-deploy:~/ceph-cluster$ rbd --image data-img1 --pool rbd-data1 info rbd image ‘data-img1‘: size 3 GiB in 768 objects order 22 (4 MiB objects) snapshot_count: 0 id: 1245f7ae95595 block_name_prefix: rbd_data.1245f7ae95595 format: 2 features: layering op_features: flags: create_timestamp: Sun Aug 29 00:08:41 2021 access_timestamp: Sun Aug 29 00:08:41 2021 modify_timestamp: Sun Aug 29 00:08:41 2021
2.33 以 json 格式显示镜像信息
#deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd ls --pool rbd-data1 -l --format json --pretty-format [ { "image": "data-img1", "id": "1245f7ae95595", "size": 3221225472, "format": 2 }, { "image": "data-img2", "id": "12468e5b9a04b", "size": 5368709120, "format": 2 } ]
2.34 镜像特性的启用
#deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd feature enable exclusive-lock --pool rbd-data1 --image data-img1 test@ceph-deploy:~/ceph-cluster$ rbd feature enable object-map --pool rbd-data1 --image data-img1 test@ceph-deploy:~/ceph-cluster$ rbd feature enable fast-diff --pool rbd-data1 --image data-img1 #验证镜像特性 test@ceph-deploy:~/ceph-cluster$ rbd --image data-img1 --pool rbd-data1 info rbd image ‘data-img1‘: size 3 GiB in 768 objects order 22 (4 MiB objects) snapshot_count: 0 id: 1245f7ae95595 block_name_prefix: rbd_data.1245f7ae95595 format: 2 features: layering, exclusive-lock, object-map, fast-diff op_features: flags: object map invalid, fast diff invalid create_timestamp: Sun Aug 29 00:08:41 2021 access_timestamp: Sun Aug 29 00:08:41 2021 modify_timestamp: Sun Aug 29 00:08:41 2021
2.35 镜像特性的禁用
#deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd feature disable fast-diff --pool rbd-data1 --image data-img1 test@ceph-deploy:~/ceph-cluster$ rbd --image data-img1 --pool rbd-data1 info rbd image ‘data-img1‘: size 3 GiB in 768 objects order 22 (4 MiB objects) snapshot_count: 0 id: 1245f7ae95595 block_name_prefix: rbd_data.1245f7ae95595 format: 2 features: layering, exclusive-lock #少了一个fast-diff 特性 op_features: flags: create_timestamp: Sun Aug 29 00:08:41 2021 access_timestamp: Sun Aug 29 00:08:41 2021 modify_timestamp: Sun Aug 29 00:08:41 2021
2.4 配置客户端使用 RBD
在 ubuntu 客户端挂载 RBD,并分别使用 admin 及普通用户挂载 RBD 并验证使用
2.41 客户端配置源
wget -q -O- ‘https://download.ceph.com/keys/release.asc‘ | sudo apt-key add - cat > /etc/apt/sources.list <<EOF # 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释 deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse EOF sudo echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main" >> /etc/apt/sources.list apt update
2.42 客户端安装 ceph-common
#client节点 root@ubuntu:~# apt install ceph-common
2.43 客户端使用 admin 账户挂载并使用 RBD
从部署服务器同步认证文件
#deploy节点
test@ceph-deploy:~/ceph-cluster$ scp ceph.conf ceph.client.admin.keyring root@10.0.0.200:/etc/ceph/
2.43.1 客户端映射镜像
#client节点 root@ubuntu:~# rbd -p rbd-data1 map data-img1 /dev/rbd0 root@ubuntu:~# rbd -p rbd-data1 map data-img2 /dev/rbd1
2.43.2 客户端验证镜像
#client节点 root@ubuntu:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 20G 0 disk └─sda1 8:1 0 20G 0 part / sr0 11:0 1 1024M 0 rom rbd0 252:0 0 3G 0 disk rbd1 252:16 0 5G 0 disk
2.43.3 客户端格式化磁盘并挂载使用
#client节点 #客户端格式化 rbd root@ubuntu:~# mkfs.xfs /dev/rbd0 meta-data=/dev/rbd0 isize=512 agcount=9, agsize=97280 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=786432, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 root@ubuntu:~# mkfs.xfs /dev/rbd1 meta-data=/dev/rbd1 isize=512 agcount=9, agsize=162816 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=1310720, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 #挂载 root@ubuntu:~# mkdir /data /data1 -p root@ubuntu:~# mount /dev/rbd0 /data root@ubuntu:~# mount /dev/rbd1 /data1 root@ubuntu:~# df -TH Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 1.1G 0 1.1G 0% /dev tmpfs tmpfs 207M 7.0M 200M 4% /run /dev/sda1 ext4 22G 3.0G 17G 15% / tmpfs tmpfs 1.1G 0 1.1G 0% /dev/shm tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs tmpfs 1.1G 0 1.1G 0% /sys/fs/cgroup tmpfs tmpfs 207M 0 207M 0% /run/user/1000 /dev/rbd0 xfs 3.3G 38M 3.2G 2% /data /dev/rbd1 xfs 5.4G 41M 5.4G 1% /data1
2.43.4 客户端验证写入数据
#client节点 root@ubuntu:~# sudo cp /var/log/syslog /data root@ubuntu:~# sudo cp /var/log/syslog /data1 root@ubuntu:~# df -h Filesystem Size Used Avail Use% Mounted on udev 964M 0 964M 0% /dev tmpfs 198M 6.7M 191M 4% /run /dev/sda1 20G 2.8G 16G 15% / tmpfs 986M 0 986M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 986M 0 986M 0% /sys/fs/cgroup tmpfs 198M 0 198M 0% /run/user/1000 /dev/rbd0 3.0G 38M 3.0G 2% /data /dev/rbd1 5.0G 40M 5.0G 1% /data1
2.43.5 验证 rbd 数据
#client节点 root@ubuntu:~# ll /data total 1160 drwxr-xr-x 2 root root 20 Aug 28 09:42 ./ drwxr-xr-x 24 root root 4096 Aug 28 09:38 ../ -rw-r----- 1 root root 1181490 Aug 28 09:42 syslog root@ubuntu:~# ll /data1 total 1160 drwxr-xr-x 2 root root 20 Aug 28 09:43 ./ drwxr-xr-x 24 root root 4096 Aug 28 09:38 ../ -rw-r----- 1 root root 1181490 Aug 28 09:43 syslog
2.43.6 查看存储池空间
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 240 GiB 239 GiB 861 MiB 861 MiB 0.35 TOTAL 240 GiB 239 GiB 861 MiB 861 MiB 0.35 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 76 GiB mypool 2 32 0 B 0 0 B 0 76 GiB myrbd1 3 64 12 MiB 18 35 MiB 0.02 76 GiB .rgw.root 4 32 1.3 KiB 4 48 KiB 0 76 GiB default.rgw.log 5 32 3.6 KiB 209 408 KiB 0 76 GiB default.rgw.control 6 32 0 B 8 0 B 0 76 GiB default.rgw.meta 7 8 0 B 0 0 B 0 76 GiB cephfs-metadata 8 32 56 KiB 22 254 KiB 0 76 GiB cephfs-data 9 64 121 MiB 31 363 MiB 0.16 76 GiB rbd-data1 10 32 23 MiB 32 69 MiB 0.03 76 GiB
2.44 客户端使用普通账户挂载并使用 RBD
测试客户端使用普通账户挂载并使用 RBD
2.44.1 创建普通账户并授权 (资源有限，使用了之前的虚拟机，可以新建一台client来做实验)
#deploy节点 #创建普通账户 test@ceph-deploy:~/ceph-cluster$ ceph auth add client.shijie mon ‘allow r‘ osd ‘allow rwx pool=rbd-data1‘ added key for client.shijie #验证用户信息 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.shijie [client.shijie] key = AQCAaCphzIAHMxAAddWTSYWGP6+lQuJV2OW/mQ== caps mon = "allow r" caps osd = "allow rwx pool=rbd-data1" exported keyring for client.shijie #创建 keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph-authtool --create-keyring ceph.client.shijie.keyring creating ceph.client.shijie.keyring #导出用户 keyring test@ceph-deploy:~/ceph-cluster$ ceph auth get client.shijie -o ceph.client.shijie.keyring exported keyring for client.shijie
2.44.2 安装 ceph 客户端
#ceph-client root@ceph-client:~# wget -q -O- ‘https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc‘ | sudo apt-key add - root@ceph-client:~# vim /etc/apt/sources.list root@ceph-client:~# apt install ceph-common
2.44.3 同步普通用户认证文件
#deploy节点 test@ceph-deploy:~/ceph-cluster$ scp ceph.conf ceph.client.admin.keyring root@10.0.0.200:/etc/ceph/
2.44.4 在客户端验证权限
#ceph-client root@ceph-client:~# ll /etc/ceph/ total 20 drwxr-xr-x 2 root root 4096 Aug 28 09:56 ./ drwxr-xr-x 81 root root 4096 Aug 28 09:51 ../ -rw-r--r-- 1 root root 125 Aug 28 09:47 ceph.client.shijie.keyring -rw-r--r-- 1 root root 261 Aug 20 10:11 ceph.conf -rw-r--r-- 1 root root 92 Jun 7 07:39 rbdmap #默认使用 admin 账户 root@ceph-client:~# # ceph --user shijie -s
2.44.5 映射 rbd
#ceph-client节点 #映射 rbd root@ceph-client:~# rbd --user shijie -p rbd-data1 map data-img2 /dev/rbd2 #验证rbd root@ceph-client:~# fdisk -l /dev/rbd0 Disk /dev/rbd0: 3 GiB, 3221225472 bytes, 6291456 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
2.44.6 格式化并使用 rbd 镜像
#ceph-client节点 root@ceph-client:~# mkfs.ext4 /dev/rbd2 mke2fs 1.44.1 (24-Mar-2018) /dev/rbd2 contains a xfs file system Proceed anyway? (y,N) y Discarding device blocks: done Creating filesystem with 1310720 4k blocks and 327680 inodes Filesystem UUID: fb498e3f-e8cb-40dd-b10d-1e91e0bfbbed Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done root@ceph-client:~# mkdir /data2 root@ceph-client:~# mount /dev/rbd2 /data2/ root@ceph-client:~# # cp /var/log/messages /data2/ root@ceph-client:~# ll /data2 total 24 drwxr-xr-x 3 root root 4096 Aug 28 10:00 ./ drwxr-xr-x 25 root root 4096 Aug 28 10:01 ../ drwx------ 2 root root 16384 Aug 28 10:00 lost+found/ root@ceph-client:~# df -TH Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 1.1G 0 1.1G 0% /dev tmpfs tmpfs 207M 7.0M 200M 4% /run /dev/sda1 ext4 22G 3.0G 17G 15% / tmpfs tmpfs 1.1G 0 1.1G 0% /dev/shm tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs tmpfs 1.1G 0 1.1G 0% /sys/fs/cgroup /dev/rbd0 xfs 3.3G 39M 3.2G 2% /data /dev/rbd1 xfs 5.4G 42M 5.4G 1% /data1 tmpfs tmpfs 207M 0 207M 0% /run/user/1000 /dev/rbd2 ext4 5.3G 21M 5.0G 1% /data2 #deploy节点 #管理端验证镜像状态 test@ceph-deploy:~/ceph-cluster$ rbd ls -p rbd-data1 -l NAME SIZE PARENT FMT PROT LOCK data-img1 3 GiB 2 excl data-img2 5 GiB 2
2.44.7 验证 ceph 内核模块
挂载 rbd 之后系统内核会自动加载 libceph.ko 模块
#client节点 root@ceph-client:~# lsmod|grep ceph libceph 315392 1 rbd libcrc32c 16384 2 xfs,libceph root@ceph-client:~# modinfo libceph filename: /lib/modules/4.15.0-112-generic/kernel/net/ceph/libceph.ko license: GPL description: Ceph core library author: Patience Warnick <patience@newdream.net> author: Yehuda Sadeh <yehuda@hq.newdream.net> author: Sage Weil <sage@newdream.net> srcversion: 899059C79545E4ADF47A464 depends: libcrc32c retpoline: Y intree: Y name: libceph vermagic: 4.15.0-112-generic SMP mod_unload signat: PKCS#7 signer: sig_key: sig_hashalgo: md4
2.44.8 rbd 镜像空间拉伸
#deploy节点 #当前 rbd 镜像空间大小 test@ceph-deploy:~/ceph-cluster$ rbd ls -p rbd-data1 -l NAME SIZE PARENT FMT PROT LOCK data-img1 3 GiB 2 excl data-img2 5 GiB 2 #拉伸 rbd 镜像空间 test@ceph-deploy:~/ceph-cluster$ rbd resize --pool rbd-data1 --image data-img2 --size 8G Resizing image: 100% complete...done. test@ceph-deploy:~/ceph-cluster$ rbd resize --pool rbd-data1 --image data-img1 --size 6G Resizing image: 100% complete...done. #验证rgb信息 test@ceph-deploy:~/ceph-cluster$ rbd ls -p rbd-data1 -l NAME SIZE PARENT FMT PROT LOCK data-img1 6 GiB 2 data-img2 8 GiB 2
2.44.9 客户端验证镜像空间
#client节点 root@ceph-client:~# fdisk -l /dev/rbd2 Disk /dev/rbd2: 8 GiB, 8589934592 bytes, 16777216 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
2.44.10 开机自动挂载
#client节点 root@ceph-client:~# cat /etc/fstab rbd --user shijie -p rbd-data1 map data-img2 mount /dev/rbd2 /data2/ root@ceph-client:~# chmod a+x /etc/fstab root@ceph-client:~# reboot #查看映射 root@ceph-client:~# rbd showmapped id pool image snap device 0 rbd-data1 data-img2 - /dev/rbd2
2.44.11 卸载 rbd 镜像
#client节点
root@ceph-client:~# umount /data2
root@ceph-client:~# umount /data2 rbd --user shijie -p rbd-data1 unmap data-img2
2.44.12 删除 rbd 镜像
删除存储池 rbd -data1 中的 data-img1 镜像
#deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd rm --pool rbd-data1 --image data-img1 Removing image: 100% complete...done.
2.44.13 rbd 镜像回收站机制
删除的镜像数据无法恢复，但是还有另外一种方法可以先把镜像移动到回收站，后期确认删除的时候再从回收站删除即可。
#deploy节点 #查看镜像状态 test@ceph-deploy:~/ceph-cluster$ rbd status --pool rbd-data1 --image data-img2 #将进行移动到回收站 test@ceph-deploy:~/ceph-cluster$ rbd trash move --pool rbd-data1 --image data-img2 #查看回收站的镜像 test@ceph-deploy:~/ceph-cluster$ rbd trash list --pool rbd-data1 12468e5b9a04b data-img2 #从回收站删除镜像 如果镜像不再使用，可以直接使用 trash remove 将其从回收站删除 #还原镜像 test@ceph-deploy:~/ceph-cluster$ rbd trash restore --pool rbd-data1 --image data-img2 --image-id 12468e5b9a04b #验证镜像 test@ceph-deploy:~/ceph-cluster$ rbd ls --pool rbd-data1 -l NAME SIZE PARENT FMT PROT LOCK data-img2 8 GiB 2
2.5 镜像快照
2.51 客户端当前数据
#client节点 root@ceph-client:~# ll /data2 total 24 drwxr-xr-x 3 root root 4096 Aug 28 10:00 ./ drwxr-xr-x 25 root root 4096 Aug 28 10:01 ../ drwx------ 2 root root 16384 Aug 28 10:00 lost+found/
2.52 创建并验证快照
#deploy节点 #创建快照 test@ceph-deploy:~/ceph-cluster$ rbd snap create --pool rbd-data1 --image data-img2 --snap img2-snap-12468e5b9a04b Creating snap: 100% complete...done. #验证快照 test@ceph-deploy:~/ceph-cluster$ rbd snap list --pool rbd-data1 --image data-img2 SNAPID NAME SIZE PROTECTED TIMESTAMP 4 img2-snap-12468e5b9a04b 8 GiB Sun Aug 29 01:41:32 2021
2.53 删除数据并还原快照
#客户端删除数据 root@ceph-client:~# rm -rf /data2/lost+found #验证数据 root@ceph-client:~# ll /data2 total 8 drwxr-xr-x 2 root root 4096 Aug 28 10:01 ./ drwxr-xr-x 25 root root 4096 Aug 28 10:01 ../ #卸载 rbd root@ceph-client:~# umount /data2 root@ceph-client:~# rbd unmap /dev/rbd2 #回滚快照 #deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd snap rollback --pool rbd-data1 --image data-img2 --snap img2-snap-12468e5b9a04b
2.54 客户端验证数据
#client节点 #客户端映射 rbd root@ceph-client:~# rbd --user shijie -p rbd-data1 map data-img2 #客户端挂载 rbd root@ceph-client:~# mount /dev/rbd0 /data/ #客户端验证数据 root@ceph-client:~# ll /data/
2.55 删除快照
#deploy节点 test@ceph-deploy:~/ceph-cluster$ rbd snap remove --pool rbd-data1 --image data-img2 --snap img2-snap-12468e5b9a04b Removing snap: 100% complete...done. #验证快照是否删除 test@ceph-deploy:~/ceph-cluster$ rbd snap list --pool rbd-data1 --image data-img2
2.56 快照数量限制
#deploy节点 #设置与修改快照数量限制 test@ceph-deploy:~/ceph-cluster$ rbd snap limit set --pool rbd-data1 --image data-img2 --limit 30 #清除快照数量限制 test@ceph-deploy:~/ceph-cluster$ rbd snap limit clear --pool rbd-data1 --image data-img2
3 CephFS 使用
ceph FS 即 ceph filesystem，可以实现文件系统共享功能,客户端通过 ceph 协议挂载并使用
3.1 部署 MDS 服务
#mgr节点 test@ceph-mgr1:~$ apt-cache madison ceph-mds ceph-mds | 16.2.5-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages ceph-mds | 12.2.13-0ubuntu0.18.04.8 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/universe amd64 Packages ceph-mds | 12.2.13-0ubuntu0.18.04.4 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/universe amd64 Packages ceph-mds | 12.2.4-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages test@ceph-mgr1:~$ sudo apt install ceph-mds
3.2 创建 CephFS metadata 和 data 存储池
使用 CephFS 之前需要事先于集群中创建一个文件系统，并为其分别指定元数据和数据相关的存储池
#deploy节点 test@ceph-deploy:~$ ceph osd pool create cephfs-metadata 32 32 test@ceph-deploy:~$ ceph osd pool create cephfs-data 64 64 test@ceph-deploy:~$ ceph -s cluster: id: 635d9577-7341-4085-90ff-cb584029a1ea health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 7m) mgr: ceph-mgr2(active, since 6m), standbys: ceph-mgr1 mds: 1/1 daemons up osd: 12 osds: 12 up (since 6m), 12 in (since 39h) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 329 pgs objects: 328 objects, 213 MiB usage: 894 MiB used, 239 GiB / 240 GiB avail pgs: 329 active+clean
3.3 创建 cephFS 并验证
#deploy节点 test@ceph-deploy:~$ ceph fs new mycephfs cephfs-metadata cephfs-data test@ceph-deploy:~$ ceph fs ls name: mycephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ] test@ceph-deploy:~$ ceph fs status mycephfs mycephfs - 0 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 12 15 12 0 POOL TYPE USED AVAIL cephfs-metadata metadata 247k 75.5G cephfs-data data 362M 75.5G MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
3.4 验证 cepfFS 服务状态
#deploy节点 test@ceph-deploy:~$ ceph mds stat mycephfs:1 {0=ceph-mgr1=up:active}
3.5 创建客户端账户
#deploy节点 #创建用户 test@ceph-deploy:~/ceph-cluster$ ceph auth add client.yanyan mon ‘allow r‘ mds ‘allow rw‘ osd ‘allow rwx pool=cephfs-data‘ added key for client.yanyan #验证账户 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.yanyan [client.yanyan] key = AQAhMCth/3d/HxAA7sMakmCr5tOFj8l2vmmaRA== caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rwx pool=cephfs-data" exported keyring for client.yanyan #创建keyring 文件 test@ceph-deploy:~/ceph-cluster$ ceph auth get client.yanyan -o ceph.client.yanyan.keyring exported keyring for client.yanyan #创建 key 文件 test@ceph-deploy:~/ceph-cluster$ ceph auth print-key client.yanyan > yanyan.key #验证用户的 keyring 文件 test@ceph-deploy:~/ceph-cluster$ cat ceph.client.yanyan.keyring [client.yanyan] key = AQAhMCth/3d/HxAA7sMakmCr5tOFj8l2vmmaRA== caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rwx pool=cephfs-data"
3.6 安装 ceph 客户端
#client节点 root@ceph-client:/etc/ceph# apt install ceph-common -y
3.7 同步客户端认证文件
#deploy节点 test@ceph-deploy:~/ceph-cluster$ scp ceph.conf ceph.client.yanyan.keyring yanyan.key root@10.0.0.200:/etc/ceph/
3.8 客户端验证权限
#client节点 root@ceph-client2:/etc/ceph# ceph --user yanyan -s cluster: id: 635d9577-7341-4085-90ff-cb584029a1ea health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 55m) mgr: ceph-mgr2(active, since 54m), standbys: ceph-mgr1 mds: 1/1 daemons up osd: 12 osds: 12 up (since 54m), 12 in (since 39h) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 329 pgs objects: 328 objects, 213 MiB usage: 895 MiB used, 239 GiB / 240 GiB avail pgs: 329 active+clean
3.9 内核空间挂载 ceph-fs
客户端挂载有两种方式，一是内核空间一是用户空间，内核空间挂载需要内核支持 ceph 模块，用户空间挂载需要安装 ceph-fuse
3.91 客户端通过 key 文件挂载
#deploy节点 root@ceph-client2:~# mount -t ceph 10.0.0.101:6789,10.0.0.102:6789,10.0.0.103:6789:/ /data -o name=yanyan,secretfile=/etc/ceph/yanyan.key root@ceph-client2:~# df -h Filesystem Size Used Avail Use% Mounted on udev 964M 0 964M 0% /dev tmpfs 198M 6.6M 191M 4% /run /dev/sda1 20G 2.8G 16G 16% / tmpfs 986M 0 986M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 986M 0 986M 0% /sys/fs/cgroup tmpfs 198M 0 198M 0% /run/user/1000 10.0.0.101:6789,10.0.0.102:6789,10.0.0.103:6789:/ 76G 120M 76G 1% /data #验证写入数据 root@ceph-client2:~# cp /var/log/syslog /data/ root@ceph-client2:~# dd if=/dev/zero of=/data/testfile bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.0415206 s, 2.5 GB/s
3.92 客户端通过 key 挂载
#client节点 root@ceph-client2:~# tail /etc/ceph/yanyan.key AQAhMCth/3d/HxAA7sMakmCr5tOFj8l2vmmaRA== root@ceph-client2:~# umount /data/ root@ceph-client2:~# mount -t ceph 10.0.0.101:6789,10.0.0.102:6789,10.0.0.103:6789:/ /data -o name=yanyan,secret=AQAhMCth/3d/HxAA7sMakmCr5tOFj8l2vmmaRA== root@ceph-client2:~# df -h Filesystem Size Used Avail Use% Mounted on udev 964M 0 964M 0% /dev tmpfs 198M 6.6M 191M 4% /run /dev/sda1 20G 2.8G 16G 16% / tmpfs 986M 0 986M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 986M 0 986M 0% /sys/fs/cgroup tmpfs 198M 0 198M 0% /run/user/1000 10.0.0.101:6789,10.0.0.102:6789,10.0.0.103:6789:/ 76G 220M 76G 1% /data #测试写入数据 root@ceph-client2:~# cp /var/log/syslog /data/ #查看挂载点状态 root@ceph-client2:~# stat -f /data/ File: "/data/" ID: 2f5ea2f36fe16833 Namelen: 255 Type: ceph Block size: 4194304 Fundamental block size: 4194304 Blocks: Total: 19319 Free: 19264 Available: 19264 Inodes: Total: 56 Free: -1
3.93 开机挂载
#client节点 root@ceph-client2:~# cat /etc/fstab 10.0.0.101:6789,10.0.0.102:6789,10.0.0.103:6789:/ /data ceph defaults,name=yanyan,secretfile=/etc/ceph/yanyan.key,_netdev 0 0 root@ceph-client2:~# mount -a
3.94 客户端模块
客户端内核加载 ceph.ko 模块挂载 cephfs 文件系统
#client节点 root@ceph-client2:~# lsmod|grep ceph ceph 376832 1 libceph 315392 1 ceph libcrc32c 16384 1 libceph fscache 65536 1 ceph root@ceph-client2:~# madinfo ceph
3.10 ceph mds 高可用
Ceph mds(etadata service)作为 ceph 的访问入口，需要实现高性能及数据备份，假设启动 4个 MDS 进程，设置 2 个 Rank。这时候有 2 个 MDS 进程会分配给两个 Rank，还剩下 2 个 MDS进程分别作为另外个的备份。
3.10.1 当前 mds 服务器状态
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph mds stat mycephfs:1 {0=ceph-mgr1=up:active}
3.10.2 添加 MDS 服务器
将 ceph-mgr2 和 ceph-mon2 和 ceph-mon3 作为 mds 服务角色添加至 ceph 集群，最后实两主两备的 mds 高可用和高性能结构。
#mds 服务器安装 ceph-mds 服务 test@ceph-mgr2:~$ sudo apt install ceph-mds -y test@ceph-mon2:~$ sudo apt install ceph-mds -y test@ceph-mon3:~$ sudo apt install ceph-mds -y #添加 mds 服务器 test@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mgr2 test@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mon2 test@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mon3 #验证 mds 服务器当前状态： test@ceph-deploy:~/ceph-cluster$ ceph mds stat mycephfs:1 {0=ceph-mgr1=up:active} 3 up:standby
3.10.3 验证 ceph 集群当前状态
当前处于激活状态的 mds 服务器有一台，处于备份状态的 mds 服务器有三台
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs - 1 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 13 16 12 2 POOL TYPE USED AVAIL cephfs-metadata metadata 379k 75.2G cephfs-data data 663M 75.2G STANDBY MDS ceph-mon2 ceph-mgr2 ceph-mon3 MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
3.10.4 当前的文件系统状态
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph fs get mycephfs Filesystem ‘mycephfs‘ (1) fs_name mycephfs epoch 37 flags 12 created 2021-08-27T11:06:31.193582+0800 modified 2021-08-29T14:48:37.814878+0800 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 551 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=84172} failed damaged stopped data_pools [9] metadata_pool 8 inline_data disabled balancer standby_count_wanted 1 [mds.ceph-mgr1{0:84172} state up:active seq 7 addr [v2:10.0.0.104:6800/3031657167,v1:10.0.0.104:6801/3031657167]]
3.10.5 设置处于激活状态 mds 的数量
目前有四个 mds 服务器，但是有一个主三个备，可以优化一下部署架构，设置为为两主两备
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph fs set mycephfs max_mds 2#设置同时活跃的主 mds 最大值为 2 test@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs - 1 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 13 16 12 2 1 active ceph-mon3 Reqs: 0 /s 10 13 11 0 POOL TYPE USED AVAIL cephfs-metadata metadata 451k 75.2G cephfs-data data 663M 75.2G STANDBY MDS ceph-mon2 ceph-mgr2 MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
3.10.6 MDS 高可用优化
目前的状态是 ceph-mgr1 和 ceph-mon2 分别是 active 状态，ceph-mon3 和 ceph-mgr2 分别处于 standby 状态，现在可以将 ceph-mgr2 设置为 ceph-mgr1 的 standby，将 ceph-mon3 设置为 ceph-mon2 的 standby，以实现每个主都有一个固定备份角色的结构，
#deploy节点 test@ceph-deploy:~/ceph-cluster$ cat ceph.conf [global] fsid = 635d9577-7341-4085-90ff-cb584029a1ea public_network = 10.0.0.0/24 cluster_network = 192.168.133.0/24 mon_initial_members = ceph-mon1 mon_host = 10.0.0.101 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx mon clock drift allowed = 2 mon clock drift warn backoff = 30 [mds.ceph-mgr2] #mds_standby_for_fscid = mycephfs mds_standby_for_name = ceph-mgr1 mds_standby_replay = true [mds.ceph-mon3] mds_standby_for_name = ceph-mon2 mds_standby_replay = true
3.10.7 分发配置文件并重启 mds 服务
#deploy节点 #分发配置文件保证各 mds 服务重启有效 test@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf config push ceph-mon3 test@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf config push ceph-mon2 test@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf config push ceph-mgr1 test@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf config push ceph-mgr2 test@ceph-mon2:~$ sudo systemctl restart ceph-mds@ceph-mon2.service test@ceph-mon3:~$ sudo systemctl restart ceph-mds@ceph-mon3.service test@ceph-mgr1:~$ sudo systemctl restart ceph-mds@ceph-mgr1.service test@ceph-mgr2:~$ sudo systemctl restart ceph-mds@ceph-mgr2.service
3.10.8 ceph 集群 mds 高可用状态
#deploy节点 test@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs - 1 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr2 Reqs: 0 /s 13 16 12 1 1 active ceph-mon2 Reqs: 0 /s 10 13 11 0 POOL TYPE USED AVAIL cephfs-metadata metadata 451k 75.2G cephfs-data data 663M 75.2G STANDBY MDS ceph-mon3 ceph-mgr1 MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) #查看 active 和 standby 对应关系 test@ceph-deploy:~/ceph-cluster$ ceph fs get mycephfs Filesystem ‘mycephfs‘ (1) fs_name mycephfs epoch 67 flags 12 created 2021-08-27T11:06:31.193582+0800 modified 2021-08-29T16:34:16.305266+0800 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 557 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=84753,1=84331} failed damaged stopped data_pools [9] metadata_pool 8 inline_data disabled balancer standby_count_wanted 1 [mds.ceph-mgr2{0:84753} state up:active seq 7 addr [v2:10.0.0.105:6802/2338760756,v1:10.0.0.105:6803/2338760756]] [mds.ceph-mon2{1:84331} state up:active seq 14 addr [v2:10.0.0.102:6800/3841027813,v1:10.0.0.102:6801/3841027813]]
3.11 通过 ganesha 将 cephfs 导出为 NFS
通过 ganesha 将 cephfs 通过 NFS 协议共享使用
3.11.1 服务端配置
#mgr1节点 test@ceph-mgr1:~$ sudo apt install nfs-ganesha-ceph test@ceph-mgr1:~$ cd /etc/ganesha/ test@ceph-mgr1:/etc/ganesha$ cat /etc/ganesha/ganesha.conf NFS_CORE_PARAM { #disable NLM Enable_NLM = false; # disable RQUOTA (not suported on CephFS) Enable_RQUOTA = false; # NFS protocol Protocols = 4; } EXPORT_DEFAULTS { # default access mode Access_Type = RW; } EXPORT { # uniq ID Export_Id = 1; # mount path of CephFS Path = "/"; FSAL { name = CEPH; # hostname or IP address of this Node hostname="10.0.0.104"; } # setting for root Squash Squash="No_root_squash"; # NFSv4 Pseudo path Pseudo="/test"; # allowed security options SecType = "sys"; } LOG { # default log level Default_Log_Level = WARN; } test@ceph-mgr1:/etc/ganesha$ sudo systemctl restart nfs-ganesha