准备环境
| 主机名 | 业务IP | 集群IP | 磁盘规划 | 功能介绍 |
| ceph-deploy | 172.16.143.110 | 172.16.138.110 | 200/1 | 管理Ceph集群 |
| ceph-mon1 | 172.16.143.111 | 172.16.138.131 | 200/1 | 监控Ceph集群状态，拓扑等信息 |
| ceph-mon2 | 172.16.143.112 | 172.16.138.132 | 200/1 | 监控Ceph集群状态，拓扑等信息 |
| ceph-mon3 | 172.16.143.113 | 172.16.138.113 | 200/1 | 监控Ceph集群状态，拓扑等信息 |
| ceph-mgr1 | 172.16.143.114 | 172.16.138.114 | 200/1 | Ceph 集群的管理，实现统一接口 |
| ceph-mgr2 | 172.16.143.115 | 172.16.138.115 | 200/1 | Ceph 集群的管理，实现统一接口 |
| ceph-node1 | 172.16.143.116 | 172.16.138.116 | 200/1 100/2 | Ceph 集群的存储节点 |
| ceph-node2 | 172.16.143.117 | 172.16.138.117 | 200/1 100/2 | Ceph 集群的存储节点 |
| ceph-node3 | 172.16.143.118 | 172.16.138.118 | 200/1 100/2 | Ceph 集群的存储节点 |
| ceph-node4 | 172.16.143.119 | 172.16.138.119 | 200/1 100/2 | Ceph 集群的存储节点 |
- 配置2张网卡，一个是业务访问使用，一个是集群内部访问使用
- 关闭防火墙
- 关闭selinux
- 配置时间同步
- 配置hosts
配置Ceph的源
wget -q -O- ‘https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc‘ | sudo apt-key add - sudo apt-add-repository ‘deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific/ bionic main‘ sudo apt update
配置ceph用户
groupadd -r -g 2022 ceph && useradd -r -m -s /bin/bash -u 2022 -g 2022 ceph && echo ceph:123456 | chpasswd echo "ceph ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
配置ceph-deploy节点免密登录
ssh-keygen -t rsa ssh-copy-id -i .ssh/id_rsa.pub ceph@172.16.143.110 ... ... ssh-copy-id -i .ssh/id_rsa.pub ceph@172.16.143.119
安装Ceph的部署工具ceph-deploy
sudo apt install ceph-deploy
安装Ceph
初始化mon
mkdir ceph-cluster cd ceph-cluster
生成配置文件(ceph-deploy节点)
$ ceph-deploy new --cluster-network 172.16.138.0/24 --public-network 172.16.143.0/24 ceph-mon1 $ ls ceph.conf ceph-deploy-ceph.log ceph.mon.keyring
安装ceph-mon(ceph-moin)
$ sudo apt install ceph-mon -y
回车
回车
初始化mon
ceph-deploy mon create-initial
验证mon进程是否启动
$ ps -ef |grep mon message+ 757 1 0 04:40 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 802 1 0 04:40 ? 00:00:00 /usr/lib/accountsservice/accounts-daemon daemon 832 1 0 04:40 ? 00:00:00 /usr/sbin/atd -f ceph 6739 1 0 05:14 ? 00:00:00 /usr/bin/ceph-mon -f --cluster ceph --id ceph-mon1 --setuser ceph --setgroup ceph root 7343 1566 0 05:16 pts/0 00:00:00 grep --color=auto mon
安装ceph-common及分发ceph-admin管理秘钥
//root用户执行 $ apt install ceph-common -y $ ceph-deploy admin ceph-deploy
$ sudo chown ceph:ceph /etc/ceph/ceph.client.admin.keyring
$ ceph -s
cluster:
id: 6e278817-8019-4a06-82b3-b4d24d7dd743
health: HEALTH_WARN
mon is allowing insecure global_id reclaim
services:
mon: 1 daemons, quorum ceph-mon1 (age 60m)
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
安装ceph-mgr
mgr节点执行
apt install ceph-mgr -y
deploy节点执行
ceph-deploy mgr create ceph-mgr1
安装ceph-node
deploy节点执行
//分别执行1,2,3,4节点 ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node4
擦除远端磁盘（node节点的都要执行，sdb sdc）
ceph-deploy disk zap ceph-node1 /dev/sdb
添加主机OSD（node节点的所有磁盘都要执行）
ceph-deploy osd create ceph-node2 --data /dev/sdb
查看集群状态
$ ceph -s cluster: id: 6e278817-8019-4a06-82b3-b4d24d7dd743 health: HEALTH_OK services: mon: 1 daemons, quorum ceph-mon1 (age 3d) mgr: ceph-mgr1(active, since 3d) osd: 8 osds: 8 up (since 9s), 8 in (since 17s) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 45 MiB used, 800 GiB / 800 GiB avail pgs: 1 active+clean
扩展mon服务
mon节点2/3 安装ceph-mon
$ apt install ceph-mon
添加mon
$ ceph-deploy mon add ceph-mon3
$ ceph -s cluster: id: 6e278817-8019-4a06-82b3-b4d24d7dd743 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 29s) mgr: ceph-mgr1(active, since 3d) osd: 8 osds: 8 up (since 2h), 8 in (since 2h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 45 MiB used, 800 GiB / 800 GiB avail pgs: 1 active+clean
查看mon状态
$ ceph quorum_status --format json-pretty
扩展mgr
mgr节点安装mgr
$ apt install ceph-mgr
添加mgr
$ ceph-deploy mgr create ceph-mgr2
查看状态
$ ceph -s cluster: id: 6e278817-8019-4a06-82b3-b4d24d7dd743 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 5m) mgr: ceph-mgr1(active, since 3d), standbys: ceph-mgr2 osd: 8 osds: 8 up (since 2h), 8 in (since 2h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 45 MiB used, 800 GiB / 800 GiB avail pgs: 1 active+clean