本次实验演示如何快速地在centos上部署ceph分布式存储集群。目标是在ceph-node1和ceph-node2上创建一个两节点的集群。

---

环境

硬件环境

主机名	IP	功能	备注
ceph-node1	192.168.1.120	deploy，mon＊1，osd*3
ceph-node2	192.168.1.121	deploy，mon＊1，osd*3
ceph-node3	192.168.1.122	横向扩展节点
cloud	192.168.1.102	openstack ocata
test	192.168.1.123	Openstack测试环境，Rally，Sharker

软件环境

操作系统：Centos 7.3
Openstack：Ocata
Ceph：Jewel

---

安装Ceph

准备repo

在所有ceph节点上准备下面的源。
 yum clean all
 rm -rf /etc/yum.repos.d/*.repo
 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
 wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
 sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo
 sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo
 sed -i 's/$releasever/7/g' /etc/yum.repos.d/CentOS-Base.repo

 #vi /etc/yum.repos.d/ceph.repo
 [ceph]
 name=ceph
 baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/x86_64/
 gpgcheck=0
 [ceph-noarch]
 name=cephnoarch
 baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/
 gpgcheck=0

 yum update -y

---

操作系统配置

    1:无密码访问配置
      在deploy节点(ceph-node1)上执行。配置deploy节点和其他ceph节点之间的无密码访问。
      sudo su -
      ssh-keygen
      ssh-copy-id root@ceph-node2
      ssh-copy-id root@cloud

    2:启用Ceph monitor OSD端口，在所有ceph节点上执行
      firewall-cmd --zone=public --add-port=6789/tcp --permanent
      firewall-cmd --zone=public --add-port=6800-7100/tcp --permanent                                              
      firewall-cmd --reload 
      firewall-cmd --zone=public --list-all

    3:禁用Selinux，在所有ceph节点上执行
      setenforce 0

    4:安装ntp，在所有ceph节点上执行
      yum install ntp ntpdate -y
      systemctl restart ntpdate.service
      systemctl restart ntpd.service
      systemctl enable ntpd.service ntpdate.service

---

部署Ceph集群

    1:安装ceph-deploy
      yum install ceph-deploy -y
    
    2:用Ceph-deploy创建Ceph集群
      mkdir /etc/ceph
      cd /etc/ceph
      ceph-deploy new ceph-node1
      生成一个新的ceph集群，集群包括ceph配置文件以及monitor的密钥环。

    3:安装ceph二进制软件包
      ceph-deploy install --no-adjust-repos ceph-node1 ceph-node2

    4:修改ceph 配置文件
      [global]
      fsid = 7bac6963-0e1d-4cea-9e2e-f02bbae96ba7
      mon_initial_members = ceph-node1
      mon_host = 192.168.1.101
      auth_cluster_required = cephx
      auth_service_required = cephx
      auth_client_required = cephx
      public network = 192.168.1.0/24
 
    5:在ceph-node1上创建第一个ceph monitor
      ceph-deploy mon create-initial

    6:在ceph-node1上创建OSD
      ceph-deploy disk list ceph-node1(列出disk）
      ceph-deploy disk zap ceph-node1:sdb ceph-node1:sdc ceph-node1:sdd
      ceph-deploy osd create ceph-node1:sdb ceph-node1:sdc ceph-node1:sdd
    
    7:用Ceph-deploy在ceph-node2上创建monitor 
      ceph-deploy mon create ceph-node2
      ceph –s
      ceph mon stat
    8:在ceph-node2上创建OSD
      ceph-deploy disk zap ceph-node2:sdb ceph-node2:sdc ceph-node2:sdd
      ceph-deploy osd create ceph-node2:sdb ceph-node2:sdc ceph-node2:sdd
    9:调整rbd存储池的pg_num和pgp_num
      ceph osd pool set rbd pg_num 256
      ceph osd pool set rbd pgp_num 256

总结

通过上面的步骤，一个all in one的ceph就成功部署了。

ceph -s

参考资料

http://docs.ceph.org.cn/rados/deployment/ceph-deploy-new/