openstack 管理二十四 - ceph 与 vm 连接测试记录

目的

测试 ceph 与 vm 连接与使用

创建 vm

主机 128030 及 129094 是全新安装并利用 puppet 推送的 nova compute 主机

计划在这两个主机上进行 vm 连接 ceph 测试

nova boot --flavor b2c_web_1core --image Centos6.3_1.3 --security_group default --nic net-id=9106aee4-2dc0-4a6d-a789-10c53e2b88c1 ceph-test01.sh.vclound.com --availability-zone nova:sh-compute-128030.sh.vclound.com
+--------------------------------------+------------------------------------------------------+
| Property                             | Value                                                |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                               |
| OS-EXT-AZ:availability_zone          | nova                                                 |
| OS-EXT-SRV-ATTR:host                 | -                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000020d                                    |
| OS-EXT-STS:power_state               | 0                                                    |
| OS-EXT-STS:task_state                | scheduling                                           |
| OS-EXT-STS:vm_state                  | building                                             |
| OS-SRV-USG:launched_at               | -                                                    |
| OS-SRV-USG:terminated_at             | -                                                    |
| accessIPv4                           |                                                      |
| accessIPv6                           |                                                      |
| adminPass                            | 5DCHoj8ihwN6                                         |
| config_drive                         |                                                      |
| created                              | 2015-06-25T03:49:14Z                                 |
| flavor                               | b2c_web_1core (5)                                    |
| hostId                               |                                                      |
| id                                   | 99d37977-a13a-4a8b-b8b1-e613a4959623                 |
| image                                | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name                             | -                                                    |
| metadata                             | {}                                                   |
| name                                 | ceph-test01.sh.vclound.com                           |
| os-extended-volumes:volumes_attached | []                                                   |
| progress                             | 0                                                    |
| security_groups                      | default                                              |
| status                               | BUILD                                                |
| tenant_id                            | bb0b51d166254dc99bc7462c0ac002ff                     |
| updated                              | 2015-06-25T03:49:14Z                                 |
| user_id                              | 226e71f1c1aa4bae85485d1d17b6f0ae                     |
+--------------------------------------+------------------------------------------------------+

nova boot --flavor b2c_web_1core --image Centos6.3_1.3 --security_group default --nic net-id=9106aee4-2dc0-4a6d-a789-10c53e2b88c1 ceph-test02.sh.vclound.com --availability-zone nova:sh-compute-129094.sh.vclound.com
+--------------------------------------+------------------------------------------------------+
| Property                             | Value                                                |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                               |
| OS-EXT-AZ:availability_zone          | nova                                                 |
| OS-EXT-SRV-ATTR:host                 | -                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000020f                                    |
| OS-EXT-STS:power_state               | 0                                                    |
| OS-EXT-STS:task_state                | scheduling                                           |
| OS-EXT-STS:vm_state                  | building                                             |
| OS-SRV-USG:launched_at               | -                                                    |
| OS-SRV-USG:terminated_at             | -                                                    |
| accessIPv4                           |                                                      |
| accessIPv6                           |                                                      |
| adminPass                            | wHddAW33sFBE                                         |
| config_drive                         |                                                      |
| created                              | 2015-06-25T03:51:03Z                                 |
| flavor                               | b2c_web_1core (5)                                    |
| hostId                               |                                                      |
| id                                   | b433b227-14ab-4157-8f08-362ad680e35e                 |
| image                                | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name                             | -                                                    |
| metadata                             | {}                                                   |
| name                                 | ceph-test02.sh.vclound.com                           |
| os-extended-volumes:volumes_attached | []                                                   |
| progress                             | 0                                                    |
| security_groups                      | default                                              |
| status                               | BUILD                                                |
| tenant_id                            | bb0b51d166254dc99bc7462c0ac002ff                     |
| updated                              | 2015-06-25T03:51:03Z                                 |
| user_id                              | 226e71f1c1aa4bae85485d1d17b6f0ae                     |
+--------------------------------------+------------------------------------------------------+

instance 状态

[root@sh-controller-129022 ~(keystone_admin)]# nova list
+--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+
| ID                                   | Name                        | Status | Task State | Power State | Networks                  |
+--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+
| 99d37977-a13a-4a8b-b8b1-e613a4959623 | ceph-test01.sh.vclound.com  | ACTIVE | -          | Running     | SH_DEV_NET=10.198.192.254 |
| b433b227-14ab-4157-8f08-362ad680e35e | ceph-test02.sh.vclound.com  | ACTIVE | -          | Running     | SH_DEV_NET=10.198.192.255 |
+--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+

创建云盘

[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2015-06-25T06:11:58.840626      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | 8516fb02-b578-4e57-9678-d30d2b0a6734 |
|       metadata      |                  {}                  |
|         size        |                  50                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2015-06-25T06:12:07.151001      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f |
|       metadata      |                  {}                  |
|         size        |                  50                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2015-06-25T06:12:14.321030      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 |
|       metadata      |                  {}                  |
|         size        |                  50                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

查询云盘

[root@sh-controller-129022 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 8516fb02-b578-4e57-9678-d30d2b0a6734 | available |     None     |  50  |     None    |  false   |             |
| 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | available |     None     |  50  |     None    |  false   |             |
| a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | available |     None     |  50  |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

连接云盘

[root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach 99d37977-a13a-4a8b-b8b1-e613a4959623 8516fb02-b578-4e57-9678-d30d2b0a6734
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | 8516fb02-b578-4e57-9678-d30d2b0a6734 |
| serverId | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| volumeId | 8516fb02-b578-4e57-9678-d30d2b0a6734 |
+----------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach 99d37977-a13a-4a8b-b8b1-e613a4959623 9d8aa395-5e6a-411a-9f19-6375f29e9f9f
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdd                             |
| id       | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f |
| serverId | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| volumeId | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f |
+----------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach b433b227-14ab-4157-8f08-362ad680e35e a5751c38-01c0-4f25-a02c-7d2a05d6ea36
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 |
| serverId | b433b227-14ab-4157-8f08-362ad680e35e |
| volumeId | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 |
+----------+--------------------------------------+

检测云盘

[root@sh-controller-129022 ~(keystone_admin)]# nova show 99d37977-a13a-4a8b-b8b1-e613a4959623
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                            |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                           |
| OS-EXT-AZ:availability_zone          | nova                                                                                             |
| OS-EXT-SRV-ATTR:host                 | sh-compute-128030.sh.vclound.com                                                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | sh-compute-128030.sh.vclound.com                                                                 |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000020d                                                                                |
| OS-EXT-STS:power_state               | 1                                                                                                |
| OS-EXT-STS:task_state                | -                                                                                                |
| OS-EXT-STS:vm_state                  | active                                                                                           |
| OS-SRV-USG:launched_at               | 2015-06-25T03:49:26.000000                                                                       |
| OS-SRV-USG:terminated_at             | -                                                                                                |
| SH_DEV_NET network                   | 10.198.192.254                                                                                   |
| accessIPv4                           |                                                                                                  |
| accessIPv6                           |                                                                                                  |
| config_drive                         |                                                                                                  |
| created                              | 2015-06-25T03:49:14Z                                                                             |
| flavor                               | b2c_web_1core (5)                                                                                |
| hostId                               | 8b5b75df8b0271d739323f1373b7363d432bb9c68b079ab3e94e1c1a                                         |
| id                                   | 99d37977-a13a-4a8b-b8b1-e613a4959623                                                             |
| image                                | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4)                                             |
| key_name                             | -                                                                                                |
| metadata                             | {}                                                                                               |
| name                                 | ceph-test01.sh.vclound.com                                                                       |
| os-extended-volumes:volumes_attached | [{"id": "8516fb02-b578-4e57-9678-d30d2b0a6734"}, {"id": "9d8aa395-5e6a-411a-9f19-6375f29e9f9f"}] | <-挂载两个云盘
| progress                             | 0                                                                                                |
| security_groups                      | default                                                                                          |
| status                               | ACTIVE                                                                                           |
| tenant_id                            | bb0b51d166254dc99bc7462c0ac002ff                                                                 |
| updated                              | 2015-06-25T03:49:26Z                                                                             |
| user_id                              | 226e71f1c1aa4bae85485d1d17b6f0ae                                                                 |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# nova show b433b227-14ab-4157-8f08-362ad680e35e
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | sh-compute-129094.sh.vclound.com                         |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | sh-compute-129094.sh.vclound.com                         |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000020f                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2015-06-25T03:52:05.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| SH_DEV_NET network                   | 10.198.192.255                                           |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2015-06-25T03:51:03Z                                     |
| flavor                               | b2c_web_1core (5)                                        |
| hostId                               | a5239c63509fa00ab056ca701363538ecc0afe41d8f886f82b345b4d | <- 挂载一个
| id                                   | b433b227-14ab-4157-8f08-362ad680e35e                     |
| image                                | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4)     |
| key_name                             | -                                                        |
| metadata                             | {}                                                       |
| name                                 | ceph-test02.sh.vclound.com                               |
| os-extended-volumes:volumes_attached | [{"id": "a5751c38-01c0-4f25-a02c-7d2a05d6ea36"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | bb0b51d166254dc99bc7462c0ac002ff                         |
| updated                              | 2015-06-25T03:52:05Z                                     |
| user_id                              | 226e71f1c1aa4bae85485d1d17b6f0ae                         |
+--------------------------------------+----------------------------------------------------------+

检测 cinder 状态

[root@sh-controller-129022 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 8516fb02-b578-4e57-9678-d30d2b0a6734 | in-use |     None     |  50  |     None    |  false   | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | in-use |     None     |  50  |     None    |  false   | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | in-use |     None     |  50  |     None    |  false   | b433b227-14ab-4157-8f08-362ad680e35e |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

测试

测试云盘读写

[root@ceph-test01 ~]# pvcreate  /dev/vdc /dev/vdd
  Physical volume "/dev/vdc" successfully created
  Physical volume "/dev/vdd" successfully created

[root@ceph-test01 ~]# vgcreate myvg /dev/vdc /dev/vdd
  Volume group "myvg" successfully created

[root@ceph-test01 ~]# lvcreate  -i 2 -n mylv  -l 100%FREE  myvg
  Using default stripesize 64.00 KiB
  Logical volume "mylv" created

[root@ceph-test01 ~]# yum install -y xfsprogs.x86_64 > /dev/null 2>&1

[root@ceph-test01 ~]# mkfs.xfs /dev/myvg/mylv
meta-data=/dev/myvg/mylv         isize=256    agcount=16, agsize=1638256 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=26212096, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@ceph-test01 ~]# mount /dev/myvg/mylv /mnt

[root@ceph-test01 ~]# df -h
文件系统              容量  已用  可用 已用%% 挂载点
/dev/vda1              20G  1.1G   18G   6% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/mapper/myvg-mylv
                      100G   33M  100G   1% /mnt

查询当前 ceph 情况

[root@sh-ceph-128213 ~]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    290T      290T        4615M             0
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd         0           0         0        99157G           0
    volumes     1      37220k         0        99157G          28   <- 留意当前默认下 ceph 只有 28 个对象

在 vm 上执行 操作

[root@ceph-test01 ~]# dd if=/dev/zero of=/mnt/1.img bs=1M count=700000
dd: 正在写入"/mnt/1.img": 设备上没有空间
记录了102309+0 的读入
记录了102308+0 的写出
107278180352字节(107 GB)已复制,982.231 秒,109 MB/秒

监控 ceph 存储空间

[root@sh-ceph-128212 var]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    290T      290T         337G          0.11
POOLS:
    NAME        ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd         0            0         0        99033G           0
    volumes     1      102399M      0.03        99033G       25622    <-  dd 在客户端有 1 个文件,  但在 ceph 中会出现 [25622-28] 个文件 2.5 万个小文件

监控 ceph 物理存储空间

[root@sh-ceph-128212 ceph-0]# df -h
文件系统                 容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root   50G  1.8G   49G    4% /
devtmpfs                  32G     0   32G    0% /dev
tmpfs                     32G     0   32G    0% /dev/shm
tmpfs                     32G   18M   32G    1% /run
tmpfs                     32G     0   32G    0% /sys/fs/cgroup
/dev/sda2                494M  123M  372M   25% /boot
/dev/mapper/centos-home  3.6T   33M  3.6T    1% /home
/dev/sdb1                3.7T  3.8G  3.7T    1% /var/lib/ceph/osd/ceph-0   
/dev/sdc1                3.7T  4.2G  3.7T    1% /var/lib/ceph/osd/ceph-1
/dev/sdd1                3.7T  4.3G  3.7T    1% /var/lib/ceph/osd/ceph-2
/dev/sdf1                3.7T  4.2G  3.7T    1% /var/lib/ceph/osd/ceph-3
/dev/sdg1                3.7T  4.1G  3.7T    1% /var/lib/ceph/osd/ceph-4
/dev/sdh1                3.7T  4.2G  3.7T    1% /var/lib/ceph/osd/ceph-5
/dev/sde1                3.7T  4.2G  3.7T    1% /var/lib/ceph/osd/ceph-6
/dev/sdi1                3.7T  3.7G  3.7T    1% /var/lib/ceph/osd/ceph-7
/dev/sdj1                3.7T  3.9G  3.7T    1% /var/lib/ceph/osd/ceph-8  
/dev/sdk1                3.7T  4.6G  3.7T    1% /var/lib/ceph/osd/ceph-9   <- 参考已用空间

发现之前 dd 的文件是被打散, 分布在各个不同的磁盘中, 而且分布数量也不一定是一样, 没有在 /var/lib/ceph/osd/ceph* 目录中找到一个具体的文件名, 只看见一堆被打散的小文件

卸载云盘

[root@ceph-test01 ~]# rm -rf /mnt/1.img
[root@ceph-test01 ~]# umount /mnt

检测 ceph 空间

[root@sh-controller-128022 cinder]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    290T      290T         304G          0.10
POOLS:
    NAME        ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd         0            0         0        99044G           0
    volumes     1      102399M      0.03        99044G       25622   <- 删除 1.img 文件 不会直接影响 ceph 磁盘容量

卸载 openstack 中云盘

[root@sh-controller-129022 ~(keystone_admin)]# nova volume-detach 99d37977-a13a-4a8b-b8b1-e613a4959623 8516fb02-b578-4e57-9678-d30d2b0a6734
[root@sh-controller-129022 ~(keystone_admin)]# nova volume-detach 99d37977-a13a-4a8b-b8b1-e613a4959623 9d8aa395-5e6a-411a-9f19-6375f29e9f9f

删除 openstack 云盘

[root@sh-controller-129022 ~(keystone_admin)]# cinder delete 8516fb02-b578-4e57-9678-d30d2b0a6734
[root@sh-controller-129022 ~(keystone_admin)]# cinder delete 9d8aa395-5e6a-411a-9f19-6375f29e9f9f

查询 ceph 存储空间

[root@sh-controller-128022 cinder]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    290T      290T       15961M             0
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd         0           0         0        99131G           0
    volumes     1      37220k         0        99131G          24   <- 之前产生的 1img 的对象被删除了

[root@sh-ceph-128212 ~]# df -h
文件系统                 容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root   50G  1.8G   49G    4% /
devtmpfs                  32G     0   32G    0% /dev
tmpfs                     32G     0   32G    0% /dev/shm
tmpfs                     32G   18M   32G    1% /run
tmpfs                     32G     0   32G    0% /sys/fs/cgroup
/dev/sda2                494M  123M  372M   25% /boot
/dev/mapper/centos-home  3.6T   33M  3.6T    1% /home
/dev/sdb1                3.7T   58M  3.7T    1% /var/lib/ceph/osd/ceph-0    <-参考, 磁盘也恢复了之前大小
/dev/sdc1                3.7T   62M  3.7T    1% /var/lib/ceph/osd/ceph-1
/dev/sdd1                3.7T   55M  3.7T    1% /var/lib/ceph/osd/ceph-2
/dev/sdf1                3.7T   61M  3.7T    1% /var/lib/ceph/osd/ceph-3
/dev/sdg1                3.7T   63M  3.7T    1% /var/lib/ceph/osd/ceph-4
/dev/sdh1                3.7T   56M  3.7T    1% /var/lib/ceph/osd/ceph-5
/dev/sde1                3.7T   56M  3.7T    1% /var/lib/ceph/osd/ceph-6
/dev/sdi1                3.7T   63M  3.7T    1% /var/lib/ceph/osd/ceph-7
/dev/sdj1                3.7T   59M  3.7T    1% /var/lib/ceph/osd/ceph-8
/dev/sdk1                3.7T   60M  3.7T    1% /var/lib/ceph/osd/ceph-9

证明, 删除云盘能够会自动释放空间

上一篇:【Rust日报】 2019-04-20


下一篇:PTC,虚实融合推进制造业数字化转型