OpenStack实践系列⑨云硬盘服务Cinder
八、cinder
8.1存储的三大分类
块存储：硬盘，磁盘阵列DAS，SAN存储
文件存储：nfs，GluserFS，Ceph（PB级分布式文件系统），MooserFS（缺点Metadata数据丢失，虚拟机就毁了）
11.2网络类型选择
对象存储：swift，S3
8.2 cinder控制节点的部署
安装cinder
[root@node1 ~]# yum install openstack-cinder python-cinderclient -y
修改cinder配置文件
[default]
rpc_backend = rabbit
glance_host = 192.168.3.199
auth_strategy = keystone
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_rabbit]
rabbit_host = 192.168.3.199 # rabbitmq的主机
rabbit_port = 5672 # rabbitmq的端口
rabbit_userid = openstack # rabbitmq的用户
rabbit_password = openstack # rabbitmq的密码
[database]
connection = mysql://cinder:cinder@192.168.3.199/cinder # 配置mysql地址
[keystone_authtoken]
auth_uri = http://192.168.3.199:5000
auth_url = http://192.168.3.199:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder
修改后结果如下
[root@node1 cinder]# grep -n '^[a-Z]' /etc/cinder/cinder.conf
:glance_host = 192.168.3.199
:auth_strategy = keystone
:rpc_backend = rabbit
:connection = mysql://cinder:cinder@192.168.3.199/cinder
:auth_uri = http://192.168.3.199:5000
:auth_url = http://192.168.3.199:35357
:auth_plugin = password
:project_domain_id = default
:user_domain_id = default
:project_name = service
:username = cinder
:password = cinder
:lock_path = /var/lib/cinder/tmp
:rabbit_host = 192.168.3.199
:rabbit_port =
:rabbit_userid = openstack
:rabbit_password = openstack
修改nova的配置文件
[root@node1 ~]# vim /etc/nova/nova.conf
os_region_name = RegionOne # 通知nova使用cinder [cinder]部分
执行同步数据库操作
[root@node1 ~]# su -s /bin/sh -c "cinder-manage db sync" cinder 检查导入数据库结果
MariaDB [(none)]> use cinder
Database changed
MariaDB [cinder]> show tables;
+----------------------------+
| Tables_in_cinder |
+----------------------------+
| backups |
| cgsnapshots |
| consistencygroups |
| driver_initiator_data |
| encryption |
| image_volume_cache_entries |
| iscsi_targets |
| migrate_version |
| quality_of_service_specs |
| quota_classes |
| quota_usages |
| quotas |
| reservations |
| services |
| snapshot_metadata |
| snapshots |
| transfers |
| volume_admin_metadata |
| volume_attachment |
| volume_glance_metadata |
| volume_metadata |
| volume_type_extra_specs |
| volume_type_projects |
| volume_types |
| volumes |
+----------------------------+
rows in set (0.00 sec)
创建一个cinder用户,加入service项目，给予admin角色
[root@node1 ~]# source admin-openrc.sh
[root@node1 ~]# openstack user create --domain default --password-prompt cinder
User Password:
Repeat User Password: # （密码需要配置成cinder就是/etc/cinder/cinder.conf配置文件中配置的2648行）
+-----------+----------------------------------+
| Field | Value |
+-----------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 420d7573e9fc43b3b263f31bb6dd76e2 |
| name | cinder |
+-----------+----------------------------------+
[root@node1 ~]# openstack role add --project service --user cinder admin
重启nova-api服务和启动cinder服务
[root@node1 ~]# systemctl restart openstack-nova-api.service
[root@node1 ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service [root@node1 ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service
创建服务（包含V1和V2）
[root@node1 ~]# openstack service create --name cinder --description "OpenStack Block Storage" volume
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 6e3b2c3940d14300ab28aed272ade1d3 |
| name | cinder |
| type | volume |
+-------------+----------------------------------+ [root@node1 ~]# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| name | cinderv2 |
| type | volumev2 |
+-------------+----------------------------------+
分别对V1和V2创建三个环境（admin，internal，public）的endpoint
[root@node1 ~]# openstack endpoint create --region RegionOne volume public http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 007497468db7456d81157962f8740540 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+ [root@node1 ~]# openstack endpoint create --region RegionOne volume internal http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | e7543b96b69342bcabead7ad8a583860 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+ [root@node1 ~]# openstack endpoint create --region RegionOne volume admin http://192.168.3.199:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 12e4bea586384d43b16e4de5a00afb1b |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+ [root@node1 ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 07c56b0033454fbda201f3cc58ce0a1b |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+ [root@node1 ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 66b34a18d4de456ab32ebba24831b959 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+ [root@node1 ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.3.199:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | fc0bed271a7048a5aff7e63aebd0199a |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+
8.3 cinder存储节点的部署（此处使用nova的计算节点即node2.chinasoft.com）
本文中cinder后端存储使用ISCSI（类似于nova-computer使用的kvm），ISCSI使用LVM，在定义好的VG中，每创建一个云硬盘，就会增加一个LV，使用ISCSI发布。
在存储节点上加一个硬盘
查看磁盘添加情况
[root@node2 ~]# fdisk -l Disk /dev/sdb: 85.9 GB, bytes, sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes Disk /dev/sda: 128.8 GB, bytes, sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
Disk label type: dos
Disk identifier: 0x0004c2a9 Device Boot Start End Blocks Id System
/dev/sda1 * Linux
/dev/sda2 8e Linux LVM Disk /dev/mapper/centos-root: 32.2 GB, bytes, sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes Disk /dev/mapper/centos-swap: MB, bytes, sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes Disk /dev/mapper/centos-data: 42.9 GB, bytes, sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
创建一个pv和vg（名为cinder-volumes）
[root@node2 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
[root@node2 ~]# vgcreate cinder-volumes /dev/sdb
Volume group "cinder-volumes" successfully created
修改lvm的配置文件中添加filter，只有instance可以访问
[root@node2 ~]# vim /etc/lvm/lvm.conf
filter = [ "a/sdb/", "r/.*/"]
存储节点安装
[root@node2 ~]# yum install openstack-cinder targetcli python-oslo-policy -y
修改存储节点的配置文件，在这里直接拷贝控制节点的文件
[root@node1 ~]# scp /etc/cinder/cinder.conf 192.168.3.200:/etc/cinder/
修改存储(即计算节点node2.chinasoft.com)上的/etc/cinder/cinder.conf文件
添加如下配置：
默认没有[lvm]需要自己创建
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
volume_group = cinder-volumes # vg的名称：刚才创建的
iscsi_protocol = iscsi # 使用iscsi协议
iscsi_helper = lioadm
[root@node2 cinder]# grep -n '^[a-Z]' /etc/cinder/cinder.conf
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
540:enabled_backends = lvm # lvm 使用的后端是lvm，要对应添加的[lvm]，自定义也可以
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:cinder@192.168.3.199/cinder
2641:auth_uri = http://192.168.3.199:5000
2642:auth_url = http://192.168.3.199:35357
2643:auth_plugin = password
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack
[lvm] # 此行不是grep过滤出来的,因为是在配置文件最后添加上的，其对应的是540行的lvm
3416:volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
3417:volume_group = cinder-volumes # vg的名称：刚才创建的
3418:iscsi_protocol = iscsi # 使用iscsi协议
3419:iscsi_helper = lioadm
启动存储节点的cinder(这里是node2)
[root@node2 cinder]# systemctl enable openstack-cinder-volume.service target.service
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[root@node2 cinder]# systemctl start openstack-cinder-volume.service target.service
查看云硬盘服务状态（如果是虚拟机作为宿主机，时间不同步，会产生问题）
[root@node1 ~]# source admin-openrc.sh
[root@node1 ~]# cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | node1.chinasoft.com | nova | enabled | up | --28T09::54.000000 | - |
| cinder-volume | node2.chinasoft.com@lvm | nova | enabled | up | --28T09::58.000000 | - |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
创建一个云硬盘,将云硬盘挂载到虚拟机上,启动在虚拟机实例详情可以查看到，云硬盘挂载的情况
具体步骤：
1.创建云硬盘
2.点击 动作选项下的 管理已连接云硬盘，选择需要的虚拟机 将云硬盘挂载到指定的虚拟机上
在虚拟机中对挂载的硬盘进行分区格式化,如果有时不想挂载这个云硬盘了，一定不要删掉，生产环境一定要注意，否则虚拟机会出现error，应该使用umont确定卸载了，再使用dashboard进行删除云硬盘
# ssh cirros@192.168.3.103 $ sudo fdisk -l Disk /dev/vda: MB, bytes
heads, sectors/track, cylinders, total sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
Disk identifier: 0x00000000 Device Boot Start End Blocks Id System
/dev/vda1 * + Linux Disk /dev/vdb: MB, bytes
heads, sectors/track, cylinders, total sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
Disk identifier: 0x00000000 Disk /dev/vdb doesn't contain a valid partition table # 分区
$ sudo fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x3fecc8a5.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table will be corrected by w(rite) Command (m for help): n
Partition type:
p primary ( primary, extended, free)
e extended
Select (default p): p
Partition number (-, default ):
First sector (-, default ):
Using default value
Last sector, +sectors or +size{K,M,G} (-, default ):
Using default value Command (m for help): w
The partition table has been altered! Calling ioctl() to re-read partition table.
Syncing disks. 查看分区，已经生成了新的分区/dev/vdb
$ sudo fdisk -l Disk /dev/vda: MB, bytes
heads, sectors/track, cylinders, total sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
Disk identifier: 0x00000000 Device Boot Start End Blocks Id System
/dev/vda1 * + Linux Disk /dev/vdb: MB, bytes
heads, sectors/track, cylinders, total sectors
Units = sectors of * = bytes
Sector size (logical/physical): bytes / bytes
I/O size (minimum/optimal): bytes / bytes
Disk identifier: 0xfaacdc93 Device Boot Start End Blocks Id System
/dev/vdb1 Linux # 格式化
$ sudo mkfs.ext4 /dev/vdb1
mke2fs 1.42. (-Mar-)
Filesystem label=
OS type: Linux
Block size= (log=)
Fragment size= (log=)
Stride= blocks, Stripe width= blocks
inodes, blocks
blocks (5.00%) reserved for the super user
First data block=
Maximum filesystem blocks=
block groups
blocks per group, fragments per group
inodes per group
Superblock backups stored on blocks:
, , , Allocating group tables: done
Writing inode tables: done
Creating journal ( blocks): done
Writing superblocks and filesystem accounting information: done # 挂载
$ sudo mkdir /data
$ sudo mount /dev/vdb1 /data $ df -h
Filesystem Size Used Available Use% Mounted on
/dev 242.3M 242.3M % /dev
/dev/vda1 23.2M 18.0M 4.0M % /
tmpfs 245.8M 245.8M % /dev/shm
tmpfs .0K .0K .0K % /run
/dev/vdb1 1006.9M 17.3M 938.5M % /data
方法二：从云硬盘启动虚拟机
①先创建一个demo2的云硬盘
②动作-作为实例启动
九、虚拟机创建流程：
第一阶段：用户操作
1)用户使用Dashboard或者CLI连接keystone，发送用户名和密码，待keystone验证通过，keystone会返回给dashboard一个authtoken
2）Dashboard会带着上述的authtoken访问nova-api进行创建虚拟机请求
3）nova-api会通过keytoken确认dashboard的authtoken认证消息。
第二阶段：nova内组件交互阶段
4）nova-api把用户要创建的虚拟机的信息记录到数据库中.
5）nova-api使用rpc-call的方式发送请求给消息队列
6）nova-scheduler获取消息队列中的消息
7）nova-scheduler和查看数据库中要创建的虚拟机信息和计算节点的信息，进行调度
8)nova-scheduler把调度后的信息发送给消息队列
9）nova-computer获取nova-schedur发送给queue的消息
10）nova-computer通过消息队列发送消息给nova-conudctor，想要获取数据库中的要创建虚拟机信息
11）nova-conductor获取消息队列的消息
12）nova-conductor读取数据库中要创建虚拟机的信息
13）nova-conductor把从数据库获取的消息返回给消息队列
14）nova-computer获取nova-conducter返回给消息队列的信息
第三阶段：nova和其他组件进行交互
15）nova-computer通过authtoken和数据库返回的镜像id请求glance服务
16）glance会通过keystone进行认证
17）glance验证通过后把镜像返回给nova-computer
18）nova-computer通过authtoken和数据库返回的网络id请求neutron服务
19）neutron会通过keystone进行认证
20）neutron验证通过后把网络分配情况返回给nova-computer
21）nova-computer通过authtoken和数据库返回的云硬盘请求cinder服务
22）cinder会通过keystone进行认证
23）cinder验证通过后把云硬盘分配情况返回给nova-computer
第四阶段：nova创建虚拟机
24)nova-compute通过libvirt调用kvm根据已有的信息创建虚拟机，动态生成xml
25)nova-api会不断的在数据库中查询信息并在dashboard显示虚拟机的状态
生产场景注意事项：
1、新加的一个计算节点，创建虚拟机时间会很长，因为第一次使用计算节点，没有镜像，计算节点要把glance的镜像放在后端文件（/var/lib/nova/instance/_base）下，
镜像如果很大，自然会需要很长时间，然后才会在后端文件的基础上创建虚拟机（写时复制copy on write）。
2、创建虚拟机失败的原因之一：创建网桥失败。要保证eth0网卡配置文件的BOOTPROTE是static而不是dhcp状态。