openstack虚拟机磁盘io error问题

1:客户虚拟机出现readonly问题,最初操作是对磁盘进行了重新挂载,但是无效

mount -o remount rw /

2:客户进行了重启虚拟机,之后进入了单用户模式提示是fstab问题,注释掉/etc/fstab中的vdb,之后再次重启正常。
3:进入虚拟机对vdb进行挂载操作提示io error。怀疑是ceph集群集群出了问题,查看集群状态正常,物理磁盘无异常,使用如下命令对磁盘进行修复:

xfs_repair -v -L /dev/vdb
之后提示磁盘io error,如下:
[  164.966649] blk_update_request: I/O error, dev vdb, sector 0
[  164.967942] Buffer I/O error on dev vdb, logical block 0, lost async page write
[  164.968885] blk_update_request: I/O error, dev vdb, sector 16
[  164.969652] Buffer I/O error on dev vdb, logical block 2, lost async page write
[  164.970633] Buffer I/O error on dev vdb, logical block 3, lost async page write
[  164.971711] Buffer I/O error on dev vdb, logical block 4, lost async page write
[  164.972838] Buffer I/O error on dev vdb, logical block 5, lost async page write
[  164.974001] Buffer I/O error on dev vdb, logical block 6, lost async page write
[  164.975050] Buffer I/O error on dev vdb, logical block 7, lost async page write
[  164.976249] Buffer I/O error on dev vdb, logical block 8, lost async page write
[  164.977188] Buffer I/O error on dev vdb, logical block 9, lost async page write
[  164.978043] Buffer I/O error on dev vdb, logical block 10, lost async page write
[  164.978930] blk_update_request: I/O error, dev vdb, sector 29360128
[  164.983874] blk_update_request: I/O error, dev vdb, sector 29360208
[  164.992586] blk_update_request: I/O error, dev vdb, sector 29360264
[  165.000701] blk_update_request: I/O error, dev vdb, sector 29366384
[  165.010514] blk_update_request: I/O error, dev vdb, sector 29368784
[  165.018809] blk_update_request: I/O error, dev vdb, sector 29435544
[  165.026458] blk_update_request: I/O error, dev vdb, sector 29452288
[  165.034674] blk_update_request: I/O error, dev vdb, sector 33554432

4:卸载磁盘,尝试通过为磁盘拍快照,并通过快照创建磁盘的方式再次挂载
快照创建失败,提示卷为只读模式,如下:

2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3484, in rbd.Image.create_snap
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server ReadOnlyImage: [errno 30] error creating snapshot snapshot-0aa66c79-6c0f-4735-9f77-da9ade0e11fa from volume-6b3c1c7d-4337-4301-b224-3746686dec05
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server

5:查看卷状态,如下:

[root@bjyz-icontrol01 ~]# cinder show 6b3c1c7d-4337-4301-b224-3746686dec05
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attached_servers               | []                                   |
| attachment_ids                 | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2021-02-04T03:10:53.000000           |
| description                    |                                      |
| encrypted                      | False                                |
| group_id                       | None                                 |
| id                             | 6b3c1c7d-4337-4301-b224-3746686dec05 |
| metadata                       |                                      |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | 公共服务-2                           |
| os-vol-host-attr:host          | control@tencent#tencent              |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | fe2aeba987624a66864f5aa6992c64a7     |
| provider_id                    | None                                 |
| replication_status             | None                                 |
| service_uuid                   | 43314489-49f4-43e3-8d5e-6512a7ab3953 |
| shared_targets                 | False                                |
| size                           | 1024                                 |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | available                            |
| updated_at                     | 2021-09-17T10:19:15.000000           |
| user_id                        | 6fbcd30ba47b4480ba57f56fd5fa45e0     |
| volume_type                    | tencent                              |
+--------------------------------+--------------------------------------+
通过ceph查看卷状态,如下:
(ceph-mon)[root@bjyz-icontrol01 /]# rbd info tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
rbd image 'volume-6b3c1c7d-4337-4301-b224-3746686dec05':
	size 1 TiB in 262144 objects
	order 22 (4 MiB objects)
	snapshot_count: 1
	id: d301e3babf00f2
	block_name_prefix: rbd_data.d301e3babf00f2
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features: 
	flags: 
	create_timestamp: Thu Feb  4 11:10:53 2021
	access_timestamp: Fri Sep 17 18:58:02 2021
	modify_timestamp: Fri Sep 17 15:33:18 2021
 ###########################################
 (ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
There is 1 exclusive lock on this image.
Locker          ID                  Address                   
client.98051222 auto 94571522251008 192.168.2.31:0/1375643388  
(ceph-mon)[root@bjyz-icontrol01 /]# 
ssh到192.168.2.31主机,发现并没有此进程,
需要把此读写锁删除,如下:
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock rm tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05 "auto 94571522251008" client.98051222
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05

6:再次挂载卷正常

上一篇:符号模型检验(1)有限自动机


下一篇:学习笔记5