Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题

2022-10-03 12:16:57

一.问题的表象

过程:创建完windowserver2012r2版本虚拟机，挂载cinder卷，写入一定的数据
问题:关闭虚拟机所在的宿主机，执行疏散操作，虚拟机疏散失败

二.问题日志

  • 迁移的目的服务器nova-compute的日志信息如下：
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Exception during message handling: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7829, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     kwargs[‘instance‘], e, sys.exc_info())
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5436, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5389, in _detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 415, in detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 412, in _do_locked_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self._do_detach(*args, **_kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 400, in _do_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self[‘attachment_id‘])
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     ‘code‘: getattr(ex, ‘code‘, None)})
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._cs_request(url, ‘DELETE‘, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server
  • 迁移的目的服务器cinder-volume的日志信息如下：
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, ‘Lost connection to MySQL server during query‘) (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4503, in attachment_delete
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/base.py", line 314, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 1777, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6980, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _GET_METHODS[model](context, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2050, in volume_attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _attachment_get(context, attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1982, in _attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     .options(joinedload(‘volume‘))
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     ret = list(self[0:1])
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return list(res)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._execute_and_instances(context)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     close_with_result=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     **kw
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = self.session.connection(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     execution_options=execution_options)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = engine.contextual_connect(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._wrap_pool_connect(self.pool.connect, None),
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     e, dialect, self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return fn()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     rec = pool._do_get()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._dec_overflow()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._create_connection()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 699, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.connect()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 936, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._request_authentication()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     auth_packet = self._read_packet()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     packet_header = self._read_bytes(4)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, ‘Lost connection to MySQL server during query‘) (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server

日志分析:
1.我们模拟的是虚拟机的疏散过程，在疏散的过程中，我们查看到目的服务器的nova-compute.log和cinder-volume.log的日志存在错误

2.进一步分析得知，我们发现出现问题的原因在cinder-volume，对比虚拟机能够正常疏散时的log日志，我们发现正常疏散时，cinder-volume.log的疏散显示的日志如下：

2018-11-28 14:37:54.772 34 INFO cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Terminate volume connection completed successfully.
2018-11-28 14:37:54.773 34 DEBUG cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Deleting attachment 6e32750e-6134-45cc-942f-c1065826f28b. _do_attachment_delete /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py:4519

3.对比cinder-volume.log问题日志和正常日志，我们分析，在虚拟机的疏散过程，涉及卷的卸载过程，怀疑的cinder卷的卸载存在问题，对此，我模拟windows server2012的虚拟机挂载cinder卷，写大量数据，然后分离卷连接，模拟了大概50次左右，复现过一次cinder卷分离失败的错误，对比log进行分析对比，发现和上面的虚拟机疏散失败问题一致。

4.分析cinder卷的挂载流程，我们发现根据log提示和实际cinder卷的卸载流程存在中断卷连接的操作，也就是第三章节中1.4.6.2步骤。这里我们第三章节从cinder-volume的日志来看，函数在/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes此处提示丢失数据库连接错误，导致了中断卷连接时报错。

三.卷的卸载流程

Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题
上述流程图显示，卷在detach()->get vol info by id->terminate connection(call)时报错。

代码分析：

我们看到函数在_read_bytes处抛出错误：

进入cinder-volume容器，查看/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py代码的此函数如下:

def _read_bytes(self, num_bytes):
        self._sock.settimeout(self._read_timeout)
        while True:
            try:
                data = self._rfile.read(num_bytes)
                break
            except (IOError, OSError) as e:
                if e.errno == errno.EINTR:
                    continue
                self._force_close()
                raise err.OperationalError(
                    CR.CR_SERVER_LOST,
                    "Lost connection to MySQL server during query (%s)" % (e,))
        #如果MySQL服务器没有根据协议所述发送足够的数据，就会抛出如下错误
        if len(data) < num_bytes:
            self._force_close()
            raise err.OperationalError(
                CR.CR_SERVER_LOST, "Lost connection to MySQL server during queryy
")
        return data
        
#当服务器正在关闭(但显然仍在运行，因为这是从服务器发送的数据)时，将在专门为MariaDB记录的情况下抛出     
if packet_number == 0:
# MariaDB sends error packet with seqno==0 when shutdown
    raise err.OperationalError(
    CR.CR_SERVER_LOST,
    "Lost connection to MySQL server during query")
raise err.InternalError(
    "Packet sequence number wrong - got %d expected %d"
    % (packet_number, self._next_seq_id))

四.解决办法

MySQL数据库中，max_allowed_packet允许的server段接受的数据包最大为4M，有的时候大的插入和更新可能会被max_allowed_packet参数限制，导致失败，这里我们修改了max_allowed_packet参数。

[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
#增加如下行
max_allowed_packet= ‘32M‘

相关知识链接:

数据库知识: https://blog.csdn.net/hxpjava1/article/details/80522201
数据库问题分析：https://bugs.launchpad.net/tripleo/+bug/1691951
cinder的流程图：https://blog.csdn.net/inodes/article/details/24729883

Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题

上一篇：MySQL之模糊查询


下一篇：centos7安装mysql8.0.19