探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

前言:上一篇文章 只是 RabbitMQ 的科普,本文将仔细分析 Cinder 中 RabbitMQ 的各组件的使用、消息的发送和接收等。由于各流程步骤很多,本文只会使用若干流程图来加以阐述,尽量做到图文自解释,不会添加很细的文字说明了。

1. Cinder 中创建卷的端到端过程

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

该过程主要包括两部分:

第一部分即初始化部分:cinder-api 服务启动过程中 (参见另一篇文章),APIRouter 类被初始化,接着它会初始化 VolumeController 类,最终,SchedulerAPI 类以及 VolumeAPI 类会被初始化,它们分别会调用 get_client 方法获取一个 RPCClient 的实例,其中,target 即发送消息的目的 RabbitMQ exchange,它由 cinder.conf 定义。该 RPCClient 实例的方法会分别被 SchedulerAPI 类和 VolumeAPI类来发送。

第二部分即流程处理部分:当 cinder-api 的 WSGI server 收到一个卷创建的请求后,APIRouter 会负责把该请求分发到 VolumeController 类的 create 方法 (其过程请参考另一篇文章)。该方法会依次:

(16)生成一个 task flow,其中的几个 task 依次提取request中的数据、预留 Quota、在 db 中创建 entery、提交 quota,最后调用 VolumeCastTask::exectue 方法。

(17)调用 SchedulerAPI 的 create_volume 方法,它会调用 RPCClient 的 cast 方法来发出一个创建volume 的 message。

(18 ~ 20)最终 SchedulerManager 的 create_volume 方法会被调用,它会执行 filtering 和 weighting 来选择一个最优的 host。

(23)确定好该 host 后,调用 volume.API 的 create_volume 方法。

(24 ~ 25)该方法会调用 RPCClient 的 cast 方法发出一个 volume creation 消息给 cinder-volume

(26)最终,指定 host 上的 VolumeManager 的 create_volume 方法会被调用,它会调用 cinder driver 来创建 volume。

该过程显示了 volume 创建过程中,cinder 的三大组件 cinder-api、cinder-scheduler 和 cinder-volume 使用 RabbitMQ 来传递和接收消息来执行操作的过程。

2. 消息发送机制 (olso.messaging.RPCClient 的 cast 和 call 方法)

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

几个要点:

(0)cast 方法把消息发出去就算了,不需要返回值;call 方法在消息发出后需要等待消息结果返回,因此它在发消息(步骤 27)之前,会创建一个 waiter (步骤26)来等待消息返回。

(3)get_transport 方法会使用 stevedore 库来加载特定的 messaging driver 类,默认情况下 conf.rpc_backend 的值为 'rabbit', 因此其对应的 driver 类 oslo.messaging._drivers.impl_rabbit.RabbitDriver会被加载;然后生成一个 Transport 实例,该实例实际上只是 driver 实例的一个wrapper。

(6)初始化 Target 实例,它实际上表明了 message 的目的地,包括消息被发往的地方或者一个应用需要监听的对象。其属性包括:      

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收
def __init__(self, exchange=None, topic=None, namespace=None, version=None, server=None, fanout=None):
        self.exchange = exchange // MQ 的exchange,使用 cinder.conf 中的定义,默认是 control_exchange : openstack
        self.topic = topic //MQ 的 queue,由 cinder.conf 指定: scheduler_topic = cinder-scheduler,volume_topic = cinder-volume 
        self.namespace = namespace //消息处理类的方法集合,默认是 null
        self.version = version
        self.server = server //消息的特定目的consumer application,是 cinder 里面的一个消息处理类
        self.fanout = fanout //一个flag,设置的话,消息将发给所有queue的consumers
探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

(15 ~ 21) cast 方法的实现,最终是使用 RabbitMQ 将消息发出去

(20 ~ 35)call 方法的实现,在第 27 步将消息发出去之前先创建一个 waiter 来等待返回的消息,它等到返回的消息后会把结果传回到caller。

 

 3. 消息接收和分发机制 (olso.messaging.RPCDispatcher)

 

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

 

(2~3)cinder-scheduler 和在各节点上的 cinder-volume 作为 service 启动的时候,会初始化并启动一个RPC server,该 server 会监听 message queu并分发消息。

(8,30)RPCDispatcher 负责为接收到的 PRC message 调用一个 endpoint 类 的 method。

  • endpoint 类:
    • 对 cinder-scheduler 的 rpcapi 发送的消息来说,endpoint 是类 SchedulerManager;
    • 对 cinder-volume 的 rpcapi 发出的消息来说,endpoint 是类 VolumeManager。
  • endpoint 类的方法:被调用的方法名称会在 cast 方法中作为参数传入,比如厦门 create_volume 方法 cast 这个消息后,它会分发到 SchedulerManager 的 create_volume 方法。
//cinder/scheduler/rpcapi.py
def
create_volume(self, ctxt, topic, volume_id, snapshot_id=None, image_id=None, request_spec=None, filter_properties=None): cctxt = self.client.prepare(version='1.2') request_spec_p = jsonutils.to_primitive(request_spec) return cctxt.cast(ctxt, 'create_volume', topic=topic, volume_id=volume_id, snapshot_id=snapshot_id, image_id=image_id, request_spec=request_spec_p, filter_properties=filter_properties)

 (11,26 ~ 28)EventletExecutor 会初始化一个大小为 conf.rpc_thread_pool_size 的线程池,它调用 AMQPListener 的 poll 方法获取 message 后,会在一个线程中调用 RPCDispatcher 的 __call__ 方法来调用消息处理方法。

(20, 26)AMQPListener 会无限循环去获取 message,直到它被停止。获取到的 message 会被交给 EventletExecutor ,它会调用RCPDispatcher的方法将其分发给某个类的某个方法。

4. Cinder 所用到的 RabbitMQ 组件

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

注:有些组件,比如 cinder_scheduler.controller 尚不清楚其用途。

 5. 消息示例

5.1 每个节点上的 cinder-volume 服务向 cinder-scheduler 服务更新其 capabilities 的消息

 

消息发送者 发送频率 MQ exchange 消息接收者类 备注
每个cinder-volume 节点上的cinder-volume 服务 1 分钟 cinder-scheduler_fanout SchedulerManager::update_service_capabilities cinder-scheduler 的 HostManager 类维护一个数组service_states,每个cinder-volume 节点占该数组的一项,其值为该节点上 cinder-volume 服务的 capabilities,该值通过消息机制定期更新。在发生特定操作比如删除卷时,会进行立刻更新。

消息内容示例:

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收
{u'_context_domain': None, u'_context_request_id': u'req-c0a0babc-3fec-41fe-ae6a-45418beec8fc', u'_context_quota_class': None, u'_context_service_catalog': [], u'_context_auth_token': '<SANITIZED>', u'_context_user': None, u'_context_user_id': None, u'_context_is_admin': True, u'version': u'1.0', u'_context_project_domain': None, u'_context_timestamp': u'2015-03-16T10:13:14.676590', u'method': u'update_service_capabilities', u'_context_remote_address': None, u'_context_roles': [u'admin'], u'args': {u'service_name': u'volume', u'host': u'network@lvmdriver-network', u'capabilities': {u'pools': [{u'pool_name': u'lvmbackend', u'QoS_support': False, u'allocated_capacity_gb': 4, u'free_capacity_gb': 0.36, u'location_info': u'LVMVolumeDriver:network:system:default:0', u'total_capacity_gb': 9.06, u'reserved_percentage': 0}], u'driver_version': u'2.0.0', u'vendor_name': u'Open Source', u'volume_backend_name': u'lvmbackend', u'storage_protocol': u'iSCSI'}}, u'_unique_id': u'f18ce7979c3148ce933ba93ee77e4f47', u'_context_project_name': None, u'_context_read_deleted': u'no', u'_context_user_identity': u'- - - - -', u'_context_tenant': None, u'_context_project_id': None, u'_context_user_domain': None}
探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

 5.2 创建卷时 cinder-api 服务发给 cinder-scheduler 服务的消息

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

MQ Exchange: openstack

Message body:
{"oslo.message": "{"_context_domain": null, "_context_request_id": "req-7bb7ff5b-97e5-481a-97d4-c636b2e0687a", "_context_quota_class": null, "_context_service_catalog": [{"endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "id": "055951b654364d918aaa37bbe365906e", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "endpoints_links": [], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...==", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.2", "_context_timestamp": "2015-03-21T07:26:33.337831", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "create_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"request_spec": {"source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "volume_metadata": [], "volume_admin_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "display_description": null, "availability_zone": "az1", "attach_status": "detached", "source_volid": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}, "volume_type": {}, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "source_volid": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc"}, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "filter_properties": {}, "topic": "cinder-volume", "image_id": null, "snapshot_id": null}, "_unique_id": "73f2789be1124ac095909d7e4e542d8d", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}
Message routing_key: 'cinder-scheduler'
Endpoint method: SchedulerManager::create_volume
探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

5.3 创建卷时 cinder-scheduler 服务发给某个节点上 cinder-volume 服务的消息

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收
MQ Exchange: openstack

Message body:
'{"oslo.message": "{"_context_domain": null, "_context_request_id": "req-7bb7ff5b-97e5-481a-97d4-c636b2e0687a", "_context_quota_class": null, "_context_service_catalog": [{"endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "055951b654364d918aaa37bbe365906e", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "endpoints_links": [], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...==", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.4", "_context_timestamp": "2015-03-21T07:26:33.337831", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "create_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"request_spec": {"source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "availability_zone": "az1", "consistencygroup_id": null, "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "volume_admin_metadata": [], "attach_status": "detached", "display_description": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "volume_metadata": [], "replication_status": "disabled", "snapshot_id": null, "source_volid": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}, "volume_type": {}, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "source_volid": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "resource_properties": {"status": "creating", "volume_type_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "volume_metadata": [], "volume_admin_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "display_description": null, "availability_zone": "az1", "attach_status": "detached", "source_volid": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "display_name": null, "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}}, "source_replicaid": null, "allow_reschedule": true, "filter_properties": {"config_options": {}, "availability_zone": "az1", "request_spec": {"volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "volume_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "availability_zone": "az1", "user_id": "1dc0db32a936496ebfc50be54924a7cc", "attach_status": "detached", "display_description": null, "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "replication_status": "disabled", "snapshot_id": null, "encryption_key_id": null, "source_volid": null, "volume_admin_metadata": [], "source_replicaid": null, "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "consistencygroup_id": null, "size": 1, "metadata": {}}, "volume_type": {}, "image_id": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "resource_properties": {"status": "creating", "volume_type_id": null, "volume_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "source_volid": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1, "display_name": null, "source_replicaid": null, "availability_zone": "az1", "attach_status": "detached", "display_description": null, "encryption_key_id": null, "volume_admin_metadata": [], "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "metadata": {}}}, "qos_specs": null, "size": 1, "retry": {"num_attempts": 1, "hosts": ["block2@lvmdriver-b21#lvmbackend"]}, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "volume_type": {}, "resource_type": {}, "metadata": {}}, "source_volid": null, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc"}, "_unique_id": "e730b03f68cb426f906e39c64e265956", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}'
Message routing_key: 'cinder-volume.block2@lvmdriver-b21'
Endpoint method: VolumerManager::create_volume
探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

 

5.4 删除卷时 cinder-api 服务发给目的节点上 cinder-volume 服务的消息

探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收
MQ Exchange: openstack
Message body:
{"oslo.message": "{"_context_domain": null, "_context_request_id": "req-a5297b18-7b87-4ea1-914d-0174790a1ca5", "_context_quota_class": null, "_context_service_catalog": [{"endpoints_links": [], "endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "055951b654364d918aaa37bbe365906e", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.15", "_context_timestamp": "2015-03-21T07:16:02.443178", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "delete_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"unmanage_only": false, "volume_id": "a8a4bf9b-b233-4bd4-9ec4-d132d3b4b0af"}, "_unique_id": "370304cba81641ee82e8eaf84b519e25", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}

Message routing_key: 'cinder-volume.block2@lvmdriver-b21'
Endpoint method: VolumeManager::delete_volume
探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收



上一篇:【产品动态】历经3年的打磨,数据构建及管理平台Dataphin增加了什么新功能?


下一篇:本地编译tomcat的docker镜像