分布式存储Ceph(七) 对象存储RadosGW使用

2023-10-10 15:18:04

七对象存储RadosGW使用

7.1 RadosGW 对象存储简介

RadosGW是对象存储的一种实现方式，RADOS网关也称为Ceph对象网关、RadosGW、RGW，是一种服务，使客户端能够利用标准对象存储API来访问Ceph集群，它支持AWS S3和Swift, rgw运行于librado之上，在ceph0.8版本之后使用Civetweb的web服务器来响应api请求，可以使用nginx或者apache替代，客户端基于http/https协议用过RESTful API与rgw通信，而rgw则使用librados与ceph集群通信，rgw客户端通过s3或者swift api使用rgw用户进行身份验证，然后rgw网关代表用户cephx与ceph存储进行身份验证。

7.2 对象存储的特点

7.2.1 对象存储的特点

通过对象存储将数据存储为对象，每个对象除了包含数据，还包含数据自身的元数据。
对象通过Object ID来检索，无法通过普通文件系统的方式通过文件路径名称操作直接访问对象，只能通过API来访问，或者第三方客户端。
对象存储中的对象不整理到目录树中，而是存储在扁平的命名空间中，Amazon S3将这个扁平名称空间称为bucket，而swift则将其称为容器。
无论bucket还是容器，都不能嵌套。
ceph使用bucket作为存储桶，bucket需要被授权才能访问到。实现对象数据的存储和多用户隔离，数据存储在bucket中，用户的权限也是针对bucket进行授权，可以设置用户对不同的bucket拥有不同的权限，以实现权限管理。
方便横向扩展、快速检索数据。
不支持客户端挂载，且需要客户端在访问的时候指定文件名称。
不是很适用于文件过于频繁修改及删除的场景。

7.2.2 bucket 特性

存储空间是用于存储对象的容器，所有的对象都必须隶属于某个存储空间，可以设置和修改存储空间属性来控制地域、访问权限、生命周期等，这些属性设置直接作用于该存储空间内所有对象，因此您可以通过灵活创建不同的存储空间来完成不同的管理功能。
同一个存储空间内部都是扁平的，没有文件系统的目录概念，所有的对象都直接隶属于其对应的存储空间。
每个用户都可以拥有多个存储空间。
存储空间的名称在OSS范围内必须是全局唯一的，一旦创建之后无法修改名称。
存储空间内部的对象数目没有限制。

7.2.3 bucket命名规范

只能包括大小写字母、数字和短横线（-）。
必须以小写字母或者数据开头和结尾。
长度必须在3-63字节之间。

7.3 对象存储访问对比

Amazon S3：提供了user、bucket和object，分别表示为用户、存储桶和对象，其中bucket隶属于user，可以这对user设置不同bucket的名称空间访问权限，而且不用用户允许访问相同的bucket。
OpenStack Swift：提供了user、container和object，分别对应于用户、存储桶和对象，不过它还额外为user提供了父级组件account，用于表示一个项目或租户，因此一个account中它包含一个到多个user，他们可共享使用同一组container，并为container提供名称空间。
RadosGW：提供了user、subuser、bucket和object，其中的user对应S3的user，而subuser则对应于Swif的user，不过user和subuser都不支持为bucket提供名称空间，因此，不同用户的存储桶也不允许同名，不过自Jewel版本起，RadosGW引入了tenant（租户）用于为user和bucket提供名称空间，但它是个可选组件，RadosGW基于ACL为不同的用户设置不用的权限。
- Read 读加执行权限
- Write 写权限
- Readwrite 只读
- full-control 全部控制权限

7.4 部署RadosGW服务

7.4.1 安装radosgw服务

root@ceph-mgr-01:~# apt -y install radosgw
root@ceph-mgr-02:~# apt -y install radosgw

7.4.2 初始化radosgw

7.4.2.1 初始化ceph-mgr-01节点rgw服务

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr-01
[ceph_deploy.conf][DEBUG ] found configuration file at: /var/lib/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-mgr-01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr-01', 'rgw.ceph-mgr-01')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc0a912a1e0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7fc0a99d26d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr-01:rgw.ceph-mgr-01
ceph@ceph-mgr-01's password: 
[ceph-mgr-01][DEBUG ] connection detected need for sudo
ceph@ceph-mgr-01's password: 
[ceph-mgr-01][DEBUG ] connected to host: ceph-mgr-01 
[ceph-mgr-01][DEBUG ] detect platform information from remote host
[ceph-mgr-01][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr-01
[ceph-mgr-01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr-01][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr-01][DEBUG ] create a keyring file
[ceph-mgr-01][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr-01][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr-01 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr-01/keyring
[ceph-mgr-01][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr-01
[ceph-mgr-01][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr-01.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr-01][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr-01
[ceph-mgr-01][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr-01 and default port 7480

7.4.2.2 初始化ceph-mgr-02节点rgw服务

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr-02
[ceph_deploy.conf][DEBUG ] found configuration file at: /var/lib/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-mgr-02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr-02', 'rgw.ceph-mgr-02')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7efd6c1391e0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7efd6c9e16d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr-02:rgw.ceph-mgr-02
ceph@ceph-mgr-02's password: 
12[ceph-mgr-02][DEBUG ] connection detected need for sudo
ceph@ceph-mgr-02's password: 
Permission denied, please try again.
ceph@ceph-mgr-02's password: 
[ceph-mgr-02][DEBUG ] connected to host: ceph-mgr-02 
[ceph-mgr-02][DEBUG ] detect platform information from remote host
[ceph-mgr-02][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr-02
[ceph-mgr-02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr-02][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr-02][DEBUG ] create a keyring file
[ceph-mgr-02][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr-02][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr-02 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr-02/keyring
[ceph-mgr-02][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr-02
[ceph-mgr-02][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr-02.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr-02][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr-02
[ceph-mgr-02][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr-02 and default port 7480

7.4.3 验证radosgw服务状态

点击查看代码

root@ceph-mgr-01:~# systemctl status ceph-radosgw@rgw.ceph-mgr-01
root@ceph-mgr-02:~# systemctl status ceph-radosgw@rgw.ceph-mgr-02

7.4.4 验证ceph集群rgw状态

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 6e521054-1532-4bc8-9971-7f8ae93e8430 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-02,ceph-mon-03 (age 35m) mgr: ceph-mgr-01(active, since 9d), standbys: ceph-mgr-02 mds: 2/2 daemons up, 2 standby osd: 9 osds: 9 up (since 3w), 9 in (since 3w) rgw: 2 daemons active (2 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 8 pools, 265 pgs objects: 253 objects, 24 MiB usage: 1.5 GiB used, 178 GiB / 180 GiB avail pgs: 265 active+clean

7.4.5 验证radosgw服务进程

root@ceph-mgr-01:~# ps -ef | grep radosgw ceph 13642 1 0 17:26 ? 00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr-01 --setuser ceph --setgroup ceph

root@ceph-mgr-02:~# ps -ef | grep radosgw ceph 10660 1 0 17:28 ? 00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr-02 --setuser ceph --setgroup ceph

7.4.6 rgw存储池功能

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
device_health_metrics
.rgw.root           # 包含realm（领域信息），比如zone和zonegroup
default.rgw.log     # 存储日志信息，用于记录各种log信息
default.rgw.control # 系统控制池，在有数据更新时，通知其它rgw更新缓存
default.rgw.meta    # 元数据存储池，通过不同的名称空间分别存储不同的rados对象。这些名称空间包括用户UID及其bucket映射信息的名称空间users.uid、用户的密钥名称空间user.keys、用户email名称空间users.email、用户的subuser的名称空间users.swift，以及bucket的名称空间root等。
default.rgw.buckets.index # 有数据后会生成，存放bucket到object的索引信息
default.rgw.buckets.data  # 有数据后会生成，存放对象的数据
default.rgw.buckets.non-ec # 有数据后会生成，数据的额外信息存储池

7.4.7 验证rgw zone信息

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ radosgw-admin zone get --rgw-zone=default
{
    "id": "345c2549-82fd-4331-948f-4efff49c0f0e",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}

7.4.8 访问radosgw服务

ceph@ceph-deploy:~/ceph-cluster$ curl http://172.16.10.248:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

7.5 rgw服务配置

7.5.1 自定义端口

https://docs.ceph.com/en/latest/radosgw/frontends/

7.5.1.1 修改配置文件

点击查看代码

root@ceph-mgr-01:~# vim /etc/ceph/ceph.conf
[client.rgw.ceph-mgr-01]

rgw_host = ceph-mgr-01

rgw_frontends = civetweb port=9900

7.5.1.2 重启rgw服务

root@ceph-mgr-01:~# systemctl restart ceph-radosgw@rgw.ceph-mgr-01

7.5.1.3 验证rgw服务端口

点击查看代码

root@ceph-mgr-01:~# netstat -tnlp | grep radosgw
tcp        0      0 0.0.0.0:9900            0.0.0.0:*               LISTEN      15949/radosgw

7.5.2 rgw https

方式一自建证书
方式二购买证书通过nginx代理访问

7.6 测试数据读写

7.6.1 创建rgw账户

点击查看代码

ceph@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="wgs01" --display-name="wgs01"
{
    "user_id": "wgs01",
    "display_name": "wgs01",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "wgs01",
            "access_key": "Q4XILSOF3BQ6MRBE2084",
            "secret_key": "Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

7.6.2 安装s3cmd客户端

点击查看代码

root@ceph-client-01:~# apt -y install s3cmd

7.6.3 配置s3cmd客户端执行环境

7.6.3.1 配置s3cmd客户端域名解析

root@ceph-client-01:~# cat /etc/hosts 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters

172.16.10.225 rgw.wgs.com

7.6.3.2 配置s3cmd命令执行环境

点击查看代码

root@ceph-client-01:~# s3cmd --help
root@ceph-client-01:~# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.

Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.

Access Key: Q4XILSOF3BQ6MRBE2084      # 输入用户Access Key

Secret Key: Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76   # 输入用户Secret Key

Default Region [US]:  # region选项
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [s3.amazonaws.com]: rgw.wgs.com:9900  #RGW域名
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used

if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw.wgs.com:9900/%(bucket) #bucket域名格式
Encryption password is used to protect your files from reading

by unauthorized persons while in transfer to S3

Encryption password:  # 是否需要加密

Path to GPG program [/usr/bin/gpg]:  # gpg路径
When using secure HTTPS protocol all communication with Amazon S3

servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP, and can only be proxied with Python 2.7 or newer

Use HTTPS protocol [Yes]: No  # 是否使用https
On some networks all internet access must go through a HTTP proxy.

Try setting it here if you can't connect to S3 directly

HTTP Proxy server name:   # http代理
New settings:

Access Key: Q4XILSOF3BQ6MRBE2084

Secret Key: Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76

Default Region: US

S3 Endpoint: rgw.wgs.com:9900

DNS-style bucket+hostname:port template for accessing a bucket: rgw.wgs.com:9900/%(bucket)

Encryption password:

Path to GPG program: /usr/bin/gpg

Use HTTPS protocol: False

HTTP Proxy server name:

HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y   # 测试连接rgw是否正常

Please wait, attempting to list all buckets...

Success. Your access key and secret key worked fine

码农公寓

七 对象存储RadosGW使用

7.1 RadosGW 对象存储简介

7.2 对象存储的特点

7.2.1 对象存储的特点

7.2.2 bucket 特性

7.2.3 bucket命名规范

7.3 对象存储访问对比

7.4 部署RadosGW服务

7.4.1 安装radosgw服务

7.4.2 初始化radosgw

7.4.2.1 初始化ceph-mgr-01节点rgw服务

7.4.2.2 初始化ceph-mgr-02节点rgw服务

7.4.3 验证radosgw服务状态

7.4.4 验证ceph集群rgw状态

7.4.5 验证radosgw服务进程

7.4.6 rgw存储池功能

7.4.7 验证rgw zone信息

7.4.8 访问radosgw服务

7.5 rgw服务配置

7.5.1 自定义端口

7.5.1.1 修改配置文件

7.5.1.2 重启rgw服务

7.5.1.3 验证rgw服务端口

7.5.2 rgw https

7.6 测试数据读写

7.6.1 创建rgw账户

7.6.2 安装s3cmd客户端

7.6.3 配置s3cmd客户端执行环境

7.6.3.1 配置s3cmd客户端域名解析

7.6.3.2 配置s3cmd命令执行环境

相关文章

七对象存储RadosGW使用