CRUSHMAP详解

 

CRUSH的全称是Controlled Replication Under Scalable Hashing,是ceph数据存储的分布式选择算法,也是ceph存储引擎的核心。

ceph的客户端在往集群里读写数据时,动态计算数据的存储位置。这样ceph就无需维护一个叫metadata的东西,从而提高性能。

ceph分布式存储有关键的3R: Replication(数据复制)、Recovery(数据恢复)、Rebalancing(数据均衡)。在组件故障时,ceph默认等待300秒,然后将OSD标记为down和out,并且初始化recovery操作。这个等待时间可以在集群配置文件的mon_osd_down_out_interval参数里设置。

当新的主机或磁盘加入到集群时,CRUSH开始rebalancing操作,它将数据从存在的主机、磁盘迁移到新的主机、磁盘。rebalancing时会尽量利用所有磁盘,以提高集群性能。如果ceph集群在重度使用中,推荐做法是新加入的磁盘设置权重0,并且逐步提高权重,使得数据迁移缓慢发生,以免影响性能。所有的分布式存储在扩容时都建议这样操作。

在实际中可能经常需要调整集群的布局。默认的CRUSH布局很简单,执行ceph osd tree命令,会看到仅有host和OSD这两种bucket类型在root下面。默认的布局对分区容错很不利,没有rack、row、room这些概念。下面我们增加一种bucket类型:rack(机架)。所有的host(主机)都应位于rack下面。

 

一、修改crushmap实验

 

(1)执行ceph osd tree得到当前的集群布局:

 

 
[root@node3 ~]# ceph osd tree
ID  CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF 
 -1       0.05878 root default                           
 -3       0.01959     host node1                         
  0   hdd 0.00980         osd.0      up  1.00000 1.00000 
  3   hdd 0.00980         osd.3      up  1.00000 1.00000 
 -5       0.01959     host node2                         
  1   hdd 0.00980         osd.1      up  1.00000 1.00000 
  4   hdd 0.00980         osd.4      up  1.00000 1.00000 
 -7       0.01959     host node3                         
  2   hdd 0.00980         osd.2      up  1.00000 1.00000 
  5   hdd 0.00980         osd.5      up  1.00000 1.00000 
 

 

 

 

(2)增加rack:

 

 
[root@node3 ~]# ceph osd crush add-bucket rack03 rack
added bucket rack03 type rack to crush map
[root@node3 ~]# ceph osd crush add-bucket rack01 rack
added bucket rack01 type rack to crush map
[root@node3 ~]# ceph osd crush add-bucket rack02 rack
added bucket rack02 type rack to crush map
 

 

 

(3)将host移动到rack下面:

 

 
[root@node3 ~]# ceph osd crush move node1 rack=rack01
moved item id -3 name 'node1' to location {rack=rack01} in crush map
[root@node3 ~]# ceph osd crush move node2 rack=rack02
moved item id -5 name 'node2' to location {rack=rack02} in crush map
[root@node3 ~]# ceph osd crush move node3 rack=rack03
moved item id -7 name 'node3' to location {rack=rack03} in crush map
 

 

 

(4)将rack移动到默认的root下面:

 

 
[root@node3 ~]# ceph osd crush move rack01 root=default
moved item id -9 name 'rack01' to location {root=default} in crush map
[root@node3 ~]# ceph osd crush move rack02 root=default
moved item id -10 name 'rack02' to location {root=default} in crush map
[root@node3 ~]# ceph osd crush move rack03 root=default
moved item id -11 name 'rack03' to location {root=default} in crush map
 

 

 

(5)再次运行ceph osd tree命令:

 

 
[root@node3 ~]# ceph osd tree
ID  CLASS WEIGHT  TYPE NAME          STATUS REWEIGHT PRI-AFF 
 -1       0.05878 root default                               
 -9       0.01959     rack rack01                            
 -3       0.01959         host node1                         
  0   hdd 0.00980             osd.0      up  1.00000 1.00000 
  3   hdd 0.00980             osd.3      up  1.00000 1.00000 
-10       0.01959     rack rack02                            
 -5       0.01959         host node2                         
  1   hdd 0.00980             osd.1      up  1.00000 1.00000 
  4   hdd 0.00980             osd.4      up  1.00000 1.00000 
-11       0.01959     rack rack03                            
 -7       0.01959         host node3                         
  2   hdd 0.00980             osd.2      up  1.00000 1.00000 
  5   hdd 0.00980             osd.5      up  1.00000 1.00000 
 

 

会看到新的布局已产生,所有host都位于特定rack下面。按此操作,就完成了对CRUSH布局的调整。

对一个已知对象,可以根据CRUSH算法,查找它的存储结构。比如data这个pool里有一个文件test.txt:

 

[root@node3 ~]# echo "this is test! ">>test.txt
[root@node3 ~]# rados -p data ls
[root@node3 ~]# rados -p data put test.txt test.txt 
[root@node3 ~]# rados -p data ls
test.txt

 

 

显示它的存储结构:

 

[root@node3 ~]# ceph osd map data test.txt 
osdmap e42 pool 'data' (1) object 'test.txt' -> pg 1.8b0b6108 (1.8) -> up ([3,4,2], p3) acting ([3,4,2], p3)

 

 

crushmap与ceph的存储架构有关,在实际中可能需要经常调整它。如下先把它dump出来,再反编译成明文进行查看。

 

 
[root@node3 ~]# ceph osd getcrushmap -o crushmap
22
[root@node3 ~]# crushtool -d crushmap -o crushmap
[root@node3 ~]# cat crushmap 
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host node1 {
        id -3       # do not change unnecessarily
        id -4 class hdd         # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.0 weight 0.010
        item osd.3 weight 0.010
}
rack rack01 {
        id -9       # do not change unnecessarily
        id -14 class hdd                # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item node1 weight 0.020
}
host node2 {
        id -5       # do not change unnecessarily
        id -6 class hdd         # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.1 weight 0.010
        item osd.4 weight 0.010
}
rack rack02 {
        id -10      # do not change unnecessarily
        id -13 class hdd                # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item node2 weight 0.020
}
host node3 {
        id -7       # do not change unnecessarily
        id -8 class hdd         # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.2 weight 0.010
        item osd.5 weight 0.010
}
rack rack03 {
        id -11      # do not change unnecessarily
        id -12 class hdd                # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item node3 weight 0.020
}
root default {
        id -1       # do not change unnecessarily
        id -2 class hdd         # do not change unnecessarily
        # weight 0.059
        alg straw2
        hash 0  # rjenkins1
        item rack01 weight 0.020
        item rack02 weight 0.020
        item rack03 weight 0.020
}

# rules
rule replicated_rule {
        id 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}

# end crush map
 

 

 

这个文件包括几节,大概说明下:

 

 
    crushmap设备:见上述文件#device后面的内容。这里列举ceph的OSD列表。不管新增还是删除OSD,这个列表会自动更新。通常你无需更改此处,ceph会自动维护。

    crushmap bucket类型:见上述文件#types后面的内容。定义bucket的类型,包括root、datacenter、room、row、rack、host、osd等。默认的bucket类型对大部分ceph集群来说够用了,不过你也可以增加自己的类型。

    crushmap bucket定义:见上述文件#buckets后面的内容。这里定义bucket的层次性架构,也可以定义bucket所使用的算法类型。

    crushmap规则:见上述文件#rules后面的内容。它定义pool里存储的数据应该选择哪个相应的bucket。对较大的集群来说,有多个pool,每个pool有它自己的选择规则。
 

 

 

crushmap应用的实际场景中,我们可以定义一个pool名字为SSD,它使用SSD磁盘来提高性能。再定义一个pool名字为SATA,它使用SATA磁盘来获取更好的经济性。假设有3个ceph存储node,每个node上都有独立的osd服务。

首先在crushmap文件里修改root default为:

 

 
root default {
        id -1           # do not change unnecessarily
        id -2 class hdd         # do not change unnecessarily
        # weight 0.059
        alg straw2
        hash 0  # rjenkins1
        item rack01 weight 0.020
}
 

 

主要修改其item,删除item rack02 weight 0.020,item rack03 weight 0.020 的内容

 

并增加如下内容:

 

 
root ssd {
        id -15
        alg straw
        hash 0
        item rack02 weight 0.020

} 
root sata {
        id -16
        alg straw
        hash 0
        item rack03 weight 0.020

}

# rules
rule replicated_rule {
        id 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule ssd-pool {
        ruleset 1
        type replicated
        min_size 1
        max_size 10
        step take ssd
        step chooseleaf firstn 0 type osd
        step emit
}

rule sata-pool {
        ruleset 2
        type replicated
        min_size 1
        max_size 10
        step take sata
        step chooseleaf firstn 0 type osd
        step emit
 

 

ruleset 2这个规则里,step take sata表示优先选择sata的bucket
ruleset 1这个规则里,step take ssd表示优先选择ssd的bucket
需要注意的就是bucket id不要重复就好

编译文件,并上传到集群:

 

[root@node3 ~]# crushtool -c crushmap -o crushmap.new
[root@node3 ~]# ceph osd setcrushmap -i crushmap.new
23

 

 

 

再次查看此时的集群布局:

 

 
[root@node3 ~]# ceph osd tree
ID  CLASS WEIGHT  TYPE NAME          STATUS REWEIGHT PRI-AFF 
-16       0.01999 root sata                                  
-11       0.01999     rack rack03                            
 -7       0.01999         host node3                         
  2   hdd 0.00999             osd.2      up  1.00000 1.00000 
  5   hdd 0.00999             osd.5      up  1.00000 1.00000 
-15       0.01999 root ssd                                   
-10       0.01999     rack rack02                            
 -5       0.01999         host node2                         
  1   hdd 0.00999             osd.1      up  1.00000 1.00000 
  4   hdd 0.00999             osd.4      up  1.00000 1.00000 
 -1       0.01999 root default                               
 -9       0.01999     rack rack01                            
 -3       0.01999         host node1                         
  0   hdd 0.00999             osd.0      up  1.00000 1.00000 
  3   hdd 0.00999             osd.3      up  1.00000 1.00000 
 

 

 

 

接下来观察ceph -s是否健康状态OK。如果健康OK,增加2个pool:

 

[root@node3 ~]# ceph osd pool create sata 64 64
pool 'sata' created
[root@node3 ~]# ceph osd pool create ssd 64 64
pool 'ssd' created

 

 

给上述2个新创建的pool分配crush规则:

 

[root@node3 ~]# ceph osd pool set sata crush_rule sata-pool
set pool 2 crush_rule to sata-pool
[root@node3 ~]# ceph osd pool set ssd crush_rule ssd-pool
set pool 3 crush_rule to ssd-pool

 

 

查看规则是否生效:

 

[root@node3 ~]# ceph osd dump |egrep -i "ssd|sata"
pool 2 'sata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 55 flags hashpspool stripe_width 0
pool 3 'ssd' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 60 flags hashpspool stripe_width 0

 

现在写往sata pool的目标,将优先存储到SATA设备上。写往ssd pool的目标,将优先存储到SSD设备上。

 

 

用rados命令进行测试:

 

[root@node3 ~]# touch file.ssd
[root@node3 ~]# touch file.sata
[root@node3 ~]# rados -p ssd put filename file.ssd
[root@node3 ~]# rados -p sata put filename file.sata

 

 

最后使用ceph osd map命令检查它们的存储位置:

 

[root@node3 ~]# ceph osd map ssd file.ssd 
osdmap e69 pool 'ssd' (3) object 'file.ssd' -> pg 3.46b33220 (3.20) -> up ([4,1], p4) acting ([4,1,0], p4)
[root@node3 ~]# ceph osd map sata file.sata
osdmap e69 pool 'sata' (2) object 'file.sata' -> pg 2.df856dd1 (2.11) -> up ([5,2], p5) acting ([5,2,0], p5)

 

可以看到对应类型的对象优先存储到对应类型的设备上

 

 

二、Crush class实验

 

 

luminous版本的ceph新增了一个功能crush class,这个功能又可以称为磁盘智能分组。因为这个功能就是根据磁盘类型自动的进行属性的关联,然后进行分类。无需手动修改crushmap,极大的减少了人为的操作。以前的操作有多麻烦可以看看:ceph crushmap

ceph中的每个osd设备都可以选择一个class类型与之关联,默认情况下,在创建osd的时候会自动识别设备类型,然后设置该设备为相应的类。通常有三种class类型:hdd,ssd,nvme。

由于当前实验环境下没有ssd和nvme设备,只好修改class标签,假装为有ssd设备,然后进行实验。

 

一,实验环境

[root@node3 ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
[root@node3 ~]# ceph -v
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)

 

二,修改crush class:

1,查看当前集群布局:

 
[root@node3 ~]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF 
-1       0.05878 root default                           
-3       0.01959     host node1                         
 0   hdd 0.00980         osd.0      up  1.00000 1.00000 
 3   hdd 0.00980         osd.3      up  1.00000 1.00000 
-5       0.01959     host node2                         
 1   hdd 0.00980         osd.1      up  1.00000 1.00000 
 4   hdd 0.00980         osd.4      up  1.00000 1.00000 
-7       0.01959     host node3                         
 2   hdd 0.00980         osd.2      up  1.00000 1.00000 
 5   hdd 0.00980         osd.5      up  1.00000 1.00000 
 

 

 

可以看到只有第二列为CLASS,只有hdd类型。
通过查看crush class,确实只有hdd类型

 

[root@node3 ~]# ceph osd crush class ls
[
    "hdd"
]

 

 

2,删除osd.0,osd.1,osd.2的class:

 

[root@node3 ~]# for i in 0 1 2;do ceph osd crush rm-device-class osd.$i;done
done removing class of osd(s): 0
done removing class of osd(s): 1
done removing class of osd(s): 2

 

 

再次通过命令ceph osd tree查看osd.0,osd.1,osd.2的class

 
[root@node3 ~]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF 
-1       0.05878 root default                           
-3       0.01959     host node1                         
 0       0.00980         osd.0      up  1.00000 1.00000 
 3   hdd 0.00980         osd.3      up  1.00000 1.00000 
-5       0.01959     host node2                         
 1       0.00980         osd.1      up  1.00000 1.00000 
 4   hdd 0.00980         osd.4      up  1.00000 1.00000 
-7       0.01959     host node3                         
 2       0.00980         osd.2      up  1.00000 1.00000 
 5   hdd 0.00980         osd.5      up  1.00000 1.00000 
 

 

可以发现osd.0,osd.1,osd.2的class为空

3,设置osd.0,osd.1,osd.2的class为ssd:

 

[root@node3 ~]# for i in 0 1 2;do ceph osd crush set-device-class ssd osd.$i;done
set osd(s) 0 to class 'ssd'
set osd(s) 1 to class 'ssd'
set osd(s) 2 to class 'ssd'

 

 

再次通过命令ceph osd tree查看osd.0,osd.1,osd.2的class

 

 
[root@node3 ~]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF 
-1       0.05878 root default                           
-3       0.01959     host node1                         
 3   hdd 0.00980         osd.3      up  1.00000 1.00000 
 0   ssd 0.00980         osd.0      up  1.00000 1.00000 
-5       0.01959     host node2                         
 4   hdd 0.00980         osd.4      up  1.00000 1.00000 
 1   ssd 0.00980         osd.1      up  1.00000 1.00000 
-7       0.01959     host node3                         
 5   hdd 0.00980         osd.5      up  1.00000 1.00000 
 2   ssd 0.00980         osd.2      up  1.00000 1.00000 
 

 

 

可以看到osd.0,osd.1,osd.2的class变为ssd
再查看一下crush class:

[root@node3 ~]# ceph osd crush class ls
[
    "hdd",
    "ssd"
]

 

可以看到class中多出了一个名为ssd的class

4,创建一个优先使用ssd设备的crush rule:

创建了一个rule的名字为:rule-ssd,在root名为default下的rule

[root@node3 ~]# ceph osd crush rule create-replicated rule-ssd default  host ssd 

 

查看集群的rule:

[root@node3 ~]# ceph osd crush rule ls
replicated_rule
rule-ssd

 

可以看到多出了一个名为rule-ssd的rule
通过下面的命令下载集群crushmap查看有哪些变化:

 
[root@node3 ~]# ceph osd getcrushmap -o crushmap
20
[root@node3 ~]# crushtool -d crushmap -o crushmap
[root@node3 ~]# cat crushmap
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host node1 {
        id -3           # do not change unnecessarily
        id -4 class hdd         # do not change unnecessarily
        id -9 class ssd         # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.0 weight 0.010
        item osd.3 weight 0.010
}
host node2 {
        id -5           # do not change unnecessarily
        id -6 class hdd         # do not change unnecessarily
        id -10 class ssd                # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.1 weight 0.010
        item osd.4 weight 0.010
}
host node3 {
        id -7           # do not change unnecessarily
        id -8 class hdd         # do not change unnecessarily
        id -11 class ssd                # do not change unnecessarily
        # weight 0.020
        alg straw2
        hash 0  # rjenkins1
        item osd.2 weight 0.010
        item osd.5 weight 0.010
}
root default {
        id -1           # do not change unnecessarily
        id -2 class hdd         # do not change unnecessarily
        id -12 class ssd                # do not change unnecessarily
        # weight 0.059
        alg straw2
        hash 0  # rjenkins1
        item node1 weight 0.020
        item node2 weight 0.020
        item node3 weight 0.020
}

# rules
rule replicated_rule {
        id 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule rule-ssd {
        id 1
        type replicated
        min_size 1
        max_size 10
        step take default class ssd
        step chooseleaf firstn 0 type host
        step emit
}

# end crush map
 

 

可以看到在root default下多了一行: id -12 class ssd。在rules下,多了一个rule rule-ssd其id为1

5,创建一个使用该rule-ssd规则的存储池:

[root@node3 ~]# ceph osd pool create ssdpool 64 64 rule-ssd
pool 'ssdpool' created

 

查看ssdpool的信息可以看到使用的crush_rule 为1,也就是rule-ssd

[root@node3 ~]# ceph osd pool ls detail
pool 1 'ssdpool' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 39 flags hashpspool stripe_width 0

 

6,创建对象测试ssdpool:

创建一个对象test并放到ssdpool中:

[root@node3 ~]# rados -p ssdpool ls
[root@node3 ~]# echo "hahah" >test.txt
[root@node3 ~]# rados -p ssdpool put test test.txt 
[root@node3 ~]# rados -p ssdpool ls
test

 

查看该对象的osd组:

[root@node3 ~]# ceph osd map ssdpool test
osdmap e46 pool 'ssdpool' (1) object 'test' -> pg 1.40e8aab5 (1.35) -> up ([1,2,0], p1) acting ([1,2,0], p1)

 

可以看到该对象的osd组使用的都是ssd磁盘,至此验证成功。可以看出crush class相当于一个辨别磁盘类型的标签。

 

参考:

https://www.cnblogs.com/sisimi/p/7799980.html

https://www.cnblogs.com/sisimi/p/7804138.html

 https://www.bladewan.com/2017/08/05/crush/

 

上一篇:2019阿里云双11活动-MySQL 8.0 IoT固件安全检测(包年)包年优惠


下一篇:TensorFlow 中文资源精选,官方网站,安装教程,入门教程,实战项目,学习路径。