redis cluster搭建

2022-01-30 03:06:57

从redis官网看最新稳定版本已经更新到3.2.6了，我这里补充一篇以前3.2.0的集群配置。
说明：

redis从3.0版本以后开始支持cluster模式，原理和功能大家可以上网查，对于运维人员来说最重要的去掉了代理层、避免出现单点故障，但是redis cluster至少需要三个主节点和三个从节点。每一对主从形成一组关系。因此需要三台服务器，每台服务器上两个实例。因为一个主对应一个从，所以一台服务器上不能同时是一个主和从，必须三台服务器分开主从，来保证服务的正常运行。我这里为了演示方便就在一台服务器上部署。注意：生产环境必须是三台机器，不然达不到高可用。

部署环境：

主机名

IP地址

操作系统版本

用途

test01

192.168.2.9

CentOS6.3（64位）

实例 7000/7001

test02

192.168.2.10

CentOS 6.3（64位）

实例 7002/7003

test03

192.168.2.7

CentOS 6.3（64位）

实例 7004/7005

部署步骤：

1、编译安装redis。

把安装包放在/data/下解压并编译

点击(此处)折叠或打开

  1. [root@test01 data]# tar -zxf redis-3.2.0.tar.gz
  2. [root@test01 redis]# cd redis-3.2.0
  3. [root@test01 redis-3.2.0]# make && make install
  4. [root@test01 redis]# ln -s /data/redis/redis-3.2.0 /usr/local/redis
2、创建redis集群节点

点击(此处)折叠或打开

  1. [root@test01 local]# mkdir redis_cluster
  2. [root@test01 local]# cd redis_cluster/
  3. [root@test01 redis_cluster]# mkdir 7000 7001
3、复制默认的节点配置文件到集群节点中

点击(此处)折叠或打开

  1. [root@test01 redis_cluster]# cp /usr/local/redis/redis.conf ./7000

4、修改默认配置文件

点击(此处)折叠或打开

  1. [root@test01 7000]# vim redis.conf
  2. daemonize yes //redis后台运行
  3. pidfile /var/run/redis_7000.pid //pidfile文件对应7000
  4. port 7000 //端口7000
  5. cluster-enabled yes //开启集群 把注释#去掉
  6. cluster-config-file nodes.conf //集群的配置 配置文件首次启动自动生成
  7. cluster-node-timeout 6000 //请求超时
  8. appendonly yes //日志开启，环境测试开启，正式环境建议关闭
  9. bind 192.168.2.9 //修改监听的地址为本机地址

5、修改完后把配置文件复制到7001上并修改相应的端口。

点击(此处)折叠或打开

  1. [root@test01 7000]# cp redis.conf ../7001/

6、另外两台服务器重复上面配置。

7、启动各个节点，注意一定要到端口目录下去启动，不然有的启动不起来。

点击(此处)折叠或打开

  1. [root@test01 7000]# /usr/local/redis/src/redis-server redis.conf
  2. [root@test01 7000]# cd ../7001/
  3. [root@test01 7001]# /usr/local/redis/src/redis-server redis.conf

查看启动进程

点击(此处)折叠或打开

  1. [root@test01 7001]# ps -ef|grep redis
  2. root 8858 1 0 11:38 ? 00:00:00 /usr/local/redis/src/redis-server 192.168.2.9:7000 [cluster]
  3. root 8865 1 0 11:39 ? 00:00:00 /usr/local/redis/src/redis-server 192.168.2.9:7001 [cluster]
  4. root 8870 27799 0 11:39 pts/1 00:00:00 grep redis

再分别启动其余服务器节点。

8、创建集群必须的软件，三台服务器都执行

点击(此处)折叠或打开

  1. [root@test01 7000]# yum -y install ruby ruby-devel rubygems rpm-build
  2. [root@test01 7000]# ruby -v
  3. ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
  4. [root@redis 7000]# rpm -qa|grep rubyge
  5. rubygems-1.3.7-5.el6.noarch
  6. [root@redis 7000]# gem install redis
  7. Successfully installed redis-3.3.0
  8. 1 gem installed
  9. Installing ri documentation for redis-3.3.0...
  10. Installing RDoc documentation for redis-3.3.0.
9、确认所有的节点都启动，接下来使用参数create 创建

点击(此处)折叠或打开

  1. [root@test01 7000]# /usr/local/redis/src/redis-trib.rb create --replicas 1 192.168.2.9:7000 192.168.2.10:7002 192.168.2.7:7004 192.168.2.10:7003 192.168.2.7:7005 192.168.2.9:7001
  2. >>> Creating cluster
  3. >>> Performing hash slots allocation on 6 nodes...
  4. Using 3 masters:
  5. 192.168.2.7:7004
  6. 192.168.2.9:7000
  7. 192.168.2.10:7002
  8. Adding replica 192.168.2.9:7001 to 192.168.2.7:7004
  9. Adding replica 192.168.2.7:7005 to 192.168.2.9:7000
  10. Adding replica 192.168.2.10:7003 to 192.168.2.10:7002
  11. M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
  12. slots:5461-10922 (5462 slots) master
  13. M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
  14. slots:10923-16383 (5461 slots) master
  15. M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
  16. slots:0-5460 (5461 slots) master
  17. S: 6f13ca12a9be3b0c093d02c81fed337307f295af 192.168.2.10:7003
  18. replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
  19. S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
  20. replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
  21. S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
  22. replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
  23. Can I set the above configuration? (type 'yes' to accept): yes
  24. >>> Nodes configuration updated
  25. >>> Assign a different config epoch to each node
  26. >>> Sending CLUSTER MEET messages to join the cluster
  27. Waiting for the cluster to join...
  28. >>> Performing Cluster Check (using node 192.168.2.9:7000)
  29. M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
  30. slots:5461-10922 (5462 slots) master
  31. M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
  32. slots:10923-16383 (5461 slots) master
  33. M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
  34. slots:0-5460 (5461 slots) master
  35. M: 6f13ca12a9be3b0c093d02c81fed337307f295af 192.168.2.10:7003
  36. slots: (0 slots) master
  37. replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
  38. M: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
  39. slots: (0 slots) master
  40. replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
  41. M: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
  42. slots: (0 slots) master
  43. replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
  44. [OK] All nodes agree about slots configuration.
  45. >>> Check for open slots...
  46. >>> Check slots coverage...
  47. [OK] All 16384 slots covered.

--replicas 1  创建集群指定一个从节点。

因为创建集群的时候不能指定主从，它是随机的。加入集群的节点顺序不分前后，如果一台服务器上正好有主节点和从节点，而且是一组的话，必须将同一台服务器上的主从节点踢掉，换成不同服务器的节点。
从上面可以看到很不辛7003和7002分配了一台服务器上。经过测试一般都会有一台服务器上存在同一组的主和从节点。
主从关系图：
192.168.2.10:7002        192.168.2.10:7003
192.168.2.9:7000         192.168.2.7:7005
192.168.2.7:7004         192.168.2.9:7001

10、我们把192.168.2.10:7003和192.168.2.7:7005调换下。

删除7003节点

点击(此处)折叠或打开

  1. [root@test01 7000]# /usr/local/redis/src/redis-trib.rb del-node 192.168.2.10:7003 6f13ca12a9be3b0c093d02c81fed337307f295af
  2. >>> Removing node 6f13ca12a9be3b0c093d02c81fed337307f295af from cluster 192.168.2.10:7003
  3. >>> Sending CLUSTER FORGET messages to the cluster...
  4. >>> SHUTDOWN the node.

del-node  删除节点后面跟主机和对应的端口还有实例的id。这样才能删除。

11、修改192.168.2.7:7005的主为192.168.2.10:7002

进入192.168.2.7:7005的redis

点击(此处)折叠或打开

  1. [root@test02 redis_cluster]# /usr/local/redis/src/redis-cli -c -p 7005 -h 192.168.2.7
  2. 192.168.2.7:7005> cluster replicate 01594f84df9e743a74a47f9aaa58fa41402dfe25 #新主的id
  3. OK

查看现在主从状态

点击(此处)折叠或打开

  1. [root@test02 redis_cluster]# /usr/local/redis/src/redis-trib.rb check 192.168.2.10:7002
  2. >>> Performing Cluster Check (using node 192.168.2.10:7002)
  3. M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
  4. slots:10923-16383 (5461 slots) master
  5. 1 additional replica(s)
  6. M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
  7. slots:5461-10922 (5462 slots) master
  8. 0 additional replica(s)
  9. M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
  10. slots:0-5460 (5461 slots) master
  11. 1 additional replica(s)
  12. S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
  13. slots: (0 slots) slave
  14. replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
  15. S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
  16. slots: (0 slots) slave
  17. replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
  18. [OK] All nodes agree about slots configuration.
  19. >>> Check for open slots...
  20. >>> Check slots coverage...
  21. [OK] All 16384 slots covered.

可以看到主从关系已经改变。

12、再添加刚删除的7003节点。

点击(此处)折叠或打开

  1. [root@test02 redis_cluster]# ps -ef|grep redis
  2. root 4441 1 0 11:45 ? 00:00:13 /usr/local/redis/src/redis-server 192.168.2.10:7002 [cluster]
  3. root 4673 4275 0 15:02 pts/0 00:00:00 grep redis
  4. [root@test02 redis_cluster]# cd 7003/
  5. [root@test02 7003]# ls
  6. appendonly.aof dump.rdb nodes.conf redis.conf
  7. [root@test02 7003]# rm -f appendonly.aof dump.rdb nodes.conf
  8. [root@test02 7003]# /usr/local/redis/src/redis-server redis.conf
  9. [root@test02 7003]# ps -ef|grep redis
  10. root 4441 1 0 11:45 ? 00:00:14 /usr/local/redis/src/redis-server 192.168.2.10:7002 [cluster]
  11. root 4677 1 0 15:02 ? 00:00:00 /usr/local/redis/src/redis-server 192.168.2.10:7003 [cluster]
  12. root 4681 4275 0 15:02 pts/0 00:00:00 grep redis

13、执行下列命令添加从指定的主节点。

点击(此处)折叠或打开

  1. [root@test02 7003]# /usr/local/redis/src/redis-trib.rb add-node --slave --master-id bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.10:7003 192.168.2.9:7000
  2. >>> Adding node 192.168.2.7:7005 to cluster 192.168.2.9:7000
  3. >>> Performing Cluster Check (using node 192.168.2.9:7000)
  4. M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
  5. slots:5461-10922 (5462 slots) master
  6. 0 additional replica(s)
  7. S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
  8. slots: (0 slots) slave
  9. replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
  10. S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
  11. slots: (0 slots) slave
  12. replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
  13. M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
  14. slots:0-5460 (5461 slots) master
  15. 1 additional replica(s)
  16. M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
  17. slots:10923-16383 (5461 slots) master
  18. 1 additional replica(s)
  19. [OK] All nodes agree about slots configuration.
  20. >>> Check for open slots...
  21. >>> Check slots coverage...
  22. [OK] All 16384 slots covered.
  23. >>> Send CLUSTER MEET to node 192.168.2.10:7003 to make it join the cluster.
  24. Waiting for the cluster to join.
  25. >>> Configure node as replica of 192.168.2.9:7000.
  26. [OK] New node added correctly.

从上面显示7003节点成为7000的从节点。

14、查看主从现在情况

点击(此处)折叠或打开

  1. [root@test02 7003]# /usr/local/redis/src/redis-trib.rb check 192.168.2.10:7002
  2. >>> Performing Cluster Check (using node 192.168.2.10:7002)
  3. M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
  4. slots:10923-16383 (5461 slots) master
  5. 1 additional replica(s)
  6. M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
  7. slots:5461-10922 (5462 slots) master
  8. 1 additional replica(s)
  9. S: 2e26f92f8dca2dfb13f159a59b6260f106bc8cb4 192.168.2.10:7003
  10. slots: (0 slots) slave
  11. replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
  12. M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
  13. slots:0-5460 (5461 slots) master
  14. 1 additional replica(s)
  15. S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
  16. slots: (0 slots) slave
  17. replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
  18. S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
  19. slots: (0 slots) slave
  20. replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
  21. [OK] All nodes agree about slots configuration.
  22. >>> Check for open slots...
  23. >>> Check slots coverage...
  24. [OK] All 16384 slots covered
这样就是一个正常的集群了~~运维人员再也不会因为宕机或者redis挂了的问题头疼了！













上一篇：sentinel搭建redis集群经验总结


下一篇：ArcEngine 创建工作空间工厂对象IWorkSpaceFactory的两种方式