Redis Cluster是一种服务器sharding分片技术，关于Redis的集群方案应该怎么做，请参考我的另一篇博客http://www.cnblogs.com/xckk/p/6134655.html
本文主要介绍Redis Cluster集群的工作原理，详细讲解了Redis Cluster集群如何搭建与配置。
一、redis安装
redis官网下载后是源码包，需要make安装。
1、解压redis-3.2.5.tar.gz
2、cd redis-3.2.5
3、make
4、cd src/
5、make install
安装完成，src包下面新增redis-server、redis-cli等这些可执行脚本，表示安装完成。
二、启动实例集群配置
搭建Redis Cluster，建立6个redis实例，3主3从，端口号分别为7000-7005
创建以6个端口命名的子目录，之后我们将在6个子目录上都开启Redis实例
cd redis-3.2. mkdir cluster-test cd cluster-test mkdir
Redis实例配置
在7000到7005每一个目录中都创建redis.conf文件，配置项如下：
cd redis-3.2. mkdir cluster-test cd cluster-test mkdir
各配置描述如下：
|
配置项redis.conf |
示例 |
备注 |
| port | 7000 | 端口号 |
| cluster-enabled | yes |
yes，表示支持Redis Cluster |
| cluster-config-file | nodes-7000.conf |
Redis Cluster记录的启动信息文件，文件由cluster自动生成，不需要用户编辑 |
| cluster-node-timeout | 5000 | 毫秒单位，失效时间 |
| appendonly | yes | 支持appendonly持久化方式 |
| protected-mode | no | 默认是打开的，此处关闭。因为jedis访问时，如果是保护模式，影响访问 |
| bind | 192.168.121.130 | 机器IP，此处最好写IP，不要写localhost或127.0.0.1，可能影响jedis客户端访问 |
Redis实例启动
将第一步安装生成的redis-server拷贝到cluster-test目录，分别后台启动6个redis实例
nohup ./redis-server /redis.conf & nohup ./redis-server /redis.conf & nohup ./redis-server /redis.conf & nohup ./redis-server /redis.conf & nohup ./redis-server /redis.conf & nohup ./redis-server /redis.conf &
启动完后，cluster-test目录下生成了nodes-7000.conf等启动文件信息。
查看进程启动成功：[root@centos2 cluster-test]# ps -ef|grep redis
三、创建集群
6个redis实例已经创建，它们目前是相互独立的，没有关联。接下来是创建Redis Cluster集群，将6个实例组成一个集群。
redis-trib命令创建集群
src目录下make生成的redis-trib命令，可用于Redis Cluster集群创建的一些脚本启动，它是一个ruby程序。因此我们需要配置ruby环境
安装ruby
yum install ruby
yum install ruby-irb
具体ruby的安装可参考：http://jingyan.baidu.com/article/b7001fe173fe9a0e7382dd57.html
安装Gem
Gem是一个管理Ruby库和程序的标准包，它通过RubyGem（如 http://rubygems.org/ ）源来查找、安装、升级和卸载软件包，非常的便捷。
配置了ruby环境后，还需要安装redis整个包才能够运行redis-trib。安装过程中，需要先安装gem库，不然会出现gem command not found错误。
[root@centos2 src]# gem install redis -bash: gem: command not found
安装gem过程中，可能出现以下问题
yum install rubygems ERROR: Could not find a valid gem 'redis' (>= ) in any repository
问题原因：无法连接gem服务器，原因是该gem服务器被墙了。按如下步骤手动下载安装。
wget https://rubygems.global.ssl.fastly.net/gems/redis-3.2.1.gem gem install -l ./redis-3.2..gem
安装完后，src包下就会生成redis-trib.rb文件
启动命令
./redis-trib.rb create --replicas 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130:
启动后，各Redis实例之间分别会新增一个端口用于通信，通信端口是Redis实例端口+1W，分别为17000，17001，17002，17003，17004，17005
命令的解释如下：
1、给定的命令将会被ruby程序翻译为create，这表示我们想要创建一个新的集群。
2、选项--replicas 1意思是为每一个master节点创建一个slave节点。
3、其他参数表示redis实例的地址及端口，以空格为间隔。
启动日志分析
出现[OK] All 16384 slots covered.表示启动成功。从启动日志可以看出，master是7000,7001,7002，对应的slave是7003,7004,7005
[root@centos1 src]# ./redis-trib.rb create --replicas 127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1: >>> Creating cluster >>> Performing hash slots allocation on nodes... Using masters: 127.0.0.1: 127.0.0.1: 127.0.0.1: Adding replica 127.0.0.1: to 127.0.0.1: Adding replica 127.0.0.1: to 127.0.0.1: Adding replica 127.0.0.1: to 127.0.0.1: M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1: slots:- ( slots) master M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1: slots:- ( slots) master M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1: slots:- ( slots) master S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1: replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1: replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1: replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:) M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1: slots:- ( slots) master additional replica(s) S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1: slots: ( slots) slave replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1: slots: ( slots) slave replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1: slots:- ( slots) master additional replica(s) S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1: slots: ( slots) slave replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1: slots:- ( slots) master additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All slots covered.
集群启动状态检测
我们可以通过./redis-trib.rb check 192.168.121.130:7000来检测Redis Cluster的启动状态。
[root@centos2 src]# ./redis-trib.rb check 192.168.121.130: >>> Performing Cluster Check (using node 192.168.121.130:) M: dca47b274799927fd15aa4e6312b94752418f0c0 192.168.121.130: slots:- ( slots) master additional replica(s) M: fe1082655e832bd2afb7d4bfabfec3ce2354868b 192.168.121.130: slots:- ( slots) master additional replica(s) S: 7fe93723c48b7ea33bbf21f12e97d18bdd0221ea 192.168.121.130: slots: ( slots) slave replicates dca47b274799927fd15aa4e6312b94752418f0c0 M: 59c926f7bb1e24e293d379507742c52b6dd08cc2 192.168.121.130: slots:- ( slots) master additional replica(s) S: a554424a9969af2b55f4e11f9eedf1bc3bce9000 192.168.121.130: slots: ( slots) slave replicates 59c926f7bb1e24e293d379507742c52b6dd08cc2 S: 065fb213e9db64627c27af93500420351b80ac34 192.168.121.130: slots: ( slots) slave replicates fe1082655e832bd2afb7d4bfabfec3ce2354868b [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All slots covered.
登录集群
通过命令./redis-cli -c -h 192.168.121.129 -p 7000用来登录
重定向
从以下命令可以看出，我们登录的是192.168.121.130机器的7000端口Redis实例，set foo 123，foo经过hash后落到了12182槽，12182槽存储在192.168.121.130机器的7002端口实例，因此日志显示重定向到7002端口的实例。
[root@centos2 src]# ./redis-cli -c -h 192.168.121.130 -p 192.168.121.130:> set foo -> Redirected to slot [] located at 192.168.121.130: OK 192.168.121.130:> get foo "" 192.168.121.130:>
redis cluster的主从同步采用异步复制（提高性能），因此会存在部分写内容丢失的情况
四、创建集群-通过create-cluster脚本
通过redis-trib.rb命令创建集群涉及命令较多，稍显复杂。好在redis提供了create-cluster脚本utils/create-cluster/create-cluster来创建集群，方便很多。
create-cluster命令介绍
从create-cluster源码看，其实现原理也是利用了redis-trib.rb命令。该脚本默认创建3个master节点，3个salve节点。
1、create-cluster start
创建6个redis实例，默认端口号是30001-30006，如果想改端口号,在create-cluster里面更改PORT值即可。
如果想要端口号是7000-7005，设置PORT=6999即可
2、create-cluster create
创建集群，3个master,3个slave
3、create-cluster stop
关闭Redis Cluster集群和Redis实例
4、create-cluster clean
清除所在Redis实例的data, logs, config文件
使用方法
cd /home/redis-3.2/utils/create-cluster --进入目录
./create-cluster clean ---先清除Redis实例的文件
./create-cluster start ---启动Redis实例
./create-cluster create ---创建Redis Cluster集群
启动完后，使用redis-cli命令登录，即可正常访问。
[root@centos1 create-cluster]# redis-cli -c -p 127.0.0.1:> set foo hello -> Redirected to slot [] located at 127.0.0.1: OK 127.0.0.1:> set hello world -> Redirected to slot [] located at 127.0.0.1: OK 127.0.0.1:> get foo -> Redirected to slot [] located at 127.0.0.1: "hello" 127.0.0.1:> get hello -> Redirected to slot [] located at 127.0.0.1: "world" 127.0.0.1:>
五、Jedis测试程序
Jedis是Redis官方首选的 Java 客户端开发包。我们通过Junit进行单元测试
maven添加jedis和junit依赖包
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency> <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.8.1</version>
</dependency> </dependencies>
java代码
package com.hk.test.helloWorld; import java.util.HashSet;
import java.util.List;
import java.util.Set; import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.junit.Before;
import org.junit.Test; import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster; public class TestJedis
{
JedisCluster jc = null; @Before
public void before()
{
Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
// Jedis Cluster will attempt to discover cluster nodes automatically
jedisClusterNodes.add(new HostAndPort("centos2", 7000));
GenericObjectPoolConfig config;
jc = new JedisCluster(jedisClusterNodes);
} @Test
public void testFoo()
{
for (int i = 0; i <= 3000; i++)
{
String key = "foo" + i;
String value = String.valueOf(i);
jc.set(key, value);
}
} @Test
public void test_incr()
{ String key = "page_view";
jc.del(key);
jc.incr(key);
String result = jc.get(key);
System.out.println(result); } @Test
public void test_setAndGetStringVal()
{
String key = "foo";
String value = "bar";
jc.set(key, value);
String result = jc.get(key);
System.out.println(result); } @Test
public void test_setAndGetStringVal_and_set_expire() throws InterruptedException
{
String key = "hello";
String value = "world";
int seconds = 3;
jc.setex(key, seconds, value);
String result = jc.get(key);
System.out.println(result); Thread.sleep(seconds * 1000);
result = jc.get(key);
System.out.println(result); } @Test
public void test_setAndGetHashVal()
{ String key = "website";
String field = "google";
String value = "google.com";
jc.del(key);
jc.hset(key, field, value);
String result = jc.hget(key, field);
System.out.println(result); } @Test
public void test_setAndGetListVal()
{ String key = "mylist";
jc.del(key);
String[] vals =
{ "a", "b", "c" };
jc.rpush(key, vals);
List<String> result = jc.lrange(key, 0, -1);
System.out.println(result); } @Test
public void test_setAndGetSetVal()
{ String key = "language";
jc.del(key);
String[] members =
{ "java", "ruby", "python" };
jc.sadd(key, members);
Set<String> result = jc.smembers(key);
System.out.println(result); }
}
六、常见问题
一、Could not connect to Redis at 127.0.0.1:30001 Connection refused
原因：create-cluster脚本默认设定的端口从30001开始，由于我设置的redis端口是7000-7005，当时启动cluster时用的redis-trib，因此端口号不同，
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
解决方法：
1、create-cluster里面将PORT由30001改为6999。
2、关闭Redis实例进程
可以通过netstat -apn|grep redis或者ps -ef|grep redis查找进程号,kill -9 ｛进程号｝进行关闭。
3、create-cluster stop
4、create-cluster clean 清除data,log,conf文件
5、create-cluster start
6、create-cluster create
二、redis.clients.jedis.exceptions.JedisDataException: DENIED Redis is running in protected mode because protected mode is enabled
原因：Redis实例启动时默认设置了保护模式
解决方法：需要在启动时增加配置项 protected-mode no
1、如果是redis-trib.rb启动，启动配置文件内增加 protected-mode no
port cluster-enabled yes cluster-config-file nodes-.conf cluster-node-timeout appendonly yes protected-mode no bind 192.168.121.129
2、如果是create-cluster启动，在create-cluster脚本里start分支中增加protected-mode no
if [ "$1" == "start" ]
then
while [ $((PORT < ENDPORT)) != "" ]; do
PORT=$((PORT+))
echo "Starting $PORT"
../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --bind 192.168.121.130 --protected-mode no --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes
done
exit
fi
三、Jedis访问时，redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?
若用JedisCluster操作集群报以上错误，就是有Redis节点通讯被拒绝，最好是将redis.conf文件中的bind属性设置成redis节点所在服务器的物理ip地址，而不是localhost
1、更改node节点配置文件绑定地址
2、重启redis节点
3、重新create cluster
./redis-trib.rb create --replicas 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130:7003192.168.121.130: 192.168.121.130:
四、[ERR] Node 172.168.63.202:7001 is not empty. Either the nodealready knows other nodes (check with CLUSTER NODES) or contains some
解决方法：（一般1，2步即可解决）
1)、将需要新增的节点下aof、rdb等本地备份文件删除；
2)、同时将新Node的集群配置文件删除,即：删除你redis.conf里面cluster-config-file所在的文件；
3)、再次添加新节点如果还是报错，则登录新Node,./redis-cli–h x –p对数据库进行清除：
172.168.63.201:7001> flushdb #清空当前数据库
4)、删除后再执行第2,3步，再./create-cluster start即可
五、ERR Invalid node address specified
./redis-trib.rb create --replicas 1 centos2:7000 centos2:7001 centos2:7002 centos2:7003 centos2:7004 centos2:7005
报" ERR Invalid node address specified"
由于Redis-trib.rb 对域名或主机名支持不好,故在创建集群的时候要使用ip:port的方式
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
六、ERR Slot 4648 is already busy (Redis::CommandError)
这是由于之前创建集群没有成功,需要
1、将nodes.conf和dir里面的文件全部删除(注意不要删除了redis.conf)
2、将redis实例关闭后重新启动
3、再执行/redis-trib.rb create 。。。脚本
Sorry, the cluster configuration file nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration files.
七、[ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
解决方法：先stop和clean，再start和create
./create-cluster stop
./create-cluster clean
./create-cluster start
./create-cluster create
秀才坤坤出品