redis数据同步之redis-shake

redis-shake简介

redis-shake是阿里开源的用于redis数据同步的工具,基本功能有:

  • 恢复restore:将RDB文件恢复到目的redis数据库。
  • 备份dump:将源redis的全量数据通过RDB文件备份起来。
  • 解析decode:对RDB文件进行读取,并以json格式解析存储。
  • 同步sync:支持源redis和目的redis的数据同步,支持全量和增量数据的迁移,支持从云下到阿里云云上的同步,也支持云下到云下不同环境的同步,支持单节点、主从版、集群版之间的互相同步。需要注意的是,如果源端是集群版,可以启动一个RedisShake,从不同的db结点进行拉取,同时源端不能开启move slot功能;对于目的端,如果是集群版,写入可以是1个或者多个db结点。
  • 同步rump:支持源redis和目的redis的数据同步,仅支持全量的迁移。采用scan和restore命令进行迁移,支持不同云厂商不同redis版本的迁移。

基本原理

redis数据同步之redis-shake

如果源端是集群模式,只需要启动一个redis-shake进行拉取,同时不能开启源端的move slot操作。如果目的端是集群模式,可以写入到一个结点,然后再进行slot的迁移,当然也可以多对多写入。
目前,redis-shake到目的端采用单链路实现,对于正常情况下,这不会成为瓶颈,但对于极端情况,qps比较大的时候,此部分性能可能成为瓶颈。另外,redis-shake到目的端的数据同步采用异步的方式,读写分离在2个线程操作,降低因为网络时延带来的同步性能下降。

全量同步阶段并发执行,增量同步阶段异步执行,能够达到毫秒级别延迟(取决于网络延迟)。同时,对大key同步进行分批拉取,优化同步性能。

用户可以通过我们提供的restful拉取metric来对redis-shake进行实时监控:curl 127.0.0.1:9320/metric。

支持

  • 支持2.8-5.0版本的同步。
  • 支持codis。
  • 支持云下到云上,云上到云上,云上到云下(阿里云目前支持主从版),其他云到阿里云等链路,帮助用户灵活构建混合云场景。

使用

1、直接下载二进制包这里
2、修改配置文件

]# tar -zxf release-v2.1.0-20210819.tar.gz
]# cd redis-shake
]# ll
total 41252
-rw-r--r-- 1  502 games    12542 Aug 25 17:50 redis-shake.conf
-rwxr-xr-x 1  502 games 11853504 Aug 19 10:11 redis-shake.darwin
-rwxr-xr-x 1  502 games 11918954 Aug 19 10:11 redis-shake.linux
-rwxr-xr-x 1  502 games 12227584 Aug 19 10:11 redis-shake.windows

修改redis-shake.conf,我的环境是从两套redis主从之间同步数据,配置文件如下(删减过部分默认配置)

conf.version = 1
id = redis-shake
log.file = /opt/redis-shake/redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
parallel = 32
source.type = standalone
source.address = 10.26.31.172:6379
source.password_raw =
source.auth_type = auth
source.tls_enable = false
source.rdb.input =
source.rdb.parallel = 0
source.rdb.special_cloud =
target.type = standalone
target.address = 10.26.29.235:6389
target.password_raw =
target.auth_type = auth
target.db = 0
key_exists = rewrite
filter.db.whitelist = 0

说明:
a、主要需要修改源redis和目的redis的信息
b、当源和目的有重复 key 时是否进行覆写, 可选值:

	1. rewrite: 源端覆盖目的端
	2. none: 一旦发生进程直接退出
	3. ignore: 保留目的端key,忽略源端的同步 key. 该值在 rump 模式下不会生效.

key_exists = rewrite
c、指定的db被通过,比如0;5;10将会使db0, db5, db10通过, 其他的被过滤
filter.db.whitelist = 0
更多详细参数参见官网

3、启动
用户配置完配置文件,然后以不同的模式启动即可:

./redis-shake -conf=redis-shake.conf -type=sync &

然后可以查看配置的日子路径,查看服务运行状态,一般正常情况下日志如下

2021/08/25 17:19:54 [WARN] source.auth_type[auth] != auth
2021/08/25 17:19:54 [WARN] target.auth_type[auth] != auth
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.29.235:6389] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] source rdb[10.26.31.172:6379] checksum[yes]
2021/08/25 17:19:54 [WARN]
______________________________
\                             \           _         ______ |
 \                             \        /   \___-=O‘/|O‘/__|
  \   RedisShake, here we go !! \_______\          / | /    )
  /                             /        ‘/-==__ _/__|/__=-|  -GM
 /        Alibaba Cloud        /         *             \ | |
/                             /                        (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ

2021/08/25 17:19:54 [INFO] redis-shake configuration: {"ConfVersion":1,"Id":"redis-shake","LogFile":"/opt/redis-shake/redis-shake.log","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"Parallel":32,"SourceType":"standalone","SourceAddress":"10.26.31.172:6379","SourcePasswordRaw":"***","SourcePasswordEncoding":"***","SourceAuthType":"auth","SourceTLSEnable":false,"SourceRdbInput":[],"SourceRdbParallel":1,"SourceRdbSpecialCloud":"","TargetAddress":"10.26.29.235:6389","TargetPasswordRaw":"***","TargetPasswordEncoding":"***","TargetDBString":"0","TargetDBMapString":"","TargetAuthType":"auth","TargetType":"standalone","TargetTLSEnable":false,"TargetRdbOutput":"local_dump","TargetVersion":"6.0.9","FakeTime":"","KeyExists":"none","FilterDBWhitelist":[],"FilterDBBlacklist":[],"FilterKeyWhitelist":[],"FilterKeyBlacklist":[],"FilterSlot":[],"FilterLua":false,"BigKeyThreshold":524288000,"Metric":true,"MetricPrintLog":false,"SenderSize":104857600,"SenderCount":4095,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","ScanKeyNumber":50,"ScanSpecialCloud":"","ScanKeyFile":"","Qps":200000,"ResumeFromBreakPoint":false,"Psync":true,"NCpu":0,"HeartbeatUrl":"","HeartbeatInterval":10,"HeartbeatExternal":"","HeartbeatNetworkInterface":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"FilterKey":null,"FilterDB":"","Rewrite":false,"SourceAddressList":["10.26.31.172:6379"],"TargetAddressList":["10.26.29.235:6389"],"SourceVersion":"4.0.11","HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetReplace":false,"TargetDB":0,"Version":"tidy-code-for-release,ba0f3e1aa6e0956bfa2973a4c894cbc1a8d3d732,go1.16.7,2021-08-19_10:11:39","Type":"sync","TargetDBMap":null}
2021/08/25 17:19:54 [INFO] DbSyncer[0] starts syncing data from 10.26.31.172:6379 to [10.26.29.235:6389] with http[9321], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync connect ‘10.26.31.172:6379‘ with auth type[auth] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync send listening port[9320] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] try to send ‘psync‘ command: run-id[?], offset[-1]
2021/08/25 17:22:30 [INFO] Event:FullSyncStart  Id:redis-shake
2021/08/25 17:22:30 [INFO] DbSyncer[0] psync runid = 52aa8214f9f8238c32f172badbb4f4ea325c6cc4, offset = 9452286340998, fullsync
2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +

日志信息

同步分为三个阶段:
1、等待源端save rdb完毕,日志如下

2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +

2、全量同步阶段,显示百分比

2021/08/25 17:28:09 [INFO] DbSyncer[0] total = 9.06GB -       3.82MB [  0%]  entry=56515
2021/08/25 17:28:10 [INFO] DbSyncer[0] total = 9.06GB -       7.67MB [  0%]  entry=114522
2021/08/25 17:28:11 [INFO] DbSyncer[0] total = 9.06GB -      11.28MB [  0%]  entry=168877

3、增量同步,出现字样sync rdb done后,当前dbSyncer进入增量同步

2021/08/26 09:40:58 [INFO] DbSyncer[0] sync:  +forwardCommands=3      +filterCommands=3      +writeBytes=33
2021/08/26 09:40:59 [INFO] DbSyncer[0] sync:  +forwardCommands=2      +filterCommands=0      +writeBytes=23
2021/08/26 09:41:00 [INFO] DbSyncer[0] sync:  +forwardCommands=1      +filterCommands=3      +writeBytes=11

注意事项

1、如果目标库的数据逐出策略(maxmemory-policy)配置为noeviction以外的值,可能导致目标库的数据与源库不一致。关于数据逐出策略详情,请参见Redis数据逐出策略介绍。
2、如果源库中的某些Key使用了过期(expire)机制,由于可能存在Key已过期但未被及时删除的情形,所以在目标库中查看(如通过info命令)到的Key数量会比源库的Key数量少。

redis数据同步之redis-shake

上一篇:TCP拥塞控制机制


下一篇:共享windows7网络时出现依赖的组件无法启动的解决办法