记一次Redis内存诡异增长

2022-01-07 04:44:36

一、现象

  • 实例名：r-bp1cxxxxxxxxxd04(主从)
  • 时间：2017-11-16 12:26~12:27
  • 问题：一分钟内存上涨了2G，如下图所示：
  • 键值规模：6000万左右

记一次Redis内存诡异增长

二、Redis内存分析

1.内存组成

上图中的内存统计的是Redis的info memory命令中的used_memory属性，例如：

redis> info memory
# Memory
used_memory:9195978072
used_memory_human:8.56G
used_memory_rss:9358786560
used_memory_peak:10190212744
used_memory_peak_human:9.49G
used_memory_lua:38912
mem_fragmentation_ratio:1.02
mem_allocator:jemalloc-3.6.0

每个属性的详细说明

属性名 属性说明
used_memory Redis 分配器分配的内存量，也就是实际存储数据的内存总量
used_memory_human 以可读格式返回 Redis 使用的内存总量
used_memory_rss 从操作系统的角度，Redis进程占用的总物理内存
used_memory_peak 内存分配器分配的最大内存，代表used_memory的历史峰值
used_memory_peak_human 以可读的格式显示内存消耗峰值
used_memory_lua Lua引擎所消耗的内存
mem_fragmentation_ratio used_memory_rss /used_memory比值，表示内存碎片率
mem_allocator Redis 所使用的内存分配器。默认: jemalloc

计算公式如下：

used_memory = 自身内存+对象内存+缓冲内存+lua内存
used_rss = used_memory + 内存碎片

如下图所示：

记一次Redis内存诡异增长

2.内存分析：

(1) 自身内存：一个空的Redis占用很小，可以忽略不计
(2) kv内存：key对象 + value对象
(3) 缓冲区：客户端缓冲区(普通 + slave伪装 + pubsub)以及aof缓冲区(比较固定，一般没问题)
(4) Lua：Lua引擎所消耗的内存

3.内存突增常见问题

(1) kv内存：bigkey、大量写入
(2) 客户端缓冲区：一般常见的有普通客户端缓冲区(例如monitor命令)或者pubsub客户端缓冲区

三、问题排查

(1) bigkey?

经扫描未发现bigkey

Sampled 67234427 keys in the keyspace!
Total key length in bytes is 1574032382 (avg len 23.41)

Biggest string found 'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209' has 20862 bytes
Biggest   list found 'CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY' has 51 items
Biggest   hash found 'CCARD_VALID_DEVICE_TRAIN_MAP_KEY' has 51 fields

67234359 strings with 71767890 bytes (100.00% of keys, avg size 1.07)
67 lists with 151 items (00.00% of keys, avg size 2.25)
0 sets with 0 members (00.00% of keys, avg size 0.00)
1 hashs with 51 fields (00.00% of keys, avg size 51.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

(2) 键值个数增加？

未发现键值有明显变化

记一次Redis内存诡异增长

(3) 客户端缓冲区

由于内存增上去后，长时间没下落，如果是因为缓冲区问题，会从info clients找到明显问题：

执行后发现：

redis> info clients
# Clients
connected_clients:43
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
admin_clients:6
rejected_vpc_conn_count:0
close_idle_unknown_conn_count:0

执行client中也没有明显的omem大于0的情况

id=80207 addr=10.xx.0.4:63920 fd=46 name= age=624 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80215 addr=10.xx.0.23:43489 fd=36 name= age=591 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80366 addr=10.xx.0.8:59785 fd=18 name= age=84 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=del read=0 write=0 type=user
id=80356 addr=10.xx.0.33:32117 fd=13 name= age=114 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80064 addr=10.xx.59.4:53446 fd=38 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin
id=80276 addr=10.xx.0.23:48511 fd=8 name= age=387 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80188 addr=10.xx.0.33:16265 fd=42 name= age=681 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80326 addr=10.xx.0.32:59779 fd=16 name= age=209 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80065 addr=10.xx.59.4:53447 fd=45 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin
id=79936 addr=10.xx.0.22:10607 fd=30 name= age=1480 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80174 addr=10.xx.0.5:60914 fd=6 name= age=722 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80300 addr=10.xx.0.22:22757 fd=48 name= age=298 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80037 addr=10.xx.0.5:55189 fd=15 name= age=1143 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80330 addr=10.xx.0.8:48533 fd=17 name= age=199 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=79896 addr=10.xx.0.30:26814 fd=11 name= age=1616 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80299 addr=10.xx.0.24:11227 fd=44 name= age=303 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80086 addr=10.xx.0.32:52526 fd=40 name= age=1002 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80202 addr=10.xx.0.33:16658 fd=26 name= age=636 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80256 addr=10.xx.0.24:60496 fd=19 name= age=448 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=79908 addr=10.xx.0.29:18975 fd=12 name= age=1583 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80365 addr=10.xx.0.29:46429 fd=14 name= age=85 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=79869 addr=10.xx.27.4:48455 fd=35 name= age=1700 idle=1700 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin
id=80334 addr=10.xx.0.23:50012 fd=39 name= age=189 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80041 addr=10.xx.0.32:51107 fd=33 name= age=1132 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=79992 addr=10.xx.0.22:12068 fd=28 name= age=1289 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80251 addr=10.xx.0.30:44213 fd=23 name= age=468 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80006 addr=10.xx.0.2:45895 fd=31 name= age=1242 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80321 addr=10.xx.0.30:48048 fd=5 name= age=224 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80381 addr=10.xx.0.8:13360 fd=22 name= age=24 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=del read=0 write=0 type=user
id=80200 addr=10.xx.0.24:59183 fd=24 name= age=640 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80113 addr=10.xx.0.2:52492 fd=21 name= age=915 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=174 addr=11.216.117.242:53027 fd=9 name= age=281390 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=replconf read=0 write=0 type=admin
id=79991 addr=10.xx.0.4:48412 fd=25 name= age=1296 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80301 addr=127.0.0.1:47869 fd=49 name= age=291 idle=261 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=strlen read=0 write=0 type=admin
id=80047 addr=10.xx.59.4:53184 fd=41 name= age=1114 idle=1114 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin
id=80236 addr=10.xx.0.5:62546 fd=47 name= age=516 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80364 addr=10.xx.0.4:18794 fd=7 name= age=85 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80175 addr=10.xx.0.4:62245 fd=29 name= age=718 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80336 addr=10.xx.0.29:45701 fd=50 name= age=180 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80050 addr=10.xx.59.4:53188 fd=43 name= age=1114 idle=1114 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin
id=79765 addr=10.xx.0.2:33832 fd=37 name= age=2027 idle=177 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info read=0 write=0 type=user
id=80170 addr=10.xx.0.2:57853 fd=20 name= age=728 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user
id=80390 addr=127.0.0.1:49449 fd=27 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client read=0 write=0 type=admin

四、揪出元凶

常用的几招都用了，还是不行，同事@径远帮忙一起分析，怀疑是不是因为Redis的kv哈希表做了rehash。

1.Redis的kv存储结构

如下图所示，Redis的所有kv保存在dict中，其中ht对应两个哈希表ht[0]和ht[1]，平时一个空闲，一个用于存储数据，只有当需要rehash时，ht[1]才会用到。

记一次Redis内存诡异增长

2.Redis的字典rehash

为了保证哈希表的负载，当哈希表的元素个数等于哈希表槽数时候，会进行rehash扩容。

扩容后h[1]的容量等于第一个大于等于ht[0].size*2的2n，例如hash表的初始化容量是4，那么下一次扩容就是8，以此类推。

3.测试

(1) 测试方法

先批量写入到rehash阈值附近，然后在逐条去写，观察内存变化

// 为每个键设置1天过期时间
int expireTime = 60 * 60 * 24;
// rehash阈值 - 50为了方便观察rehash内存变化
int rehashThreshold = (int) Math.pow(2, 25) - 50;

// 1.批量写入：pipeline批量写入，由于是本机测试，这里用10000，实际生产不要这么用
Pipeline pipeline = jedis.pipelined();
pipeline = jedis.pipelined();
for (int i = 0; i < rehashThreshold; i++) {
    pipeline.setex(String.valueOf(i), expireTime, String.valueOf(i));
    if (i % 10000 == 0) {
        pipeline.sync();
    }
}
pipeline.sync();

// 2.等待写增量
TimeUnit.SECONDS.sleep(5);
for (int i = rehashThreshold; i < rehashThreshold + 200; i++) {
    jedis.setex(String.valueOf(i), expireTime, String.valueOf(i));
    TimeUnit.SECONDS.sleep(1);
}

(2) 开始测试

(a) 当阈值=215=32768，从下面可以看出到key的个数为32769时，内存涨了一些，但是还不明显。

keys       mem      clients blocked requests            connections
32766      4.69M    3       0       32797 (+2)          4
32767      4.69M    3       0       32799 (+2)          4
32768      4.69M    3       0       32801 (+2)          4
32769      5.44M    3       0       32803 (+2)          4

(b) 当阈值=220=1048576，从下面可以看出到key的个数为1048577时，内存涨了32M。
因为rehash会扩容，所以新的哈希表中的槽位变为了221 * 2（因为每个key都设置了过期时间，expires表），指针为8个字节，221 ️ 2 ️ 8 = 225 = 32MB

keys       mem      clients blocked requests            connections
1048574    128.69M  3       0       3364129 (+2)        16
1048575    128.69M  3       0       3364131 (+2)        16
1048576    128.69M  3       0       3364133 (+2)        16
1048577    160.69M  3       0       3364135 (+2)        16
1048578    160.69M  3       0       3364137 (+2)        16

(c) 当阈值=226=67108864，从下面可以看出到key的个数为67108865时，内存涨了2GB。
因为rehash会扩容，所以新的哈希表中的槽位变为了227 * 2（因为每个key都设置了过期时间，expires表），指针为8个字节，227 ️ 2 ️ 8 = 231 = 2GB

keys       mem      clients blocked requests            connections
67108862   9.70G    3       0       70473683 (+2)       18
67108863   9.70G    3       0       70473685 (+2)       18
67108864   9.70G    3       0       70473687 (+2)       18
67108865   11.70G   3       0       70473689 (+2)       18
67108866   11.70G   3       0       70473691 (+2)       18
67108867   11.70G   3       0       70473693 (+2)       18

回过来看r-bp1c15fd9b142d04的key和内存变化图，可以发现上面的规则是正确的：

记一次Redis内存诡异增长

记一次Redis内存诡异增长

4 后续观察

17点时，rehash结束，内存降了增加的2G的一半。

记一次Redis内存诡异增长

五、总结

  • 由于哈希表的特性，Redis中键值数量大，不会对存取造成性能影响，但是会出现本文提到的问题。控制键个数有几个建议：

    • 无用的键值设置过期时间或者定期删除。
    • 优化键值设计：例如可以使用ziplist hash合并优化部分字符串类型。
  • 未来改进：内核层面支持rehash的审计日志以及增强rehash的速度。

广告

云数据库Redis版（ApsaraDB for Redis）是一种稳定可靠、性能卓越、可弹性伸缩的数据库服务。基于飞天分布式系统和全SSD盘高性能存储，支持主备版和集群版两套高可用架构。提供了全套的容灾切换、故障迁移、在线扩容、性能优化的数据库解决方案。欢迎各位购买使用:云数据库 Redis 版

日常环境申请：
https://www.atatech.org/articles/77542

内部用户接入KVStore，请先阅读接入文档 https://www.atatech.org/articles/75783

上一篇：LeetCode 551. 学生出勤记录 I


下一篇：我的阿里云ACE（认证架构师）通关秘技