redis.clients.jedis.exceptions.JedisConnectionException之异常解决

2023-01-05 14:10:54

报错描述

     项目在本地运行时，没发现异常，待部署服务器之后，检查日志文件有多个redis超时报错如下：
redis.clients.jedis.exceptions.JedisConnectionException之异常解决
     顺藤摸瓜可见，在调用redis工具类的scanRedis方法后，间接调用了hscan方法，出现socket读超时，先来看一下redis工具类的代码：

public class RedisUtils {
    private JedisPool pool;
    private String spiderUUID;
    private String coordinatesHashKey;
    private ScanParams scanParams = new ScanParams().match("*").count(100);
    private static Logger logger = Logger.getLogger(RedisUtils.class);

    public RedisUtils(String host, int port, String password, String spiderUUID) {
        this(new JedisPool(new JedisPoolConfig(), host, port, 2000, password));
        this.spiderUUID = spiderUUID;
        coordinatesHashKey = this.spiderUUID + ":coordinates";
    }

    public RedisUtils(JedisPool pool) {
        this.pool = pool;
    }
	
	public void resetDownloadSuccessFlag(){
        Jedis jedis = null;
        try {
            jedis = this.pool.getResource();
            Map<String, String> result = jedis.hgetAll(coordinatesHashKey);
            Set<String> keySet = result.keySet();
            Iterator iterator = keySet.iterator();
            while (iterator.hasNext()){
                String key = (String) iterator.next();
                String value = jedis.get(this.spiderUUID + ":" + key);
                if (!"1".equals(value)){
                    jedis.set(this.spiderUUID + ":" + key, String.valueOf(1));
                }
            }
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }


    public Boolean isPanoIDExists(String panoId) {
        Jedis jedis = null;
        Boolean exists;
        try {
            jedis = this.pool.getResource();
            exists = jedis.exists(this.spiderUUID + ":" + panoId);
            return exists;
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

    public boolean removeKeys(){
        Jedis jedis = this.pool.getResource();
        try {
            Set<String> keys = jedis.keys(this.spiderUUID + ":*" );
            if(keys != null && !keys.isEmpty()) {
                logger.info("redis has stored " + keys.size() + " keys, now ready to remove them all!");
                String[] array = new String[keys.size()];
                jedis.del(keys.toArray(array));
            }
            return true;
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
        return true;
    }

    public boolean panoIdPush(String panoId) {
        Jedis jedis = this.pool.getResource();
        try {
            long num = jedis.setnx(this.spiderUUID + ":" + panoId, String.valueOf(1));
            return num==1;
        } finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

	public boolean coordinatesPush(Coordinates coordinates) {
        Jedis jedis = this.pool.getResource();
        String panoId = coordinates.getPanoId();
        try {
            jedis.setnx(this.spiderUUID + ":" + panoId, String.valueOf(1));
            return 1==jedis.hsetnx(coordinatesHashKey, panoId, JSON.toJSONString(coordinates));
        } finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

    public boolean particularCoordinatesPush(ParticularCoordinates coordinates) {
        Jedis jedis = this.pool.getResource();
        String panoId = coordinates.getPanoId();
        try {
            jedis.setnx(this.spiderUUID + ":" + panoId, String.valueOf(1));
            long num = jedis.hsetnx(coordinatesHashKey, panoId, JSON.toJSONString(coordinates));
            return num==1;
        } finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }
	
	public ScanResult<Map.Entry<String, String>> scanRedis(int cursor){
        ScanResult<Map.Entry<String, String>> scanResults = null;
        Jedis jedis = this.pool.getResource();
        long timeBefore = System.currentTimeMillis();
        try{
            scanResults = jedis.hscan(coordinatesHashKey, String.valueOf(cursor), scanParams);
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            if (jedis != null) {
                jedis.close();
            }
        }
        return scanResults;
    }
	
	public boolean downloadSucceed(String panoId) {
        Jedis jedis = this.pool.getResource();
        long timeBefore = System.currentTimeMillis();
        int num=0;
        try {
            jedis.incr(this.spiderUUID + ":" + panoId);
            num = Integer.parseInt(jedis.get(this.spiderUUID + ":" + panoId));
        }
        catch (JedisConnectionException e1){
            e1.printStackTrace();
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
        return num == 2;
    }

}

分析

     首先回顾redisPool生成客户端的流程逻辑图
redis.clients.jedis.exceptions.JedisConnectionException之异常解决

     上述代码构造方法中将jedisPool的超时时间设置了2秒，检查源码发现默认就是2秒。一番百度谷歌之后发现，问题原因可能有如下几种：(1) 读写超时设置的过短。(2) 有慢查询或者Redis发生阻塞。(3) 网络不稳定。而网友的解决之道无非是：1、提高超时时间（将jedispool超时时间设为20秒、甚至100秒）；2、分析redis执行命令，避免执行过于耗时的运算；3、使用jedis新版本，最好2.9.0版本以上，并在使用完毕后采用jedis.close()方法解决归还已破坏的连接问题。

     但本项目中jedis正好是2.9.0版本，笔者也将hcsan遍历数量100条降到了每次返回50条，在测试过程中还简化了业务流程，一个线程只进行一次Redis操作（scan命令），且操作过程不存在锁竞争，同时也把超时时间设大，一度设置到100秒仍然出现上述异常。
     redis基于内存操作，同时又是单线程运算，一般的命令执行都是毫秒、微妙级，通过查看运维工具Grafana显示hscan的平均执行时间也是47us左右，且服务器端没有拒绝连接的情况，redis运维监测图分别如下所示：
redis.clients.jedis.exceptions.JedisConnectionException之异常解决

redis.clients.jedis.exceptions.JedisConnectionException之异常解决
     因此，设置超时等待100秒之后还是出现以上问题不免大大的疑惑，心想公司网络不可能断网超过100秒吧。

     继续Google发现，这个异常是客户端缓冲区异常，产生这个问题可能有以下三个原因：

     (1) 常见原因：多个线程使用一个Jedis连接。正常的情况是一个线程使用一个Jedis连接，可以使用JedisPool管理Jedis连接，实现线程安全，避免出现这种情况。例如下面代码就是两个线程共用了一个Jedis连接：

new Thread(new Runnable() {
    public void run() {
        for (int i = 0; i < 100; i++) {
            jedis.get("hello");
        }
    }
}).start();
new Thread(new Runnable() {
    public void run() {
        for (int i = 0; i < 100; i++) {
            jedis.hget("haskey", "f");
        }
    }
}).start();

     (2) 客户端缓冲区满了

     Redis有三种客户端缓冲区：
     1. 普通客户端缓冲区(normal)：用于接受普通的命令，例如get、set、mset、hgetall、zrange等。
     2. slave客户端缓冲区(slave)：用于同步master节点的写命令，完成复制。
     3. 发布订阅缓冲区(pubsub)：pubsub不是普通的命令，因此有单独的缓冲区。
redis.clients.jedis.exceptions.JedisConnectionException之异常解决
     (3) 长时间闲置连接会被服务端主动断开，可以查询timeout配置的设置以及自身连接池配置确定是否需要做空闲检测。

解决

     此时看到这里依然感觉是云里雾里，但又看到了 这篇文章 和自己报错一样，文中指出底层的socket超时是由网络异常造成，解决方法是：重置连接状态，重置计数器。让我想到了以前netty框架中有个ByteBuf也是解决了buffer游标重置的问题。
文中介绍了以下两个方法：

jedis.disconnect();                  //直接断开当前连接，将连接从连接池中踢出去。
jedis.resetPipelinedCount();         // 直接重置计数器

     好了，在工具类方法中客户端归还时加上了jedis.disconnect()方法后，部署运行发现，redis.clients.util.RedisInputStream.ensureFill异常没有了，但随之而来的又是一个新异常：redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
redis.clients.jedis.exceptions.JedisConnectionException之异常解决
     异常出现在JedisPool中，该异常明眼一看就是连接数量不够，等待超时后报错产生。那就先看看连接池JedisPool源码，可以看到JedisPool的构造方法中有个JedisPoolConfig配置器，该配置器继承了apache.commons.pool包中的GenericObjectPoolConfig对象：

public static final int DEFAULT_MAX_TOTAL = 8;
public static final int DEFAULT_MAX_IDLE = 8;
public static final int DEFAULT_MIN_IDLE = 0;
private int maxTotal = 8;
private int maxIdle = 8;
private int minIdle = 0;

     源码一看便知，最大连接数量也不过是8，而本项目核心线程数量设置为10，肯定不够。故扩大连接数量和空闲数量即可。

     再观察一下GenericObjectPoolConfig类，该类继承自抽象类BaseObjectPoolConfig，继续点进BaseObjectPoolConfig去可以看到有如下源码：

//显示部分源码
private boolean testOnCreate = false;
private boolean testOnBorrow = false;
private boolean testOnReturn = false;
private boolean testWhileIdle = false;

public boolean getTestOnCreate() {
    return this.testOnCreate;
}

public void setTestOnCreate(boolean testOnCreate) {
    this.testOnCreate = testOnCreate;
}

public boolean getTestOnBorrow() {
    return this.testOnBorrow;
}

public void setTestOnBorrow(boolean testOnBorrow) {
    this.testOnBorrow = testOnBorrow;
}

public boolean getTestOnReturn() {
    return this.testOnReturn;
}

public void setTestOnReturn(boolean testOnReturn) {
    this.testOnReturn = testOnReturn;
}

public boolean getTestWhileIdle() {
    return this.testWhileIdle;
}

public void setTestWhileIdle(boolean testWhileIdle) {
    this.testWhileIdle = testWhileIdle;
}

     百度了一下，这正是检测网络是否异常的方法，且构造连接池时默认为空，若设为true，则每次向jedisPool连接池中借还客户端之前会对网络进行ping检测。该方法与 jedis.disconnect()具有相似作用，但执行更优雅。因此，尝试在redis工具类中加以设置看看能否保证连接可靠。
     改进后的代码：

public class RedisUtils {
    private JedisPool pool;
    private String spiderUUID;
    private String coordinatesHashKey;
    private static JedisPoolConfig jedisPoolConfig;
    private static final int POOL_MAX_IDLE = 16;
    private static final int POOL_MAX_TOTAL = 36;
    private static final long POOL_MAX_WAIT = 50000;
    private ScanParams scanParams = new ScanParams().match("*").count(50);
    private static Logger logger = Logger.getLogger(RedisUtils.class);

    static {
        jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setTestOnBorrow(true);
        jedisPoolConfig.setTestOnReturn(true);
        jedisPoolConfig.setMaxIdle(POOL_MAX_IDLE);
        jedisPoolConfig.setMaxTotal(POOL_MAX_TOTAL);
        jedisPoolConfig.setMaxWaitMillis(POOL_MAX_WAIT);
    }

    public RedisUtils(String host, int port, String password, String spiderUUID) {
        this(new JedisPool(jedisPoolConfig, host, port, 30000, password));
        this.spiderUUID = spiderUUID;
        coordinatesHashKey = this.spiderUUID + ":coordinates";
    }

    public RedisUtils(JedisPool pool) {
        this.pool = pool;
    }

    public void resetDownloadSuccessFlag(){
        Jedis jedis = null;
        try {
            jedis = this.pool.getResource();
            Map<String, String> result = jedis.hgetAll(coordinatesHashKey);
            Set<String> keySet = result.keySet();
            Iterator iterator = keySet.iterator();
            while (iterator.hasNext()){
                String key = (String) iterator.next();
                String value = jedis.get(this.spiderUUID + ":" + key);
                if (!"1".equals(value)){
                    jedis.set(this.spiderUUID + ":" + key, String.valueOf(1));
                }
            }
        }finally {
            if (jedis!=null){
                jedis.getClient().resetPipelinedCount();
                jedis.close();
            }
        }
    }

    public Boolean isPanoIDExists(String panoId) {
        Jedis jedis = null;
        Boolean exists;
        try {
            jedis = this.pool.getResource();
            exists = jedis.exists(this.spiderUUID + ":" + panoId);
            return exists;
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

    public boolean removeKeys(){
        Jedis jedis = this.pool.getResource();
        try {
            Set<String> keys = jedis.keys(this.spiderUUID + ":*" );
            if(keys != null && !keys.isEmpty()) {
                logger.info("redis has stored " + keys.size() + " keys, now ready to remove them all!");
                String[] array = new String[keys.size()];
                jedis.del(keys.toArray(array));
            }
            return true;
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
        return true;
    }

    public boolean coordinatesPush(Coordinates coordinates) {
        Jedis jedis = this.pool.getResource();
        String panoId = coordinates.getPanoId();
        try {
            jedis.setnx(this.spiderUUID + ":" + panoId, String.valueOf(1));
            return 1==jedis.hsetnx(coordinatesHashKey, panoId, JSON.toJSONString(coordinates));
        } finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

    public boolean particularCoordinatesPush(ParticularCoordinates coordinates) {
        Jedis jedis = this.pool.getResource();
        String panoId = coordinates.getPanoId();
        try {
            jedis.setnx(this.spiderUUID + ":" + panoId, String.valueOf(1));
            long num = jedis.hsetnx(coordinatesHashKey, panoId, JSON.toJSONString(coordinates));
            return num==1;
        } finally {
            if (jedis!=null){
                jedis.close();
            }
        }
    }

    public ScanResult<Map.Entry<String, String>> scanRedis(int cursor){
        ScanResult<Map.Entry<String, String>> scanResults = null;
        Jedis jedis = this.pool.getResource();
        long timeBefore = System.currentTimeMillis();
        try{
            scanResults = jedis.hscan(coordinatesHashKey, String.valueOf(cursor), scanParams);
        }catch (Exception e){
            long timeAfter = System.currentTimeMillis();
            long executeTime = timeAfter-timeBefore;
            e.printStackTrace();
            logger.info("*****************************************************************");
            logger.info(Thread.currentThread().getName()+" execute hscan operation time cost : "+executeTime+"ms and current cursor is " + cursor);
            logger.info("*****************************************************************");
        }finally {
            if (jedis != null) {
                jedis.close();
            }
        }
        return scanResults;
    }
    
    public boolean downloadSucceed(String panoId) {
        Jedis jedis = this.pool.getResource();
        long timeBefore = System.currentTimeMillis();
        int num=0;
        try {
            jedis.incr(this.spiderUUID + ":" + panoId);
            num = Integer.parseInt(jedis.get(this.spiderUUID + ":" + panoId));
        }
        catch (JedisConnectionException e1){
            long timeAfter = System.currentTimeMillis();
            long executeTime = timeAfter-timeBefore;
            e1.printStackTrace();
            logger.info("********************************************************************");
            logger.info(Thread.currentThread().getName()+" execute incr and get operation time cost : "+executeTime+"ms!");
            logger.info("********************************************************************");
        }finally {
            if (jedis!=null){
                jedis.close();
            }
        }
        return num == 2;
    }

}

     再次部署运行，无异常抛出，问题得到解决！

参考

https://tech.antfin.com/docs/2/71967
https://www.jishuwen.com/d/2VHU/zh-tw#tuit
https://blog.zongwei.li/2016/10/28/resolve-jedis-read-time-exception/

上一篇：Jedis连接 HelloWorld实现


下一篇：kafka-clients源码分析三：producer发送消息详解