JAVA开发过程中遇到几个数据处理问题

2021-09-14 09:29:37

1、获取用户的某种类型的行为数据列表（某一时间段内发生的），行为记录时间需要精确到分钟数取整，丢弃掉秒、毫秒的数据

实现方案①：

// 查询出这批数据集合，然后遍历数据集合，分别对每个日期进行精度处理
List<ActionData> actionList ---》 查询数据库获取
for(int i = 0; i < actionList.size(); i++) {
    actionList.get(i).setStartTime(this.process(actionList.get(i).getStartTime().getTime()));
}

    /**
     * 时间戳 移除秒、豪秒部分(忽略掉秒的精度，只保留整分钟数的精度)
     *
     * @param timestamp
     * @return
     */
    public Long process(long timestamp) {
        Calendar calendar = Calendar.getInstance();
        calendar.setTimeInMillis(timestamp);
        calendar.set(Calendar.SECOND, 0);
        long value = calendar.getTimeInMillis() / 1000;
        return value * 1000;
    }

实现方案②：

// 因为查询的数据是通过跨服务查询的，所以可以利用在跨服务传输时，返回结果的参数进行json序列化时，自动忽略掉，秒、毫秒部分的精度

​ 实现过程图解：

JAVA开发过程中遇到几个数据处理问题

public class QueryResponsePO {

    /**  主键id */
    @JsonSerialize(using = ToStringSerializer.class)
    @ApiModelProperty(value = "主键ID")
    private Long id;


    /**
     * 科室名称
     */
    @ApiModelProperty(value = "科室名称")
    private String deptName;

    /** 创建时间 */
    @ApiModelProperty(value = "创建时间")
    @JsonFormat(pattern = "yyyy-MM-dd HH:mm")
    private Date createTime;
    
    // 省略getter、setter
    }

 实现效果：最后Feign调用完成后，响应的List集合数据里面的每个PO的时间格式参数已经自动忽略了秒、豪秒的数据了

2、下载excel导出模板，打包之后上开发环境时报错，路径找到不文件

错误日志如下：

[2021-10-29 09:26:30,337] ERROR [com.sisensing.cgm.common.security.handler.GlobalExceptionHandler:68] class path resource [excel/DrugExcelTemplate.xlsx] cannot be resolved to absolute file path because it does not reside in the file system: jar:file:/opt/sisensing/lib/cgm-modules-system.jar!/BOOT-INF/classes!/excel/DrugExcelTemplate.xlsx
java.io.FileNotFoundException: class path resource [excel/DrugExcelTemplate.xlsx] cannot be resolved to absolute file path because it does not reside in the file system: jar:file:/opt/sisensing/lib/cgm-modules-system.jar!/BOOT-INF/classes!/excel/DrugExcelTemplate.xlsx
        at org.springframework.util.ResourceUtils.getFile(ResourceUtils.java:217)
        at org.springframework.util.ResourceUtils.getFile(ResourceUtils.java:180)
        at com.sisensing.cgm.system.service.impl.CgmDownloadServiceServiceImpl.downloadDrugExcelTemplate(CgmDownloadServiceServiceImpl.java:104)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController.downloadExcelTemplate$original$xdz09XE0(AdminDrugInfoController.java:61)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController.downloadExcelTemplate$original$xdz09XE0$accessor$qIAmhMs3(AdminDrugInfoController.java)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController$auxiliary$QmhiI1Nj.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController.downloadExcelTemplate(AdminDrugInfoController.java)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController$$FastClassBySpringCGLIB$$a910b7fc.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.sisensing.cgm.system.controller.admin.AdminDrugInfoController$$EnhancerBySpringCGLIB$$1c4e092e.downloadExcelTemplate(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.github.xiaoymin.swaggerbootstrapui.filter.SecurityBasicAuthFilter.doFilter(SecurityBasicAuthFilter.java:80)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.github.xiaoymin.swaggerbootstrapui.filter.ProductionSecurityFilter.doFilter(ProductionSecurityFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke$original$AhFsoL6d(StandardHostValve.java:143)
        at org.apache.catalina.core.StandardHostValve.invoke$original$AhFsoL6d$accessor$6TuURzMs(StandardHostValve.java)
        at org.apache.catalina.core.StandardHostValve$auxiliary$H6ZIjzm3.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

原来的代码：

String path = "classpath:template.xlsx";
File file = ResourceUtils.getFile(path);
InputStream inputStream = new FileInputStream(file);

......

解决方案：服务打成Jar包后，文件的实际路径是存在于服务器上的，还是通过文件路径的方式去访问文件，会报错，抛出FileNotFoundException的异常，应该按照Jar里面的文件读取方式，进行文件流的操作

String path = "template.xlsx";
InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream(path);

......

重新打包上开发环境后，经验证可以正常下载模板excel文件

3、正则表达式支持英文、数字和特殊字符（仅支持英文键盘下的特殊字符）

    @Pattern(regexp = "^([a-zA-Z0-9\\\\W_!@#$%^&*`~()-+=\\.\\,\\?\\-\\/\\:\\;\\{\\}\\(\\)\\|\\'\"\\[\\]\\>\\<]){8,20}$", message = "密码需为8至20位，包含数字、字母和特殊符号")

4、2个List集合数据，大的集合数据有3000条，小的集合数据大概有300条左右（几百条的范围，反正比大的数据集合少，正常使用场景下），需要遍历2个集合进行比较里面的数据，若相同，则将小集合的数据对应的值设置到大集合对应的值中，尽量使查询的时间复杂度降低

正常的比较方式，遍历2个集合，然后一个一个元素的去比较，时间复杂度为： O(N) * O(n) --> 3000 * 300

比较实体类：

package com.opencv.domain.po;

public class CompareMessagePO {

    private Long compareNum;

    private String message;

    public Long getCompareNum() {
        return compareNum;
    }

    public void setCompareNum(Long compareNum) {
        this.compareNum = compareNum;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

方式一的比较测试类设计：

package com.opencv.controller;

import com.opencv.domain.po.CompareMessagePO;

import java.util.ArrayList;
import java.util.List;


public class ListCompareTests {

    public static List<CompareMessagePO> COMPARE_HIGH_LIST = new ArrayList<>();

    public static List<CompareMessagePO> COMPARE_LOW_LIST = new ArrayList<>();

    static {
        for(int i = 0; i < 3000; i ++) {
            CompareMessagePO compareMessagePO = new CompareMessagePO();
            compareMessagePO.setCompareNum(Long.valueOf(i));
            COMPARE_HIGH_LIST.add(compareMessagePO);
        }

        for (int i = 100; i < 300; i ++) {
            CompareMessagePO compareMessagePO = new CompareMessagePO();
            compareMessagePO.setCompareNum(Long.valueOf(i));
            compareMessagePO.setMessage("test message : " + i);
            COMPARE_LOW_LIST.add(compareMessagePO);
        }
    }

    public static void main(String[] args) {
        long startTime = System.currentTimeMillis();
        for (int i = 0; i < COMPARE_LOW_LIST.size(); i ++) {
            for(int j = 0; j < COMPARE_HIGH_LIST.size(); j ++) {
                if(COMPARE_LOW_LIST.get(i).getCompareNum().longValue() == COMPARE_HIGH_LIST.get(j).getCompareNum().longValue()) {
                    COMPARE_HIGH_LIST.get(j).setMessage(COMPARE_LOW_LIST.get(i).getMessage());
                }
            }
        }
        long endTime = System.currentTimeMillis();
        long costTime = endTime - startTime;
        long secondTime = costTime / 1000;
        System.out.println("耗时：" + secondTime + "秒 , " + costTime + "毫秒");
        System.out.println("-----------------------");
        for (int i = 100; i < 300; i ++) {
            String message = COMPARE_HIGH_LIST.get(i).getMessage();
            System.out.println(message);
        }
        System.out.println("------------------------");
    }
}

方式二的比较测试类设计： 

package com.opencv.controller;

import com.opencv.domain.po.CompareMessagePO;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;


public class ListCompareNextTests {

    public static List<CompareMessagePO> COMPARE_HIGH_LIST = new ArrayList<>();

    public static Map<Long, String> COMPARE_LOW_MAP = new HashMap<>();

    static {
        for(int i = 0; i < 3000; i ++) {
            CompareMessagePO compareMessagePO = new CompareMessagePO();
            compareMessagePO.setCompareNum(Long.valueOf(i));
            COMPARE_HIGH_LIST.add(compareMessagePO);
        }

        for (int i = 100; i < 300; i ++) {
            COMPARE_LOW_MAP.put(Long.valueOf(i), "test message : " + i);
        }
    }

    public static void main(String[] args) {
        long startTime = System.currentTimeMillis();
        for(int j = 0; j < COMPARE_HIGH_LIST.size(); j ++) {
            if(COMPARE_LOW_MAP.containsKey(COMPARE_HIGH_LIST.get(j).getCompareNum())) {
                COMPARE_HIGH_LIST.get(j).setMessage(COMPARE_LOW_MAP.get(COMPARE_HIGH_LIST.get(j).getCompareNum()));
            }
        }
        long endTime = System.currentTimeMillis();
        long costTime = endTime - startTime;
        long secondTime = costTime / 1000;
        System.out.println("耗时：" + secondTime + "秒 , " + costTime + "毫秒");
        System.out.println("-----------------------");
        for (int i = 100; i < 300; i ++) {
            String message = COMPARE_HIGH_LIST.get(i).getMessage();
            System.out.println(message);
        }
        System.out.println("------------------------");
    }
}

测试结果

        方式①：

JAVA开发过程中遇到几个数据处理问题

         方式②：

JAVA开发过程中遇到几个数据处理问题

 

  2种方式的性能比较相差了18倍，若集合的数据量很大的情况下，估计性能相差更明显，倍数更大，方式②是以空间换时间，对空间的消耗上比方式①大

上一篇：Tomcat的优化（生产中的优化）


下一篇：苹果电脑重装macOS