[SpringBoot/SpringMVC]从Webapp下载一个大文件出现java.lang.OutOfMemoryError: GC overhead limit exceeded怎么办？

2022-10-12 11:43:01

本文示例工程下载：https://files.cnblogs.com/files/xiandedanteng/WebFileDownload20191026.rar

制作一个Webapp，让其中一个网页提供下载链接，以使用户能下载本地文件或是临时生成的文件，这些都不是难事，网上也有很多既存的解决方案。

但是，这个问题难点在，但生成文件过大时，产生java.lang.OutOfMemoryError异常怎么办？有人提出修改JVM内存参数，如-Xms<min>m -Xmx<max>m方式，但终究是治标不治本的方法，如果下载数据量又超过设定的上限呢？

其实问题的本质是：在为大数据生成准备过程中，大量对象产生了来不及释放，因为还需要在接下来的步骤中使用，所以驻留在内存中，内存越积越多，终究导致java.lang.OutOfMemoryError异常。举个例子来说，有个emp表，存储员工的id姓名年龄等，当只有千百条时，取出结果集转成链表再写入csv文件自然没什么问题，但如果数据越来越多，还是结果集放链表里又来不及释放，总有内存不够的时候。

2019-10-25 14:43:05.518 ERROR 40016 --- [nio-8080-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded] with root cause

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.Integer.toString(Unknown Source) ~[na:1.8.0_201]
    at java.lang.String.valueOf(Unknown Source) ~[na:1.8.0_201]
    at com.example.demo.CsvUtil.exportBigCsv(CsvUtil.java:78) ~[classes/:na]
    at com.example.demo.SpringBootWeb1Application.downloadBigCsvFile(SpringBootWeb1Application.java:141) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_201]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

要解决这个问题，就得耐下心来，将整体读整体写改成分批读分批写（分批方案和分页方案类似，诸位可自行寻找合适自己DB的DB dialect），最后再下载。

示例页面：

<!DOCTYPE html>
<html lang="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<head>
     <title>happy go lucky</title>
    </head>

     <body>
            <h2>文件下载测试</h2>
            <hr/>
            <ol>
                <li><a href="/dldLocalFile">下载本地文件dld.rar</a></li>
                <li><a href="/dldGeneratedFile/100">下载生成百行CSV文件</a></li>
                <li><a href="/dldGeneratedFile/100000000">下载生成亿行CSV文件</a></li>
                <li><a href="/dldGeneratedFile2/10000000">下载生成千万行CSV文件 可能 出现java.lang.OutOfMemoryError: GC overhead limit exceeded</a></li>
                <li><a href="/dldGeneratedFile3/10000000">下载生成千万行CSV文件 不会可能 出现java.lang.OutOfMemoryError: GC overhead limit exceeded</a></li>
            </ol>
     </body>
</html>
<script type="text/javascript">
<!--
    // 脚本
//-->
</script>

控制类：

package com.example.demo;

import java.io.File;
import java.io.FileInputStream;
import java.net.URLDecoder;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.log4j.Logger;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
@SpringBootApplication
public class WebFileDownloadApplication {
    private static Logger logger = Logger.getLogger(WebFileDownloadApplication.class);

    public static void main(String[] args) {
        SpringApplication.run(WebFileDownloadApplication.class, args);
    }

    @RequestMapping("/")
    public String index() {
        logger.info("进入index页");
        return "index.html";
    }

    @RequestMapping("/dldLocalFile")
    public void downloadLocalFile(HttpServletResponse res, HttpServletRequest req) throws Exception {
        logger.info("下载本地文件");

        String localFilename = "dld.rar";
        String localFilepath = getClass().getResource("/static/" + localFilename).getPath();

        res.setContentType("multipart/form-data");
        res.setCharacterEncoding("UTF-8");
        res.setContentType("text/html");

        String userAgent = req.getHeader("User-Agent");
        if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
            // IE Core
            localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
        } else {
            // Non-IE Core
            localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
        }
        res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);

        localFilepath = URLDecoder.decode(localFilepath, "UTF-8");
        FileInputStream instream = new FileInputStream(localFilepath);
        ServletOutputStream outstream = res.getOutputStream();
        int b = 0;
        byte[] buffer = new byte[1024];
        while ((b = instream.read(buffer)) != -1) {
            outstream.write(buffer, 0, b);
        }
        instream.close();

        if (outstream != null) {
            outstream.flush();
            outstream.close();
        }
    }

    @RequestMapping("/dldGeneratedFile/{count}")
    public void downloadGeneratedCsvFile(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
        logger.info("Start downloadGeneratedCsvFile");

        SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
        Date time = new Date();
        String tStamp = dfs.format(time);

        String localFilename = tStamp+".csv";
        String path=req.getSession().getServletContext().getRealPath("/");
        String localFilepath = path+localFilename;
        logger.info("准备生成的本地路径文件名="+localFilepath);

        res.setContentType("multipart/form-data");
        res.setCharacterEncoding("UTF-8");
        res.setContentType("text/html");

        String userAgent = req.getHeader("User-Agent");
        if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
            // IE Core
            localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
        } else {
            // Non-IE Core
            localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
        }
        res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);

        localFilepath = URLDecoder.decode(localFilepath, "UTF-8");

        List<String> dataList=new ArrayList<String>();
        for(int i=0;i<100;i++) {
            dataList.add(String.valueOf(i));
        }

        File file=new File(localFilepath);
        CsvUtil.generateCsv(file,Integer.parseInt(count));
        logger.info("已经生成文件:"+localFilepath);

        FileInputStream instream = new FileInputStream(localFilepath);
        ServletOutputStream outstream = res.getOutputStream();
        int b = 0;
        byte[] buffer = new byte[1024];
        while ((b = instream.read(buffer)) != -1) {
            outstream.write(buffer, 0, b);
        }
        instream.close();

        if (outstream != null) {
            outstream.flush();
            outstream.close();
            boolean isDeleted=file.delete();
            if(isDeleted) {
                logger.info("已经删除文件："+localFilepath);
            }
        }
    }

    @RequestMapping("/dldGeneratedFile2/{count}")
    public void downloadGeneratedCsvFile2(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
        logger.info("Start downloadGeneratedCsvFile2");

        SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
        Date time = new Date();
        String tStamp = dfs.format(time);

        String localFilename = tStamp+".csv";
        String path=req.getSession().getServletContext().getRealPath("/");
        String localFilepath = path+localFilename;
        logger.info("准备生成的本地路径文件名="+localFilepath);

        res.setContentType("multipart/form-data");
        res.setCharacterEncoding("UTF-8");
        res.setContentType("text/html");

        String userAgent = req.getHeader("User-Agent");
        if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
            // IE Core
            localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
        } else {
            // Non-IE Core
            localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
        }
        res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);

        localFilepath = URLDecoder.decode(localFilepath, "UTF-8");

        List<String> dataList=new ArrayList<String>();
        for(int i=0;i<100;i++) {
            dataList.add(String.valueOf(i));
        }

        File file=new File(localFilepath);
        CsvUtil.generateCsv2(file,Integer.parseInt(count));
        logger.info("已经生成文件:"+localFilepath);

        FileInputStream instream = new FileInputStream(localFilepath);
        ServletOutputStream outstream = res.getOutputStream();
        int b = 0;
        byte[] buffer = new byte[1024];
        while ((b = instream.read(buffer)) != -1) {
            outstream.write(buffer, 0, b);
        }
        instream.close();

        if (outstream != null) {
            outstream.flush();
            outstream.close();
            boolean isDeleted=file.delete();
            if(isDeleted) {
                logger.info("已经删除文件："+localFilepath);
            }
        }
    }

    @RequestMapping("/dldGeneratedFile3/{count}")
    public void downloadGeneratedCsvFile3(HttpServletResponse res, HttpServletRequest req,@PathVariable String count) throws Exception {
        logger.info("Start downloadGeneratedCsvFile3");

        SimpleDateFormat dfs = new SimpleDateFormat("yyyyMMddHHmmss");
        Date time = new Date();
        String tStamp = dfs.format(time);

        String localFilename = tStamp+".csv";
        String path=req.getSession().getServletContext().getRealPath("/");
        String localFilepath = path+localFilename;
        logger.info("准备生成的本地路径文件名="+localFilepath);

        res.setContentType("multipart/form-data");
        res.setCharacterEncoding("UTF-8");
        res.setContentType("text/html");

        String userAgent = req.getHeader("User-Agent");
        if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
            // IE Core
            localFilename = java.net.URLEncoder.encode(localFilename, "UTF-8");
        } else {
            // Non-IE Core
            localFilename = new String((localFilename).getBytes("UTF-8"), "ISO-8859-1");
        }
        res.setHeader("Content-Disposition", "attachment;fileName=" + localFilename);

        localFilepath = URLDecoder.decode(localFilepath, "UTF-8");

        List<String> dataList=new ArrayList<String>();
        for(int i=0;i<100;i++) {
            dataList.add(String.valueOf(i));
        }

        File file=new File(localFilepath);
        CsvUtil.generateCsv3(file,Integer.parseInt(count));
        logger.info("已经生成文件:"+localFilepath);

        FileInputStream instream = new FileInputStream(localFilepath);
        ServletOutputStream outstream = res.getOutputStream();
        int b = 0;
        byte[] buffer = new byte[1024];
        while ((b = instream.read(buffer)) != -1) {
            outstream.write(buffer, 0, b);
        }
        instream.close();

        if (outstream != null) {
            outstream.flush();
            outstream.close();
            boolean isDeleted=file.delete();
            if(isDeleted) {
                logger.info("已经删除文件："+localFilepath);
            }
        }
    }
}

数据生成类：

package com.example.demo;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.log4j.Logger;

public class CsvUtil {
    private static Logger logger = Logger.getLogger(CsvUtil.class);

    // 这种方式不会内存溢出，因为创建的数组对象用于写完文件后在一个循环结束就释放了
    public static boolean generateCsv(File file,int rowCount) {
        try {
            FileWriter fileWriter =new FileWriter(file, true);

            for(int j=0;j<rowCount;j++) {
                int lineNo=j;

                String[] arr=new String[20];

                for(int k=0;k<arr.length;k++) {
                    arr[k]=String.valueOf(k);
                }

                String info=String.valueOf(lineNo)+","+String.join(",", arr)+System.getProperty("line.separator");
                fileWriter.write(info);
            }

            fileWriter.flush();
            fileWriter.close();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return true;
    }

    // 这种方式因为生成一个大list，容易造成内存驻留过多而溢出
    public static boolean generateCsv2(File file,int rowCount) {
        try {
            // 先模拟访问数据库创建集合，当数据量大时会出现outofmemory异常
            List<String[]> list=new ArrayList<String[]>();
            for(int i=0;i<rowCount;i++) {
                String[] arr=new String[20];
                for(int k=0;k<arr.length;k++) {
                    arr[k]=String.valueOf(k);
                }

                list.add(arr);
            }

            // 再将集合写文件
            FileWriter fileWriter =new FileWriter(file, true);

            int index=0;
            for(String[] arr:list) {
                index++;

                String info=String.valueOf(index)+","+String.join(",", arr)+System.getProperty("line.separator");
                fileWriter.write(info);
            }

            fileWriter.flush();
            fileWriter.close();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return true;
    }

    // 这种方式改成分批写入文件方式，以规避内存溢出
    public static boolean generateCsv3(File file,int rowCount) {
        logger.info("generateCsv3");

        try {
            final int LinesOnceTime=10000;
            int count=rowCount/LinesOnceTime;

            for(int i=0;i<count;i++) {
                // 再将集合写文件
                FileWriter fileWriter =new FileWriter(file, true);

                List<String[]> list=new ArrayList<String[]>();

                // 这回list被限制在了LinesOnceTime件,这一步代表着从数据库分批取，oracle有townum支持，mysql有limit支持
                for(int j=0;j<LinesOnceTime;j++) {
                    String[] arr=new String[20];

                    for(int k=0;k<arr.length;k++) {
                        arr[k]=String.valueOf(k);
                    }

                    list.add(arr);
                }

                int index=0;
                for(String[] arr:list) {
                    index++;

                    int lineNo=i*LinesOnceTime+index;
                    String info=String.valueOf(lineNo)+","+String.join(",", arr)+System.getProperty("line.separator");
                    fileWriter.write(info);
                }

                fileWriter.flush();
                fileWriter.close();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

        return true;
    }
}

续篇请看：https://www.cnblogs.com/xiandedanteng/p/11747855.html

--END-- 2019年10月26日14:49:02

码农公寓

相关文章