Linux磁盘缓存的有趣实验

2021-08-04 14:19:26

Hopefully you are now convinced that Linux didn‘t just eat your ram. Here are some interesting things you can do to learn how the disk cache works.

希望您现在已经确信Linux没有吃掉内存。 您可以执行以下一些有趣的操作来了解磁盘缓存的工作方式。

Note: The examples shown are for and from 2009 era hardware without SSDs. You may have to multiply several of the example numbers here by 10 to see an appreciable effect.

注意：所示示例适用于2009年及之后的非SSD硬件。 您可能需要将此处的几个示例数字乘以10才能看到明显的效果。

Effects of disk cache on application memory allocation（磁盘缓存对应用程序内存分配的影响）

Since I‘ve already promised that disk cache doesn‘t prevent applications from getting the memory they want, let‘s start with that. Here is a C app (munch.c) that gobbles up as much memory as it can, or to a specified limit:

（由于我们知道磁盘缓存不会阻止应用程序获取所需的内存，因此让我们开始吧。 这是一个C应用程序（munch.c），它会吞噬尽可能多的内存，或达到指定的限制：）

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char** argv) {
    int max = -1;
    int mb = 0;
    char* buffer;

    if(argc > 1)
        max = atoi(argv[1]);

    while((buffer=malloc(1024*1024)) != NULL && mb != max) {
        memset(buffer, 0, 1024*1024);
        mb++;
        printf("Allocated %d MB\n", mb);
    }

    return 0;
}

Running out of memory isn‘t fun, but the OOM killer should end just this process and hopefully the rest will remain undisturbed. We‘ll definitely want to disable swap for this, or the app will gobble up that as well.

内存不足并不是一件很好的事，但是OOM杀手会结束该过程，并希望其余的进程不会受到干扰。 我们应该要禁用交换功能，否则应用程序也会吞噬掉它。

$ sudo swapoff -a

$ free -m

(note that your free output could be different, and have an ‘available‘ column instead of a ‘-/+‘ row)

请注意，因版本不同free命令输出结果可能会有所不同，并具有“available”列而没有“-/ + buffers/cache”行

             total       used       free     shared    buffers     cached
Mem:          1504       1490         14          0         24        809
-/+ buffers/cache:        656        848
Swap:            0          0          0

$ gcc munch.c -o munch

$ ./munch
Allocated 1 MB
Allocated 2 MB
(...)
Allocated 877 MB
Allocated 878 MB
Allocated 879 MB
Killed

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504        650        854          0          1         67
-/+ buffers/cache:        581        923
Swap:            0          0          0

Even though it said 14MB "free", that didn‘t stop the application from grabbing 879MB. Afterwards, the cache is pretty empty, but it will gradually fill up again as files are read and written. Give it a try.

显示14MB是“free”，但这并没有阻止应用程序获取879MB。 往后看，缓存非常空（见末尾第4点），但随着文件的读取和写入，它会逐渐逐渐填满。 我们试试看：

Effects of disk cache on swapping（磁盘缓存对swaping的影响）

I also said that disk cache won‘t cause applications to use swap. Let‘s try that as well, with the same munch app as in the last experiment. This time we‘ll run it with swap on, and limit it to a few hundred megabytes:

前面提到过，磁盘缓存不会导致应用程序使用swap。 让我们也尝试一下，使用与上一个实验相同的munch应用程序。 这次，我们将在swap打开的情况下运行它，并将其限制为几百兆字节：

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504       1490         14          0         10        874
-/+ buffers/cache:        605        899
Swap:         2047          6       2041

$ ./munch 400
Allocated 1 MB
Allocated 2 MB
(...)
Allocated 399 MB
Allocated 400 MB

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504       1090        414          0          5        485
-/+ buffers/cache:        598        906
Swap:         2047          6       2041

munch ate 400MB of ram, which was taken from the disk cache without resorting to swap. Likewise, we can fill the disk cache again and it will not start eating swap either. If you run watch free -m in one terminal, and find . -type f -exec cat {} + > /dev/null in another, you can see that "cached" will rise while "free" falls. After a while, it tapers off but swap is never touched.

消耗了400MB的内存，该内存是从磁盘缓存中取出的，而无需进行swap。 同样，我们可以再次填充磁盘缓存，它也不会开始占用swap空间。 如果在一个终端上运行watch free -m并find . -type f -exec cat {} +> / dev / null，您可以看到“ cached”将增加，而“ free”将减少。 一段时间后，它逐渐变小，但是swap从未被触及（见末尾第2点）。

Clearing the disk cache（清除磁盘缓存）

For experimentation, it‘s very convenient to be able to drop the disk cache. For this, we can use the special file /proc/sys/vm/drop_caches. By writing 3 to it, we can clear most of the disk cache:

为了进行实验，能够方便删除磁盘缓存。 为此，我们可以使用特殊文件/proc/sys/vm/drop_caches。 通过向其写入3，我们可以清除大部分磁盘缓存：

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504       1471         33          0         36        801
-/+ buffers/cache:        633        871
Swap:         2047          6       2041

$ echo 3 | sudo tee /proc/sys/vm/drop_caches 
3

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504        763        741          0          0        134
-/+ buffers/cache:        629        875
Swap:         2047          6       2041

Notice how "buffers" and "cached" went down, free mem went up, and free+buffers/cache stayed the same.

注意在Mem这一行“ buffers”和“ cached”下降，free上升，而在 + buffers / cache行的free列的值是保持不变。

Effects of disk cache on load times（磁盘缓存对加载时间的影响）

Let‘s make two test programs, one in Python and one in Java. Python and Java both come with pretty big runtimes, which have to be loaded in order to run the application. This is a perfect scenario for disk cache to work its magic.

让我们制作两个测试程序，一个用Python，另一个用Java。 Python和Java都具有相当大的运行时，必须加载这些运行时才能运行该应用程序。 这是磁盘缓存发挥其性能的完美方案。

$ cat hello.py
print "Hello World! Love, Python"

$ cat Hello.java
class Hello {
    public static void main(String[] args) throws Exception {
        System.out.println("Hello World! Regards, Java");
    }
}

$ javac Hello.java

$ python hello.py
Hello World! Love, Python

$ java Hello
Hello World! Regards, Java

Our hello world apps work. Now let‘s drop the disk cache, and see how long it takes to run them.

我们小应用程序可以正常工作。 现在，让我们删除磁盘缓存，看看运行它们需要多长时间。

$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3

$ time python hello.py
Hello World! Love, Python

real    0m1.026s
user    0m0.020s
sys     0m0.020s

$ time java Hello
Hello World! Regards, Java

real    0m2.174s
user    0m0.100s
sys     0m0.056s

$

Wow. 1 second for Python, and 2 seconds for Java? That‘s a lot just to say hello. However, now all the file required to run them will be in the disk cache so they can be fetched straight from memory. Let‘s try again:

哇。 对于Python是1秒，对于Java是2秒？ 这只是个开始，现在运行它们所需的所有文件都将位于磁盘缓存中，因此可以直接从内存中获取它们。 让我们再试一次：

$ time python hello.py
Hello World! Love, Python

real    0m0.022s
user    0m0.016s
sys     0m0.008s

$ time java Hello
Hello World! Regards, Java

real    0m0.139s
user    0m0.060s
sys     0m0.028s

$

Yay! Python now runs in just 22 milliseconds, while java uses 139ms. That‘s 45 and 15 times faster! All your apps get this boost automatically!

好极了！ Python现在仅需22毫秒即可运行，而Java使用139ms。 分别快45和15倍！ 您所有的应用程序都会自动获得提升！

Effects of disk cache on file reading（磁盘缓存对文件读取的影响）

Let‘s make a big file and see how disk cache affects how fast we can read it. I‘m making a 200MB file, but if you have less free ram, you can adjust it.

让我们做一个大文件，看看磁盘缓存如何影响我们读取它的速度。 我正在制作200MB的文件，但是如果您的可用内存较少，则可以对其进行调整。

$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504        546        958          0          0         85
-/+ buffers/cache:        461       1043
Swap:         2047          6       2041

$ dd if=/dev/zero of=bigfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 6.66191 s, 31.5 MB/s

$ ls -lh bigfile
-rw-r--r-- 1 vidar vidar 200M 2009-04-25 12:30 bigfile

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1504        753        750          0          0        285
-/+ buffers/cache:        468       1036
Swap:         2047          6       2041

Since the file was just written, it will go in the disk cache. The 200MB file caused a 200MB bump in "cached". Let‘s read it, clear the cache, and read it again to see how fast it is:

由于文件刚刚被写入，因此它将进入磁盘缓存。 200MB的文件在“缓存”中引起了200MB的增加。 让我们阅读它，清除缓存，然后再次阅读以查看它有多快：

$ time cat bigfile > /dev/null

real    0m0.139s
user    0m0.008s
sys     0m0.128s

$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3

$ time cat bigfile > /dev/null

real    0m8.688s
user    0m0.020s
sys     0m0.336s

$

That‘s more than fifty times faster!

速度快了五十倍！

Conclusions（结论）

The Linux disk cache is very unobtrusive. It uses spare memory to greatly increase disk access speeds, and without taking any memory away from applications. A fully used store of ram on Linux is efficient hardware use, not a warning sign.

Linux磁盘缓存非常简单。 它使用备用内存来大大提高磁盘访问速度，而不会占用应用程序任何内存。 在Linux上尽可能的多使用ram存储是有效的硬件使用，而不是警告信号。

原文链接：https://www.linuxatemyram.com/play.html

These pages do simplify a little:（以上内容扩展知识点）

  1. While newly allocated memory will always (though see point #2) be taken from the disk cache instead of swap, Linux can be configured to preemptively swap out other unused applications in the background to free up memory for cache. The is tunable through the swappiness setting, accessible through /proc/sys/vm/swappiness.

  2. 虽然新分配的内存总是从磁盘缓存中获取而不是使用swap分区，但是Linux可以配置那些未经常使用的应用程序先使用swap分区供，以释放内存以供缓存使用。 可以通过swappiness设置进行调整，可通过/proc/sys/vm/swappiness访问。

  3. Some parts of the cache can‘t be dropped, not even to accomodate new applications. This includes mmap‘d pages that have been mlocked by some application, dirty pages that have not yet been written to storage, and data stored in tmpfs (including /dev/shm, used for shared memory). The mmap‘d, mlocked pages are stuck in the page cache. Dirty pages will for the most part swiftly be written out. Data in tmpfs will be swapped out if possible.

  4. 缓存的某些部分无法删除，甚至不能容纳新的应用程序。 这包括已被某些应用程序锁定的mmap页，尚未写入存储的脏页以及tmpfs中存储的数据（包括用于共享内存的/dev/shm）。mmap锁定的页面被卡在页面缓存中。 在大多数情况下，脏页很快就会被写出。 如果可能，tmpfs中的数据将被换出。

Linux磁盘缓存的有趣实验

上一篇：很详细的eclipse的设置


下一篇：Eclipse快捷键