JMH初体验

2023-10-03 23:20:52

Java 8

JMH 1.19

Eclipse Version: 2021-03 (4.19.0)

---

JMH仓库

https://github.com/openjdk/jmh

https://mvnrepository.com/search?q=jmh

最新版本：1.33 但没用起来，出现了OOM错误（后文解决了此问题）

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM.

JMH 是 OpenJDK 团队开发的一款基准测试工具，一般用于代码的性能调优，精度甚至可以达到纳秒级别，适用于 java 以及其他基于 JVM 的语言。

JMH是Java性能测试工具，主要是对工程中一些方法进行一些基准测试，支持的时间单位为：nano / micro / milli / macro

根据参考文档1，添加了下面的测试程序：

package jmh;

import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

/**
 * 使用JMH测试ArrayList、LinkedList的性能——add
 * @author ben
 * @date 2021-09-23 20:31:23 CST
 */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Thread)
public class Jmh1 {

	// java.lang.OutOfMemoryError: Java heap space
	// at java.util.Arrays.copyOf(Arrays.java:3181)
	public final static String DATA = "DUMMY DATA";
//	public final static String DATA = "D";
//	public final static String DATA = "";
	
	private List<String> arrayList;
	private List<String> linkedList;
	
	public static int cnt = 0;
	
	@Setup(Level.Iteration)
	public void setUp() {
		arrayList = new ArrayList<String>();
		linkedList = new LinkedList<String>();
	}
	
	@Benchmark
	public List<String> arrayListAdd() {
		cnt++;
		try {
			this.arrayList.add(DATA);
		} catch (OutOfMemoryError e) {
			// cnt=157704908, size=157704907
			System.out.println("cnt=" + cnt + ", size=" + this.arrayList.size());
			throw e;
		}
		return this.arrayList;
	}
	
	@Benchmark
	public List<String> linkedListAdd() {
		this.linkedList.add(DATA);
		return linkedList;
	}
	
	public static void main(String[] args) throws RunnerException {
		final Options opts = new OptionsBuilder()
				.include(Jmh1.class.getSimpleName())
				.forks(1)
				.measurementIterations(10)
				.warmupIterations(10)
				.build();
		new Runner(opts).run();
	}

}

因为是maven项目，pom.xml中添加了下面的依赖包（前面截图中展示了）：来自博客园

<!-- JMH -->
	<!-- https://mvnrepository.com/artifact/org.openjdk.jmh/jmh-core -->
	<dependency>
	    <groupId>org.openjdk.jmh</groupId>
	    <artifactId>jmh-core</artifactId>
	    <version>1.19</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.openjdk.jmh/jmh-generator-annprocess -->
	<dependency>
	    <groupId>org.openjdk.jmh</groupId>
	    <artifactId>jmh-generator-annprocess</artifactId>
	    <version>1.19</version>
        <!-- provided, compile 都可以 -->
	    <scope>provided</scope>
	</dependency>

在Eclipse中启动，出现下面的错误：java.lang.RuntimeException: ERROR: Unable to find the resource: /META-INF/BenchmarkList

根据参考文档2 解决了问题：Eclipse安装插件 m2e-apt 1.5.3，并做配置

来自博客园

不过，安装插件并配置后，对其它项目的执行有什么影响吗？还需研究下，TODO

上面使用的 JMH版本是 1.19，运行main函数：此时，笔记本的风扇呼呼地响，CPU使用率不用说，直接100%！/心疼电脑

测试结果

# JMH version: 1.19
# VM version: JDK 1.8.0_202, VM 25.202-b08
# VM invoker: D:\Program Files\Java\jdk1.8.0_202\jre\bin\java.exe
# VM options: -Dfile.encoding=UTF-8
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: jmh.Jmh1.arrayListAdd

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 0.015 us/op
# Warmup Iteration   2: 0.024 us/op
# Warmup Iteration   3: 0.015 us/op
# Warmup Iteration   4: 0.016 us/op
# Warmup Iteration   5: 0.017 us/op
# Warmup Iteration   6: 0.012 us/op
# Warmup Iteration   7: 0.021 us/op
# Warmup Iteration   8: 0.010 us/op
# Warmup Iteration   9: 0.010 us/op
# Warmup Iteration  10: 0.012 us/op
Iteration   1: 0.011 us/op
Iteration   2: 0.020 us/op
Iteration   3: 0.016 us/op
Iteration   4: 0.027 us/op
Iteration   5: 0.020 us/op
Iteration   6: 0.013 us/op
Iteration   7: 0.023 us/op
Iteration   8: 0.017 us/op
Iteration   9: 0.011 us/op
Iteration  10: 0.020 us/op


Result "jmh.Jmh1.arrayListAdd":
  0.018 ±(99.9%) 0.008 us/op [Average]
  (min, avg, max) = (0.011, 0.018, 0.027), stdev = 0.005
  CI (99.9%): [0.010, 0.025] (assumes normal distribution)


# JMH version: 1.19
# VM version: JDK 1.8.0_202, VM 25.202-b08
# VM invoker: D:\Program Files\Java\jdk1.8.0_202\jre\bin\java.exe
# VM options: -Dfile.encoding=UTF-8
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: jmh.Jmh1.linkedListAdd

# Run progress: 50.00% complete, ETA 00:00:23
# Fork: 1 of 1
# Warmup Iteration   1: 0.458 us/op
# Warmup Iteration   2: 0.513 us/op
# Warmup Iteration   3: 0.115 us/op
# Warmup Iteration   4: 0.268 us/op
# Warmup Iteration   5: 0.113 us/op
# Warmup Iteration   6: 0.275 us/op
# Warmup Iteration   7: 0.271 us/op
# Warmup Iteration   8: 0.279 us/op
# Warmup Iteration   9: 0.099 us/op
# Warmup Iteration  10: 0.279 us/op
Iteration   1: 0.097 us/op
Iteration   2: 0.269 us/op
Iteration   3: 0.095 us/op
Iteration   4: 0.249 us/op
Iteration   5: 0.095 us/op
Iteration   6: 0.258 us/op
Iteration   7: 0.093 us/op
Iteration   8: 0.256 us/op
Iteration   9: 0.092 us/op
Iteration  10: 0.260 us/op


Result "jmh.Jmh1.linkedListAdd":
  0.176 ±(99.9%) 0.131 us/op [Average]
  (min, avg, max) = (0.092, 0.176, 0.269), stdev = 0.087
  CI (99.9%): [0.046, 0.307] (assumes normal distribution)


# Run complete. Total time: 00:01:18

Benchmark           Mode  Cnt  Score   Error  Units
Jmh1.arrayListAdd   avgt   10  0.018 ± 0.008  us/op
Jmh1.linkedListAdd  avgt   10  0.176 ± 0.131  us/op

最后两行展示如下：说明 ArrayList 的add的性能更好。

# Run complete. Total time: 00:01:18

Benchmark           Mode  Cnt  Score   Error  Units
Jmh1.arrayListAdd   avgt   10  0.018 ± 0.008  us/op
Jmh1.linkedListAdd  avgt   10  0.176 ± 0.131  us/op

测试中、测试后的JVM进程对比：测试中多了 Jmh1、ForkedMain两个进程

正常测试完毕，接下来，使用最新版的 JMH 1.33进行测试——修改pom.xml文件。

执行测试：出现异常 java.lang.OutOfMemoryError: Java heap space，发生在 this.arrayList.add(DATA) 时。

# Warmup Iteration   1: <failure>

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.ArrayList.grow(ArrayList.java:265)

注，上面是没有捕获异常时的输出。

下面是本文展示的捕获异常时的输出：第一个warmup居然执行了一亿五千多万次add！然后就发生了一次。

这种情况下，LinkedList的测试是正常的，但ArrayList是没有测试结果的。

最新版本1.33为什么会这样呢？怎么解决？TODO

1.19版本执行时，ArrayList执行了多少次呢？输出每次++后的cnt：才八万多！和1.33的亿级完全没法比啊！难怪1.33会OOM

---210923 2209---

解决1.33的OOM问题：来自博客园

检查@Benchmark注解所在包下的注解，原来还有 @Warmup、@Measurement ，按照参考文档1的说法，这两个也是用来控制测试过程的，可以添加到类和方法上。

在main方法中，使用 OptionsBuilder 来建立Options对象，发现其下有多个 warmup、measurement开头的方法，是否使用这些方法可以来避免OOM呢？可以的。

下面的调用解决了问题：增加了一个 warmupTime、measurementTime 的调用，限定每次预热、测试在1秒以内——这样就不会调用add超亿次了吧（我的电脑太好、太快了？）

	public static void main(String[] args) throws RunnerException {
		final Options opts = new OptionsBuilder()
				.include(Jmh1.class.getSimpleName())
				.forks(1)
				// 预热：使代码的执行经历过了类的早起优化、JVM运行期编译、JIT优化
				.warmupIterations(10)
				.warmupTime(TimeValue.seconds(1L))
				// 真正的度量操作
				.measurementIterations(10)
				.measurementTime(TimeValue.seconds(1L))
				.build();
		new Runner(opts).run();
	}

更改后的测试结果：使用JMH 1.33成功进行了测试。来自博客园

注，前两行 Warmup、Measurement 后面的 1 s each——正是我们配置的，1.19版本也是这个值，而1.33版本时，这个值是 10 s each——难怪会超亿！

注，也可以使用前面提到的两个注解解决问题。

...
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
...
# Run complete. Total time: 00:01:19

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark           Mode  Cnt  Score   Error  Units
Jmh1.arrayListAdd   avgt   10  0.017 ± 0.012  us/op
Jmh1.linkedListAdd  avgt   10  0.170 ± 0.126  us/op

1.33版本的日志：

...
# Warmup: 10 iterations, 10 s each
# Measurement: 10 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: jmh.Jmh1.arrayListAdd

# Run progress: 0.00% complete, ETA 00:06:40
# Fork: 1 of 1
# Warmup Iteration   1: <failure>

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.ArrayList.grow(ArrayList.java:265)
...

---210924 0657---

》》》全文完《《《

参考文档

1、《Java高并发编程详解深入理解并发核心库》

书，by 汪文君

2、Eclipse Benchmark 基准测试报错：ERROR: Unable to find the resource: /META-INF/BenchmarkList

3、arrays中copyof_Java中：常见的几种内存溢出及解决方案，再遇到后就可以解决了...

4、

码农公寓

相关文章