使用-XX:+TraceClassPaths或者在服务器上执行jinfo时，都能得到classpath包含的jar包，例如：
java.class.path = local/aaa/lib/spring-data-redis-1.8.3.RELEASE.jar:/usr/local/aaa/lib/spring-tx-4.3.8.RELEASE.jar:/usr/local/aaa/lib/spring-jdbc-4.3.7.RELEASE.jar:/usr/local/aaa/lib/classmate-1.3.1.jar:/usr/local/aaa/lib/javax.servlet-api-3.1.0.jar:/usr/local/aaa/lib/mongodb-driver-3.4.2.jar:/usr/local/aaa/lib/xml-apis-2.0.2.jar:/usr/local/aaa/lib/ufc-api-utils-2.0.0.jar:/usr/local/aaa/lib/log4j-over-slf4j-1.7.25.jar:/usr/local/aaa/lib/tomcat-embed-websocket-8.5.14.jar:...
这些jar的顺序不同的机器总是不一样的，平时没有问题，所以也没有细想过，这些jar包的顺序为什么会不一样的。 在之前排查的一个问题 的结尾还留了一个问题，为什么有的机器会加载正确的类，有的就是错的。因为这一段在上线一个项目，灰度公测阶段，所以拖了些天，穿插着看了看加载相关的一些hotspot代码，以前也没看过，就边猜测边看了。因为是穿插着看，怕今天看的明天就忘了，所以博客结尾连接中流水账记录了我看的过程。
总之，经过一番查代码，最终在rt.jar中找到了JarFile这个类，代码在jdk的src.zip包里的。 用 https://github.com/saaavsaaa/warn-report/blob/master/src/main/java/report/btrace/JarBTrace.java 脚本跟踪了一下：
正常的服务器脚本的输出：
异常的服务器上脚本输出：
这个问题在多台服务器上都出现了，所以搜集多台的输出后可以确认这个规律，其实在结尾连接中代码记录里也可以看出，如果有多个同名的类只会加载其中第一个。在不出问题的服务器上commons-codec-1.10.jar是在http-1.1.0.jar之前加载的，而出问题的服务器正好相反。
/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar!/sun/misc/URLClassPath.class中的private ArrayList path有所有加载的jar的URL。这个类里还有每个jar的loader：ArrayList<URLClassPath.Loader> loaders，jar路径和jarLoader的映射：HashMap<String, URLClassPath.Loader> lmap。lmap通过private synchronized URLClassPath.Loader getLoader(int var1)方法从Stack urls中弹出的jar来构造loader并加入映射的。有两个URLClassLoader加载不同的URLClassPath实例，一个只有jre/lib下的几个jar，一个有所有的。不过这都不重要，这些jar都是从classpath加载的。于是我在几个服务器上用jinfo输出了java.class.path，并对比了一下发现正常服务器和服务器中，这两个jar的顺序果然是不一样的。那么我估计问题是出现jvm对java.class.path赋值的前面了，或许是操作系统什么的。
于是决定看看java.class.path初始化的情况，代码在/home/aaa/Github/hotspot/src/share/vm/runtime/arguments.cpp:_java_class_path = new SystemProperty("java.class.path", "", true)。SystemProperty的构造在/home/aaa/Github/hotspot/src/share/vm/runtime/arguments.hpp：
// Constructor
SystemProperty(const char* key, const char* value, bool writeable) {
if (key == NULL) {
_key = NULL;
} else {
_key = AllocateHeap(strlen(key)+1, mtInternal);
strcpy(_key, key);
}
if (value == NULL) {
_value = NULL;
} else {
_value = AllocateHeap(strlen(value)+1, mtInternal);
ba(_value, value);
}
_next = NULL;
_writeable = writeable;
}
};
构造里似乎没什么关系，然后回到cpp仔细看了下，发现：
// following are JVMTI agent writeable properties.
// Properties values are set to NULL and they are
// os specific they are initialized in os::init_system_properties_values(). // Set OS specific system properties values
os::init_system_properties_values();
这个方法的定义在os.hpp中，不过实现是不同的系统不一样，所以并没有在对应的cpp中，我是linux系统，所以我去找了 /home/aaa/Github/hotspot/src/os/linux/vm/os_linux.cpp，不过也没有我想要的。arguments.cpp中只找到了对endorsed，ext，bootclasspath的目录文件读取。
然后找到了-XX:+TraceClassPaths参数的输出位置： [classpath: ...]的调用链：
/home/aaa/Github/hotspot/src/share/vm/runtime/thread.cpp:
// Parse arguments
jint parse_result = Arguments::parse(args);
// Parse JAVA_TOOL_OPTIONS environment variable (if present)
jint result = parse_java_tool_options_environment_variable(&scp, &scp_assembly_required);
if (result != JNI_OK) {
return result;
} // Parse JavaVMInitArgs structure passed in
result = parse_each_vm_init_arg(args, &scp, &scp_assembly_required, Flag::COMMAND_LINE);
if (result != JNI_OK) {
return result;
} // Parse _JAVA_OPTIONS environment variable (if present) (mimics classic VM)
result = parse_java_options_environment_variable(&scp, &scp_assembly_required);
if (result != JNI_OK) {
return result;
}
parse_java_tool_options_environment_variable和parse_java_options_environment_variable都有调用parse_each_vm_init_arg，这里提一句_JAVA_OPTIONS会覆盖JAVA_TOOL_OPTIONS的同key配置。
parse_each_vm_init_arg这方法一进来就看到这么一段略坑，不过也无所谓了响。。。
if (!match_option(option, "-Djava.class.path", &tail) &&
!match_option(option, "-Dsun.java.command", &tail) &&
!match_option(option, "-Dsun.java.launcher", &tail)) { // add all jvm options to the jvm_args string. This string
// is used later to set the java.vm.args PerfData string constant.
// the -Djava.class.path and the -Dsun.java.command options are
// omitted from jvm_args string as each have their own PerfData
// string constant object.
build_jvm_args(option->optionString);
}
这就是打印上面那句的地方：
jint Arguments::parse_each_vm_init_arg
Arguments::fix_appclasspath():
if (!PrintSharedArchiveAndExit) {
ClassLoader::trace_class_path(tty, "[classpath: ", _java_class_path->value());
}
然后[Bootstrap loader class path=/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/classes]这一句输出:
/home/aaa/Github/hotspot/src/share/vm/runtime/init.cpp
jint init_globals() /home/aaa/Github/hotspot/src/share/vm/classfile/classLoader.cpp
void classLoader_init() {
ClassLoader::initialize();
} void ClassLoader::initialize() void ClassLoader::setup_bootstrap_search_path()
大体上代码都在这一部分了，可以看出代码中并没有对加载顺序有所改变，然后比较重要的线索就是加载方法用的是opendir和readdir函数，在ios同事的协助下，终于确定了，问题就是jar的加载顺序问题，而这个顺序实际上是由文件系统决定的，linux内部是用inode来指示文件的。
于是写了个简单的c方法来验证，打印inode号：https://github.com/saaavsaaa/warn-report/blob/master/src/main/java/report/main.c
d_off:4066763678044974396 d_name: commons-codec-1.10.jar
d_off:4317940688349827723 d_name: commons-lang-2.5.jar
d_off:4319988583919237504 d_name: jul-to-slf4j-1.7.25.jar
d_off:4356646752930279233 d_name: spring-aop-4.3.8.RELEASE.jar
d_off:4579877846802590742 d_name: tomcat-embed-core-8.5.14.jar
d_off:4731608207059974753 d_name: groovy-2.4.3.jar
d_off:4771541818858258978 d_name: jpush-client-3.2.9.jar
d_off:4817159055427520488 d_name: httpcore-4.3.jar
d_off:5037058976412869958 d_name: rocketmq-common-3.2.6.jar
d_off:5112026581585883935 d_name: xmlParserAPIs-2.6.2.jar
d_off:5223887631628650300 d_name: commons-fileupload-1.2.2.jar
d_off:5270962929984509613 d_name: logback-core-1.1.11.jar
d_off:5295594039709144434 d_name: jackson-core-2.8.8.jar
d_off:5313324231349868064 d_name: .
d_off:5369671282559728007 d_name: spring-webmvc-4.3.8.RELEASE.jar
d_off:5480616600783493234 d_name: xalan-2.6.0.jar
d_off:5598132018198955916 d_name: unbescape-1.1.0.RELEASE.jar
d_off:5620136379821311472 d_name: spring-boot-starter-aop-1.5.3.RELEASE.jar
d_off:5639942392021544761 d_name: mybatis-3.4.4.jar
d_off:5688537857783605585 d_name: validation-api-1.1.0.Final.jar
d_off:5689351964973998306 d_name: json-lib-2.4-jdk15.jar
d_off:5708471019688158398 d_name: tagsoup-0.9.7.jar
d_off:5716046632217600256 d_name: xom-1.0b3.jar
d_off:5736731302988875630 d_name: p2p-repository-1.0-SNAPSHOT.jar
d_off:5770969695350360533 d_name: ognl-3.0.8.jar
d_off:5946256519116188501 d_name: jackson-annotations-2.8.0.jar
d_off:6011005565981751131 d_name: jxl-2.6.jar
d_off:6121401373763401899 d_name: spring-data-mongodb-1.10.3.RELEASE.jar
d_off:6155887434083949861 d_name: icu4j-2.6.1.jar
d_off:6203694621577639938 d_name: jboss-logging-3.3.0.Final.jar
d_off:6241670413890043636 d_name: jaxme-api-0.3.jar
d_off:6317802240634228683 d_name: thymeleaf-2.1.5.RELEASE.jar
d_off:6362118033392674189 d_name: logback-classic-1.1.11.jar
d_off:6391821784225315730 d_name: snakeyaml-1.17.jar
d_off:6447926989912518869 d_name: javassist-3.16.1-GA.jar
d_off:6586996539318953387 d_name: spring-boot-1.5.3.RELEASE.jar
d_off:6682174700565688505 d_name: mybatis-spring-1.3.1.jar
d_off:6858079168157560275 d_name: http-1.1.0.jar
对照前面正常服务器的截图可以发现，顺序和这是一样的。而出错的服务器：
d_off:7500897893766328572 d_name: http-1.1.0.jar
d_off:7630237192571233449 d_name: jaxen-1.1-beta-4.jar
d_off:7846439931967980783 d_name: xom-1.0b3.jar
d_off:7986273690820996399 d_name: jxl-2.6.jar
d_off:8013065173263952359 d_name: spring-boot-starter-thymeleaf-1.5.3.RELEASE.jar
d_off:8231450206996036007 d_name: spring-context-support-4.3.8.RELEASE.jar
d_off:8471297127500795042 d_name: jpush-client-3.2.9.jar
d_off:8635726305307688944 d_name: commons-codec-1.10.jar
同样可以对照上面错误服务器的截图，而且一般情况下，修改了文件名，再改回来，或者从新上传一个，这个编号依然还是这个，所以在出问题的服务器上问题稳定复现。那么，这个问题终于是可以告一段落了，原因搞清楚了，心理就有底了。解决什么的就随便了，我就随便把名字改成zzzhttp然后问题就在两个包都存在的情况下解决了。
d_off:8635726305307688944 d_name: commons-codec-1.10.jar
d_off:8710228832141418589 d_name: xercesImpl-2.6.2.jar
d_off:8722513994996448409 d_name: android-json-0.0.20131108.vaadin1.jar
d_off:8754081964290049159 d_name: ufc-api-utils-2.0.0.jar
d_off:8830796801498266528 d_name: httpcore-4.3.jar
d_off:8854152647200610772 d_name: tomcat-jdbc-8.5.11.jar
d_off:8971846312288780129 d_name: spring-tx-4.3.8.RELEASE.jar
d_off:8986236906371055996 d_name: bcprov-jdk15-1.45.jar
d_off:8988385379950226997 d_name: mysql-connector-java-5.1.30.jar
d_off:8994464230154278010 d_name: p2p-common-1.0-SNAPSHOT.jar
d_off:9012703293696799571 d_name: mybatis-spring-boot-starter-1.3.0.jar
d_off:9057592519440006836 d_name: zzzhttp-1.1.0.jar
流水账记录：https://saaavsaaa.github.io/aaa/Java_Class_Path.html
微信公众号：