我正在阅读Hadoop IPC实现.
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
/**
* When the read or write buffer size is larger than this limit, i/o will be
* done in chunks of this size. Most RPC requests and responses would be
* be smaller.
*/
private static int NIO_BUFFER_LIMIT = 8*1024; //should not be more than 64KB.
/**
* This is a wrapper around {@link WritableByteChannel#write(ByteBuffer)}.
* If the amount of data is large, it writes to channel in smaller chunks.
* This is to avoid jdk from creating many direct buffers as the size of
* buffer increases. This also minimizes extra copies in NIO layer
* as a result of multiple write operations required to write a large
* buffer.
*
* @see WritableByteChannel#write(ByteBuffer)
*/
private int channelWrite(WritableByteChannel channel,
ByteBuffer buffer) throws IOException {
int count = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
channel.write(buffer) : channelIO(null, channel, buffer);
if (count > 0) {
rpcMetrics.incrSentBytes(count);
}
return count;
}
/**
* This is a wrapper around {@link ReadableByteChannel#read(ByteBuffer)}.
* If the amount of data is large, it writes to channel in smaller chunks.
* This is to avoid jdk from creating many direct buffers as the size of
* ByteBuffer increases. There should not be any performance degredation.
*
* @see ReadableByteChannel#read(ByteBuffer)
*/
private int channelRead(ReadableByteChannel channel,
ByteBuffer buffer) throws IOException {
int count = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
channel.read(buffer) : channelIO(channel, null, buffer);
if (count > 0) {
rpcMetrics.incrReceivedBytes(count);
}
return count;
}
逻辑是,
如果缓冲区很小,它将一次读/写通道.如果缓冲区很大,它会多次执行,每次读/写8kb.
我不明白javadocs以及为什么这样做.
为什么“这是为了避免jdk在缓冲区大小增加时创建许多直接缓冲区.”?
大缓冲区大小是否也会影响读取性能?
我理解缓冲区大小如何影响FileInputStream性能(link).但这里是SocketChannel.所以它是无关的.
解决方法:
好问题.在通道中写入时使用sun.nio.ch.IOUtil,并且在其write(..)函数中有以下行
int var7 = var5 <= var6?var6 - var5:0;
ByteBuffer var8 = Util.getTemporaryDirectBuffer(var7);
这是Util.getTemporaryDirectBuffer
static ByteBuffer getTemporaryDirectBuffer(int var0) {
Util.BufferCache var1 = (Util.BufferCache)bufferCache.get();
ByteBuffer var2 = var1.get(var0);
if(var2 != null) {
return var2;
} else {
if(!var1.isEmpty()) {
var2 = var1.removeFirst();
free(var2);
}
return ByteBuffer.allocateDirect(var0);
}
}
在负载较重的情况下,当int var0处于较大范围内时,它会创建大量新缓冲区并释放旧的缓冲区(..).因为bufferCache的长度有限(等于系统配置中定义的IOUtil.IOV_MAX.在现代Linux系统上,限制为1024)并且不会存储每个长度的缓冲区.
我认为这意味着这是为了避免jdk在缓冲区大小增加时创建许多直接缓冲区.