referee:Java Programming Tutorial Advanced Input & Output (I/O)
JDK 1.4+ introduced the so-called New I/O int java.nio package and its auxiliary packages to support high performance and intensive I/O operations. NIO is meant to complement the existing stanard java.io, not as a replacement.
Direct vs Indirect Buffers:
A buffer can be direct or indirect. For a direct buffer, "the JVM will make a best effort to perform native I/O operations directly upon it. That is, it will attempt to avoid copying the buffer's content to (or from) an intermediate buffer before (or after) each invocation of one of the underlying operating system's native I/O operations." In other words, direct buffer is more efficient.
For byte buffer, you can allocate a direct ByteBuffer via the allocateDirect(int capacity) method. For other buffers (char, short, int, long, float, double), you need to first allocate a ByteBuffer, and then create a view via methods such as asFloatBuffer(). As these primitive types have unit of multiple bytes (e.g., an int is 4 bytes), you need to specify the byte order of either big-endian (big byte first) or little-endian (little byte first) via order(ByteOrder order) method. The order could be ByteOrder.BIG_ENDIAN, ByteOrder.LITTLE_ENDIAN, or ByteOrder.nativeOrder() which returns the native byte order of the underlying platform for you to write portable program.
ByteBuffer: ByteBuffer is special. To summarize:
- It is used in channel I/O (see channel I/O below).
- You can allocate
ByteBufferas direct. In this case, the JVM will make a best effort to perform native I/O directly for better performance. - You can create a view as other buffer, such as FloatBuffer via
asFloatBuffer(). - You can get/put as other primitive types via
getXxx()andputXxx(). -
MapByteBufferfor mapped I/O
Here is example about direct buffer:
static void directIndirectBuf() {
float[] vertices = {
0.0f, 1.0f, 0.0f, // top (x, y, z)
-1.0f, -1.0f, 0.0f, // left-bottom (x, y, z)
1.0f, -1.0f, 0.0f // right-bottom (x, y, z)
};
FloatBuffer vertexBuffer;
//set up vertex-array buffer. Vertices in float.
//Allocate a direct ByteBuffer for the vertices. A float has 4 bytes.
ByteBuffer vbb = ByteBuffer.allocate(vertices.length * 4);
//set the byte order (big-endian or little endian) to the native
//byte order of the underlying platform for portable program.
vbb.order(ByteOrder.nativeOrder());
// Create a direct FloatBuffer as a view of this ByteBuffer.
// Position is 0.
vertexBuffer = vbb.asFloatBuffer();
vertexBuffer.put(vertices);
// Rewind by setting position to 0
vertexBuffer.position(0);
for (int i = 0; i < vertexBuffer.capacity(); i++) {
System.out.print(vertexBuffer.get(i) + " ");
}
}
1.2 java.nio.MappedByteBuffer
A direct byte buffer whose content is a memory-mapped region of a file.Mapped byte buffers are created via the FileChannel.map method. This class extends the ByteBuffer class with operations that are specific to memory-mapped file regions.A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.
The content of a mapped byte buffer can change at any time, for example if the content of the corresponding region of the mapped file is changed by this program or another. Whether or not such changes occur, and when they occur, is operating-system dependent and therefore unspecified.
All or part of a mapped byte buffer may become inaccessible at any time, for example if the mapped file is truncated. An attempt to access an inaccessible region of a mapped byte buffer will not change the buffer's content and will cause an unspecified exception to be thrown either at the time of the access or at some later time. It is therefore strongly recommended that appropriate precautions be taken to avoid the manipulation of a mapped file by this program, or by a concurrently running program, except to read or write the file's content.Mapped byte buffers otherwise behave no differently than ordinary direct byte buffers.
1.3 Channel （java.nio.channels.Channel）
A channel represents a connection to a physical I/O device (such as file, network socket, or even another program). It si similar to standard I/O's stream, but a more platform-dependent version of stream. Becase channels hava a closer ties to the underlying platform, they can achieve better I/O thoughtput.
Types of channels are FileChannel, SocketChannel, DatagramChannel.A Channel object can be obtained by calling the getChannel() methods of classes such as java.io.FileInputStream, java.io.FileOutputStream, java.io.RandomAccessFile,java.net.Socket, java.net.ServerSocket, java.net.DatagramSocket, and java.net.MulticastSocket.
For example:
File file = new File("/users/wsy/Documents/job/wok.tar");
FileInputStream in = new FileInputStream(file);
FileChannel channel = in.getChannel();
A FileChannel obtained from a FileInputStream is read-only; while a FileChannel obtained from a FileOutputStream is write-only. While stream I/O processes one byte at at a time; Channel I/O processes a buffer once.
/**
* Reads a sequence of bytes from this channel into the given buffer.
*
* <p> Bytes are read starting at this channel's current file position, and
* then the file position is updated with the number of bytes actually
* read. Otherwise this method behaves exactly as specified in the {@link
* ReadableByteChannel} interface. </p>
*/
public abstract int read(ByteBuffer dst) throws IOException; /**
* Writes a sequence of bytes to this channel from the given buffer.
*
* <p> Bytes are written starting at this channel's current file position
* unless the channel is in append mode, in which case the position is
* first advanced to the end of the file. The file is grown, if necessary,
* to accommodate the written bytes, and then the file position is updated
* with the number of bytes actually written. Otherwise this method
* behaves exactly as specified by the {@link WritableByteChannel}
* interface. </p>
*/
public abstract int write(ByteBuffer src) throws IOException;
We can also transfer data between an input channel and an touput channel directly via:
public abstract long transferFrom(ReadableByteChannel src,
long position, long count)
throws IOException; public abstract long transferTo(long position, long count,
WritableByteChannel target)
throws IOException;
Here is an example copying file using different methods:
1 Using FileChannel with indirect ByteBuffer
static void copyFileFileChannelIndirectMem() throws IOException{
String inFileStr = "/users/wsy/Documents/job/kimchi_v2.pdf";
String outFileStr = "./kimchi_v2.pdf";
long startTime, elapsedTime; // for speed benchmarking
int bufferSizeKB = 4;
int bufferSize = bufferSizeKB * 1024;
// Check file length
File fileIn = new File(inFileStr);
System.out.println("File size is " + fileIn.length() + " bytes");
System.out.println("Buffer size is " + bufferSizeKB + " KB");
// Using FileChannel with indirect ByteBuffer
System.out.println("Using FileChannel with indirect ByteBuffer of " + bufferSizeKB + " KB");
try (FileChannel in = new FileInputStream(inFileStr).getChannel();
FileChannel out = new FileOutputStream(outFileStr).getChannel();) {
// Allocate an indirect ByteBuffer
ByteBuffer bytebuf = ByteBuffer.allocate(bufferSize);
startTime = System.nanoTime();
boolean bytesCount;
while((bytesCount = in.read(bytebuf) > 0)){
// flip the buffer which set the limit to current position, and position to 0.
bytebuf.flip();
out.write(bytebuf);
bytebuf.clear();
}
elapsedTime = System.nanoTime() - startTime;
System.out.println("Elapsed Time is "
+ (elapsedTime / 1000000.0) + " msec");
} catch (IOException ex){
ex.printStackTrace();
}
}
2 Using FileChannel with direct memory
static void copyFileFileChannelDirectMem() throws IOException{
String inFileStr = "/users/wsy/Documents/job/kimchi_v2.pdf";
String outFileStr = "./kimchi_v2.pdf";
long startTime, elapsedTime; // for speed benchmarking
int bufferSizeKB = 4;
int bufferSize = bufferSizeKB * 1024;
// Check file length
File fileIn = new File(inFileStr);
System.out.println("File size is " + fileIn.length() + " bytes");
System.out.println("Buffer size is " + bufferSizeKB + " KB");
// Using FileChannel with indirect ByteBuffer
System.out.println("Using FileChannel with direct ByteBuffer of " + bufferSizeKB + " KB");
try (FileChannel in = new FileInputStream(inFileStr).getChannel();
FileChannel out = new FileOutputStream(outFileStr).getChannel();) {
// Allocate an indirect ByteBuffer
ByteBuffer bytebuf = ByteBuffer.allocateDirect(bufferSize);
startTime = System.nanoTime();
boolean bytesCount;
while((bytesCount = in.read(bytebuf) > 0)){
// flip the buffer which set the limit to current position, and position to 0.
bytebuf.flip();
out.write(bytebuf);
bytebuf.clear();
}
elapsedTime = System.nanoTime() - startTime;
System.out.println("Elapsed Time is "
+ (elapsedTime / 1000000.0) + " msec");
} catch (IOException ex){
ex.printStackTrace();
}
}
3 Using Buffered Stream I/O
static void CpFileStreamIO () throws IOException {
String inFileStr = "/users/wsy/Documents/job/kimchi_v2.pdf";
String outFileStr = "./kimchi_v2.pdf";
long startTime, elapsedTime; // for speed benchmarking
int bufferSizeKB = 4;
int bufferSize = bufferSizeKB * 1024;
System.out.println("Using Buffered Stream");
try( BufferedInputStream in = new BufferedInputStream(new FileInputStream(inFileStr));
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(outFileStr))) {
startTime = System.nanoTime();
int bytesCount;
while ((bytesCount = in.read()) != -1){
out.write(bytesCount);
}
elapsedTime = System.nanoTime() - startTime;
System.out.println("Elapsed time is " + (elapsedTime / 1000000.0) + " msec");
} catch (IOException ex){
ex.printStackTrace();
}
}
4 Using FileChannel with transferTo()
private static void CpFileChannel() {
String inFileStr = "/users/wsy/Documents/job/kimchi_v2.pdf";
String outFileStr = "./kimchi_v2.pdf";
long startTime, elapsedTime; // for speed benchmarking
int bufferSizeKB = 4;
int bufferSize = bufferSizeKB * 1024;
System.out.println("Using Buffered Stream");
try( FileChannel in = new FileInputStream(inFileStr).getChannel();
FileChannel out = new FileOutputStream(outFileStr).getChannel()) {
startTime = System.nanoTime();
in.transferTo(0, in.size(), out);
elapsedTime = System.nanoTime() - startTime;
System.out.println("Elapsed time is " + (elapsedTime / 1000000.0) + " msec");
} catch (IOException ex){
ex.printStackTrace();
}
}
5 Using a programmer-managed 4K byte-array for Disk I/O
static void CpFileDiskIO () throws IOException {
String inFileStr = "/users/wsy/Documents/job/kimchi_v2.pdf";
String outFileStr = "./kimchi_v2.pdf";
long startTime, elapsedTime; // for speed benchmarking
int bufferSizeKB = 4;
int bufferSize = bufferSizeKB * 1024;
System.out.println("Using a programmer-managed byte-array of " + bufferSizeKB + " KB");
try(FileInputStream in = new FileInputStream(inFileStr);
FileOutputStream out = new FileOutputStream(outFileStr)) {
startTime = System.nanoTime();
// Create byte array
byte[] byteArray = new byte[bufferSize];
int bytesCount;
while ((bytesCount = in.read(byteArray)) != -1){
out.write(byteArray, 0, bytesCount);
}
elapsedTime = System.nanoTime() - startTime;
System.out.println("Elapsed time is " + (elapsedTime / 1000000.0) + " msec");
} catch (IOException ex){
ex.printStackTrace();
}
}
FileChannel with a 4K direct Bytebuffer is faster than indirect ByteBuffer. Buffered Stream I/O is much slower than FileChannel. While user-managed byte-array is faster than some channels.
The following table compare the run-time for various buffer size with (a) Using FileChannel with an indirect ByteBuffer, (b) Using FileChannel with a direct ByteBuffer, (c) Using FileChannel withtransferTo(), (d) Using Buffered Stream, (e) Using a programmer-managed byte-array.
BufSize (a) (b) (c) (d) (e)
4KB 16.67 9.73 3.33 124.21 7.72
16KB 6.92 3.39 1.86 110.85 4.06
32KB 3.95 2.75 1.76 109.60 2.90
64KB 3.26 2.15 1.88 109.77 2.96
128KB 2.77 2.11 2.02 109.64 2.59
256KB 2.49 1.66 1.80 109.10 2.55
1024KB 3.57 1.86 1.97 109.08 5.88
1.4 Selector
A number of channels can be registered with a selector (java.nio.channels.Selector). A selector provides a mechanism for waiting on channels until one ore more become available for data transfer. It can be used to block the program until at least one channel is available for use. Examples are server applications that involves simultaneously waiting for responses on a number of session.
1.5 Character Set(CharSet)
waiting for complement