Skip to content
Search

PerformanceTuning:DiskIO

lastmod 2025-01-09created 2014-02-2613 edits

A summary of performance tuning issues for disk I/O.

Optimal buffer size

As discussed in comments, I believe the exact size don't matter that much, assuming it is :

  1. a small multiple of the file system size (see comment by Joachim Pileborg suggesting stat(".") etc.)
  2. a power of two (because computers and kernels like them)
  3. not too big (e.g. fitting in some cache inside your processor, e.g. L2 cache)
  4. aligned in memory (e.g. to a page size using posix_memalign).

So a power of two between 16kbytes and a few megabytes should probably fit. Most of the time is spent on reading the disk. Filesystem and disk benchmarks are quite flat in that range.

4Kbytes seems to often be the page size and the disk chunk size.

Of course, you can tune things, even tune, when making the file system with mke2fs, the file system block size.

And I'll bet that the optimal is really dependent upon your hardware (SSD, hard disks?) and your system (and its load).

Server Operations Guide: I/O Request Size

The size of requests and the rate at which they are sent are important for evaluating the way applications work with the disk. If you are an application developer, you can use the counters, such as Avg. Disk Bytes/Read, that reveal these types of information about I/O requests.

It is typically faster and more efficient to read a few large records than many small ones. However, transfer rates eventually peak due to the fact that the disk is moving blocks of data so large that each transfer occurs more slowly—although its total throughput is quite high. Unfortunately, it is not always easy to control this factor. However, if your system is used to transfer many small units of data, this inefficiency might help to explain, though not resolve, high disk use.

Requests should be at least 8 KB, and, if possible, 64 KB. Sequential I/O requests of 2 KB consume a substantial amount of processor time, which affects overall system performance. However, if you can be sure that only 2 KB of data is necessary, doing a 2 KB I/O is the most efficient, because a larger I/O wastes direct memory access (DMA) controller bandwidth. As the record size increases, the throughput increases and the transfer rate falls because it takes fewer reads to move the same amount data.

Using 64 KB requests results in faster throughput with very little processor time. Maximum throughput typically occurs at 64 KB, although some devices might have a higher maximum throughput size. When transferring data blocks greater than 64 KB, the I/O subsystem breaks the transfers into 64-KB blocks. Above 64 KB, the transfer rate drops sharply, and throughput levels off. Processor use and interrupts also appear to level off at 64 KB.

Copying data from SSD to HDD and optimization (Ubuntu)

Explains how to copy data from an SSD to an HDD on an Ubuntu system without going through the CPU, or how to transfer data as fast as possible.

Copying with the dd command (low-level copy)

The dd command copies data block by block, allowing fast copying with minimal CPU use.

Command: 

sudo dd if=/dev/sdX of=/dev/sdY bs=64M status=progress

Explanation:

  • if=/dev/sdX: Source disk (SSD) device file
  • of=/dev/sdY: Target disk (HDD) device file
  • bs=64M: Sets the block size (a larger block size improves speed)
  • status=progress: Shows progress

Caveats:

  • If the copy target (HDD) has important data, back it up beforehand.
  • Since the entire disk is copied, partition information is cloned as is.

Copying files with rsync

rsync is an efficient and fast way to copy at the file level.

Command: 

sudo rsync -avh --progress /mnt/ssd/ /mnt/hdd/

Explanation:

  • -a: Archive mode (preserves file permissions and timestamps)
  • -v: Verbose output
  • -h: Human-readable output
  • --progress: Shows copy progress

The cp command

For a simple copy, you can use the cp command.

Command: 

sudo cp -a /mnt/ssd/. /mnt/hdd/

Explanation:

  • -a: Copies while preserving file permissions and attributes
  • .: Copies all contents

High-performance file system copy (using tar)

To move data quickly without compression, you can use the tar command.

Command: 

tar cf - -C /mnt/ssd . | tar xf - -C /mnt/hdd

File system snapshot copy

Using a snapshot-capable file system such as LVM or ZFS can speed up copying.

Creating and copying an LVM snapshot

Create a snapshot: 

lvcreate --size 10G --snapshot --name snap_name /dev/volume_group/logical_volume

Copy the snapshot to the HDD: 

dd if=/dev/volume_group/snap_name of=/dev/sdY bs=64M status=progress

NVMe-to-NVMe Direct Storage Copy

With NVMe SSDs, you can use nvme-cli to copy directly between disks. However, this method is not suitable for regular HDDs.

Optimization tips

  • File system mount options: Mount with the noatime option to reduce I/O overhead. 
sudo mount -o noatime /dev/sdX /mnt/ssd
sudo mount -o noatime /dev/sdY /mnt/hdd
  • Read/write caching: Enable the disk read/write cache with hdparm. 
sudo hdparm -W1 /dev/sdY
  • I/O scheduler tuning: For HDD disks, setting the deadline scheduler is suitable. 
echo deadline | sudo tee /sys/block/sdY/queue/scheduler
  • Network transfer: If the SSD and HDD are on different systems, you can combine rsync with SSH to increase speed.

Conclusion

Of the methods above, dd and rsync are the most widely used and efficient data copy methods in Ubuntu environments. Choose the method that fits your purpose and data size.

See also

Favorite site

References

  1. File_Buffering_(Windows).pdf ↩

  2. Presentation_-Kyungroh_Kim-_Embedded_System_Random_Performance_Issue.pdf ↩

author Zer0 sitemap