Unexpected results from benchmarking (IOPs and Throughput)

[yoketong-t]

We are trying to evaluate JuiceFS and came across some unexpected results from our benchmarks. Would like to see if there are any further insights from the community.

Environment:

JuiceFS: 0.16.2 (2021-08-25T04:01:15Z 29d6fee)
Operating System: CentOS 7.8 (Virtual Machine - 18 Cores, 16GB RAM)
Object storage: minio (backed by local dir)
Metadata engine: Redis (running in a docker container)

What happened:

Unexpected performance from benchmarks in IOPs and throughput when writeback is enabled.

1. Small File Sequential Write

fio --name=small-file-seq-write --directory=/mnt/jfs --rw=write --file_service_type=sequential --bs={file_size} --filesize={file_size} --nrfiles=1000 --end_fsync=1

• When writeback is enabled, expecting little fluctuation of IOPs when varying file/block sizes due to commands overhead. However, observed decreasing IOPs with increasing file/block size

2. Small Files Concurrent Write

fio --name=small-file-multi-write --directory=/mnt/jfs --rw=write --file_service_type=sequential --bs=4k --filesize=4k --nrfiles=1000 --numjobs={1, 2, 4, 8, 16} --end_fsync=1

Throughput

• When writeback is enabled, expecting increasing throughput with increasing number of concurrent jobs but having a decrease in throughput when number of concurrent jobs increases from 8 to 16

IOPs

• When writeback is enabled, expecting little fluctuation when varying number of concurrent jobs due to commands overhead. However, observed irregular increase in IOPs with increasing number of concurrent jobs with decrease in IOPs when number of concurrent jobs increases from 8 to 16.

What you expected to happen:

• Example of expected behavior when "writeback" is enabled. IOPs capped due to commands overhead.
  

[SandyXSD]

Normally, blocks of a file are synchronously uploaded to the object storage when the file is fsynced. If writeback mode is enabled, blocks are only guaranteed to be written locally (in the cache-dir), and will be uploaded in the background. In other words, the throughtput with writeback is still limited by the speed of local disk and bandwidth of the object storage (only when the cache-dir is too full)

[yoketong-t]

I think that explains the situation I observed for "1. Small File Sequential Write". Albeit having a higher throughput, IOPs is greatly reduced due to cache-dir being too full.

How about the situation for "2. Small Files Concurrent Write" where I observed a dip in throughput from 8 to 16 concurrent jobs and irregular IOPs pattern (especially when I increase from 1 to 2 concurrent jobs ad from 8 to 16 concurrent jobs)

[SandyXSD]

1 to 2: It's not normal. Usually it should get an obvious boost (say, 30+%, not doubled though) from 1 to 2 jobs. Can you run the test more times to see if the result will change?
8 to 16: It's expected, if it has already reached the cap when you have 8 threads, in which case adding more concurrent jobs will only overrun the system.

[yoketong-t]

I have ran the tests more times and took an average of the numbers.

I am seeing the performance numbers cap off / plateau at 4 concurrent jobs and beyond. Which may suggest that the original performance that I saw with 8 concurrent jobs may be one off case.

However, the issues from 1 to 2 concurrent jobs seems to be consistent.

@SandyXSD : Do you have any further insights to the new observations that I had ?