Memory Requirements for io_uring

[yashrs76]

I had asked a question earlier #591 . Wanted some more clarification on the same.

So, I want to understand what amount of memory is allocated (be it in kernel space or userspace) when I ask for a thread to create uring of size 1K entries ? Reading documentation about io_uring_mlock_size it seems to suggest that only that memory which will be mlock’ed will be accounted by this API. Besides in newer kernels this will return zero.

Is there a better API to get the memory requirements for a particular ring size ?

The reason I need to understand the memory requirements is to make an informed decision about what the ideal uring size should be for my application.

[davidzeng0]

The cqring and sqring barely use any memory, only 16 bytes per cqe and 64 bytes per sqe. If you had the maximum size for both rings, you'd only end up using about 3 megabytes.

That's not really what matters though, as the ring size does not limit how many I/O requests you can submit. The rings are only the communication layer between userspace and kernel space, eliminating a memory copy (one which aio and epoll do).

It matters more how many simultaneous outstanding I/O you are performing, as the kernel has to allocate a request for each sqe and can only release it once the request finishes.

Personally, I'd suggest using a sqring size of 256 or 512, which ever one performs better for you, and submit every time the ring fills up. This has a very good balance between number of syscalls and I/O latency (you want to start the requests as early as possible so that you can get a completion as soon as possible).

The cqring you can make as big as you want, but make it especially big if you are handling lots of I/O. If the cqring overflows, the kernel has to allocate an overflow entry to store your result and creates extra overhead.

[davidzeng0]

Which epoll and aio do memory copy*

[yashrs76]

Thanks for the answer @ilikdoge and suggestion for the optimal ring size. I had a few doubts in regard to your answer -

1. What is the value of maximum ring entries that you are talking about? Also are you suggesting that io_uring_mlock_size accounts for this ring sizes ? If so why is it zero in the newer Kernels. Can you please explain.

2. Moreover are the only memory requirements for io_uring are size of sqring + size of cqring.

3. I didn't got the last part where you said we can make cqring as big as one may want. Isn't the size of cqring = 2 * size of sqring . Can we independently change the cqring size without affecting sqring size ?

[davidzeng0]

1. The maximum sqring size is 32768 as permitted by the kernel. The maximum cqring size is 65536 (since it is double the sqring size).
2. AFAIK the only locked memory in use is by the sqring, cqring, and the size of all registered buffers (not provided buffers). Not 100% sure on this.
3. Yes, you can make the cqring any size you want as long as it's atleast the size of the sqring. See io_uring_setup(2)

[davidzeng0]

io_uring_mlock_size seems to only count the locked memory required to create the ring which you were right about in your original question, so it indeed returns zero for newer kernels. I wasn't aware that newer kernels do not use locked memory anymore for the rings.