RKEY size

[Blopeur]

HI is there any way to know in advance the key size ( using JUCX) or at least is their a max size ?

We are experiencing some strange variation , where depending on how the UCX library was compiled with different option, the RKEY size vary on one laptop it's 25B on another it's 89B ... server it's again 25B...
This makes it a bit difficult if we want to send a bunch of RKEY viat tagged message. We can't really predict what the default maximum number of RKEY we can pack.

Any documentation that describe the RKEY structure , it's size?

[Blopeur]

Aslo : Rkey in rdma takes 32 bits (it's in the verbs API). In shared memory it's longer, to acvount for different drivers, e.g. KNEM. You should be fine with a 64-bit value.

if we have 89B . are we passing the entire address size??

[yosefe]

Hi,
UCP remote key contains packed rkeys of all underlying transports (including rdma and shared memory, along with meta data) because it's not possible to know up front if it's going to be used for inter or intra node access. So it depends on which transports are enabled and present on the machine.
I would suggest using tag probe API to check the size of the incoming rkey message, and then allocate a buffer with the right size for it.

[Blopeur]

does the RKEY size change during the lifetime of a connection? if it's the same, I can generate the RKEY once during connection establishment negotiation and then use the size a max size for the lifetime of the connection

[yosefe]

The rkey size does not depend on a specific connection. In fact, it can be serialized right after calling ucp_mem_map() which needs only the top-level ucp_context_h.

[Blopeur]

ok, so for the lifetime of the context we have the same RKEY size .. excellent,, thanks!

[yosefe]

yes. but when receiving rkey from a peer process, it's not guaranteed the remote key size would be same as local rkey size (unless the setup is identical)