Forking hiredis and hiredis-cluster

[zuiderkwast]

Hiredis (maintained by @michael-grunder) is used in valkey-benchmark, valkey-cli and valkey-sentinel and is vendored in the valkey repo.

Hiredis-cluster is a separate project (maintained by @bjosv and me), which adds cluster support as wrapper around hiredis.

It would be convenient to fork both of these together in a "monorepo", although separate within the repo, so it's still possible to vendor only the non-cluster part in valkey. It'd be the official C client for Valkey. (In the future we could add sentinel support too.)

When we for this with a new name, would it also be acceptable to do some breaking changes (apart from API rebranding)? Or is it best to avoid it and do them later? What I'm thinking about (maybe longer term roadmap) is:

• Use char * and size_t len for strings instead of sds. It'd possible to parse RESP in a zero-copy style. The returned char * can point directly into the RESP data.
• Do something about pubsub in the async API, where a callback is called multiple times (for subscribe, messages and unsubscribe), making it very hard for a the cluster wrapper to know when the callback private data can be freed. A finalizer callback can solve this, or possibly we can just allow some way to bypass this handling and deliver all pubsub messages to a separate callback.

The name is another important question. Michael suggested libvalkey. Other ideas are hivalkey and valkey-c. (I like libvalkey more.)

[mkmkme]

libvalkey or libvalkeyclient are quite good imho

[michael-grunder]

It would be convenient to fork both of these together in a "monorepo", although separate within the repo, so it's still possible to vendor only the non-cluster part in valley

This is a good idea. Over the years many people have had to be redirected from hiredis to hiredis-cluster for cluster support. Also it would mean tighter coupling/consistency of the two projects.

When we for this with a new name, would it also be acceptable to do some breaking changes (apart from API rebranding)?

I vote yes and would have mentioned removal sds too. Presently, Redis mass renames all sds* symbols to hi_sds* to avoid conflicts.

parse RESP in a zero-copy style

Zero-copy also lets us fix the read buffer so we no longer repeatedly memmove while consuming. This can dominate CPU for some workloads.(redis/hiredis/issues/656).

Another thing worth investigating is an io_uring async adapter. I played around with this years ago but it was very early in liburing's development.

When the project is forked I'm happy to help with all this stuff and keep answering user questions as they find their way there.

[zuiderkwast]

OK, so I think we need a repo where we can start copying both projects, rebrand and then work on it a bit before we can release it officially.

We should focus on API changes first. Io_uring can't be a breaking change, so I think it can wait a bit.

Rebanding includes the API functions and names of include files. I suggest "valkey" prefix for functions like redisCommand => valkeyCommand, and for cluster redisClusterCommand => valkeyClusterCommand, etc. The names of libs and include files too. Currently we have <hiredis/hiredis.h> for the sync API and <hiredis/async.h> for the async API (slightly unintuitive).

[mkmkme]

Please also ping me when the roadmap for hiredis and hiredis-py are ready. I would love to participate!