Document High Availability Registry Setup

[forsberg]

Feature or Problem Description

I fail to find any documentation on how to configure Apicurio Registry in a high-avalability setup, i.e where one instance can fail, and the other one continues to serve requests.

For Confluent Schema Registry, there's a leader/follower approach, where write requests are redirected to the leader. I fail to find any documentation that explains if there's something similar for Apicurio, or if it's as simple as spinning up multiple instances connected to the same database backend.

Are there pros and cons with the different database backends when it comes to high availability setups?

Proposed Solution

Please clarify in documentation :-)

[dseapy]

related: #2307

[carlesarnal]

Hi @forsberg,

First of all sorry for the late response. As you might have noticed, we have two storage variants for Registry but the answer is essentially the same. As Eric points out in the issue @dseapy linked, HA can be a broad topic but the simplest approach is exactly what you mentioned, just spin up multiple instances connected to the same database (in the SQL storage) or pointing to the same Kafka topic when using the KafkaSQL storage. The former is exactly what we do in our cloud offering multiple Registry nodes pointing to the same PostgreSQL instance. That is for what's relative to Registry itself, other things might depend on your actual setup if you deploy on Kubernetes etc.

Let me know if this helps, if not, I can try to be a bit more extensive :)

[pantaoran]

In my mind, this answer does not address the aspect of what happens when multiple registry instances try to write the same schema simultaneously. In my understanding, in Confluent, the whole leader/follower approach serves to prevent situations where the same schema might be assigned different IDs by the different registry instances. Having only one leader which is the only one allowed to write to the persistent storage (kafka topic) prevents such inconsistent state.

How does Apicurio make sure that two instances cannot register the same schema with different IDs simultaneously?

[pantaoran]

Hi @carlesarnal , could you maybe take a look at my question above for how the HA setup works in Apicurio?

[EricWittmann]

The answer to this is that we utilize the transactional nature of the database to ensure that in the case of identical writes, only one of them is done. In the case of pg/mssql/mysql, this just works as you might expect. Multiple replicas point to the same DB and we structure the DB transaction to ensure that simultaneous identical writes result in only a single row in the content table and as a result a single new version of an artifact (depending on what REST call is made).

For the Kafka storage implementation, the same is true, only it's an in-memory H2 database rather that a shared DB. But every replica processes the messages in the kafka topic in the same order, which ensures that all replicas have identical state.

Does that help at all?

[pantaoran]

Thank you for taking the time to answer! @EricWittmann
In my case I use Kafka storage and don't really care about the DB case, but this shouldn't matter much.

You write "in the case of identical writes, only one of them is done". But I think that my question would be about a case where the writes wouldn't exactly be "identical".
If two instances receive requests for registering the same schema, they will process internal logic to determine a schema ID number, and that number then gets persisted to storage. I'm talking about cases where this internal logic might lead them to assigning different ID numbers for the same schema. If they both then tried to persist that to storage, the writes wouldn't be identical. One instance would write "Schema A -> ID 123" and the other instance would write "Schema A -> ID 321".

Now why would two instances come to different conclusions about the ID for a new schema I don't know, but I think the question must be "can we be absolutely sure this cannot happen"? What if one instance has a long GC pause before persisting to storage, and the other instance assigns two new IDs in the meantime?

Please let me know if you follow my thoughts, it's difficult to write such complex issues in concise messages...

[EricWittmann]

Sorry for the delay - I've been falling behind on GH notifications. So the question is what are we using for uniqueness. The answer is Artifact ID. If you provide an artifact ID when creating an artifact (optional but typically done for e.g. Kafka topic use-cases) then there is no chance that the ID will be different for different creates. If you allow the server to choose an ID, then it will use a UUID for each create operation. In this case, you'll almost certainly get multiple artifacts (UUID is "guaranteed" unique mathematically blah blah).

I would say just make sure you are using the same artifact ID (subject if using the Confluent Compatibility API).

Typically I recommend usage of the the Create Artifact operation, setting the ifExists query param to RETURN_OR_UPDATE and providing a client-supplied Artifact Id.

https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/assets-attachments/registry-rest-api.htm#tag/Artifacts/operation/createArtifact

This way you're guaranteed to either create the artifact only once, even if multiple clients/apps try at the same time. And all clients will get the same response (the metadata of the new artifact). The only time this is a problem is if the clients are all trying to create a new artifact with different schema content. If that happens, a single artifact will still get created, but you will get multiple versions of it (because the clients are uploading different content for the schema).

You can mitigate that problem a bit by also setting the canonical query parameter to true. That will cause the server to use the canonical hash of the content when determining if the content is the same (rather than the default, more strict content hash).