[compat][da-vinci][test] Global RT DIV improvement (part 1): RT and VT DIV se…

[lluwm]

…paration

Problem:
Data Integrity Validator (DIV) is used for validating and discovering data issues residing in the Kafka topics when consumed. However, without a shared global view of the RealTime (RT) topics among all leaders, DIV states for RT topics are scattered among all past leaders and without a way to pass RT DIV from leader to its successors, DIV results can be inaccurate especially during the leadership transitions.

At a high level, we proposes to replicate the leader DIV states into the local VT periodically or on-demand. Followers consume these leader DIV states and sync itself up to the leader state along the way.

This PR:
As the first part of the implementation for global DIV improvement, this PR contains the following:

1. introduce a new flag for the global rt div feature.
2. introduce a new realtimeTopicProducerStates in PartitionState in the local rocksdb checkpoint.

avro diff: 
    {
      "name": "realtimeTopicProducerStates",
      "doc": "A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.",
      "type": {
        "type": "map",
        "values": {
          "type": "map",
          "values": "com.linkedin.venice.kafka.protocol.state.ProducerPartitionState"
        },
        "java-key-class": "java.lang.String",
        "avro.java.string": "String"
      },
      "default": {}
   }

4. divide today's complete DIV into two groups: VT and RT DIV, and RT DIV is further divided by its source fabric, so that each group can be updated separately based on which topic the Kafka records come from. (without feature enabled, only VT DIV is updated).
5. Adding test to verify the read/write of an OffsetRecord with new the new added field.

Future changes:
After this PR, as the next step, we will start to implement the following:

1. In the leader replica, create a snapshot of the RT DIV and replicate it to the local VT by using a chunking supported heartbeat messages (DIV heartbeat).
2. In follower replicas, consume the leader's RT DIV snapshot and sync itself up to the leader state along the way. (this actually needs to be checked in first to handle potential deployment rollbacks).
3. The lifecycle of the in memory RT DIV states will be managed by the leader and is shown in the following figure:

How was this PR tested?

CI (with or without feature enabled)

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.

[gaojieliu]

If the SIT is not ingesting the leader replica, will SIT load the RT DIV or not?
Also when the replica transitions from leader replicas to follower replica, will SIT offload RT DIV from the memory and reload it when necessary?

Maybe this is not part of the scope of this PR, and some clarification will be hepful.

[lluwm]

If the SIT is not ingesting the leader replica, will SIT load the RT DIV or not? Also when the replica transitions from leader replicas to follower replica, will SIT offload RT DIV from the memory and reload it when necessary?

Maybe this is not part of the scope of this PR, and some clarification will be hepful.

Will update the description part to address this comment.

[gaojieliu]

Thanks!