Fix throttling replica list after a broker replacement

[kravisha91]

During a broker replacement it was noticed that a broker that was previously involved in reassignment when got replaced with a new node (it is common for brokers to fail and get a replacement node) did not throttle all the topic replicas the broker was hosting and that led to increase network usage and eventually more load on the broker affecting clients connected to it causing latency issues. The proposed fix is to refresh the replica throttled list when there is a broker override irrespective of the broker involvement in other reassignments. More details can be found in the notebook - https://ddstaging.datadoghq.com/notebook/6601106/investigating-missing-throttles-on-topics-after-broker-replacement#overview

[mjd95]

This looks like it updates leader.replication.throttled.rate and follower.replication.throttled.rate only. What happens to the value of replication.throttled.replicas?

The desired state is that, for each topic, replication.throttled.replicas is set to be the union of whatever it should be for the reassignment, plus every replica on the broker with the override.

I'm guessing this is what should happen on L222 below, but I would expect that if the replaced broker was part of a reassignment then tm.skipOverrideTopicUpdates is true (so we skip), because of this code.

[kravisha91]

The only change I did here was to remove the if condition !override.ReassignmentParticipant since we also want to include brokers that were previously part of reassignment. You are correct L222 is where the replication.throttled.replicas is set based on tm.skipOverrideTopicUpdates but the control never gets here even if skipOverrideTopicUpdates is false if a broker is a reassignment participant because there is a return nil statement on line L210 which checks for any brokers in toAssign map. Hence I had to remove the if condition and allow it happen for all broker overrides.