Add Raft State Management for Load Balancers #641
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implemented round-robin state to enhance leader election and task distribution. - Updated Raft state machine to incorporate round-robin logic. - Added tests to ensure correct round-robin behavior in various scenarios. This change improves the efficiency and fairness of task handling within the Raft cluster.
- Introduced a mutex (`proxyStateMutex`) to synchronize access to proxy state checks in `testProxy` function. - Modified the `testProxy` function to lock the mutex before checking the state of `AvailableConnections` and `busyConnections`. - Updated the test logic to ensure that one of the proxies is in the expected state, preventing race conditions where the second goroutine could access the connection state prematurely. - Removed the `proxy` parameter from `testProxy` function calls as it is no longer needed. This change addresses a race condition that could cause the second goroutine to access connection states before they are properly synchronized, ensuring reliable test results.
Add support for storing weighted round-robin load balancer state in Raft FSM to ensure consistency across cluster nodes. Changes include: - Add WeightedProxy and WeightedRRPayload structs for state management - Store proxy weights in Raft FSM using weightedRRStates map - Update WeightedRoundRobin to use Raft for weight tracking - Add new CommandUpdateWeightedRR command type - Remove local weight tracking in favor of distributed state This change ensures that proxy weights remain consistent across cluster nodes during failover and leader changes.
Improve performance of WeightedRoundRobin.NextProxy by reducing Raft operations: - Replace multiple individual Raft updates with a single batch operation - Introduce new CommandUpdateWeightedRRBatch command type - Collect all proxy weight updates in memory before applying - Reduce number of Raft.Apply calls from N+1 to 1 (where N is number of proxies) This change significantly reduces the number of Raft consensus operations needed for weight updates in the weighted round-robin load balancer.
- Add JSON marshaling error handling in WeightedRoundRobin - Simplify proxy state validation logic in server tests - Clean up test formatting The main changes improve error propagation and code clarity by: - Properly handling JSON marshaling errors in NextProxy method - Refactoring conditional logic in server tests to be more readable - Removing unnecessary empty lines in test files
Overview
Packages and Vulnerabilities (5 package changes and 0 vulnerability changes)
Changes for packages of type
|
| Package | Versionghcr.io/gatewayd-io/gatewayd:8fbdb8a |
Versiongatewaydio/gatewayd:latest |
|
|---|---|---|---|
| ➖ | ca-certificates | 20240705-r0 |
|
| ➖ | openssl | 3.3.2-r0 |
|
| ➖ | pax-utils | 1.3.7-r2 |
Changes for packages of type golang (2 changes)
| Package | Versionghcr.io/gatewayd-io/gatewayd:8fbdb8a |
Versiongatewaydio/gatewayd:latest |
|
|---|---|---|---|
| ♾️ | github.com/gatewayd-io/gatewayd | (devel) |
0.0.0-20241214123014-b6df86a6fe94 |
| ♾️ | stdlib | go1.23.4 |
1.23.4 |
Add support for persisting and restoring weighted round-robin load balancer states in the Raft FSM snapshots. This ensures the weighted round-robin configuration survives cluster restarts and leader changes. Changes: - Add weightedRRStates to FSMSnapshot struct - Update Snapshot() to copy weighted round-robin states - Extend Restore() and Persist() to handle weighted round-robin data
Add commented example of grpcAddress field in peer configuration, which specifies the gRPC endpoint for raft peer communication.
Add test cases covering: - Weighted round-robin operations (single and batch updates) - Round-robin index management - Invalid command handling - FSM snapshot restoration - Node shutdown scenarios
mostafa
reviewed
Dec 19, 2024
Replace usage of LeaderElectionTimeout with a new dedicated ApplyTimeout constant (2 seconds) for Raft command applications across different load balancing strategies (ConsistentHash, RoundRobin, WeightedRoundRobin). This change provides better separation of concerns by using a more appropriate timeout value for command applications rather than reusing the leader election timeout.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket(s)
N/A
Description
Add Raft State Management for Load Balancers
This PR enhances the distributed state management capabilities by integrating load balancer states into the Raft consensus system. Key changes include:
json.Unmarshaluses floats when unmarshalling numbers into interface valuesCommandAddRoundRobinNextCommandUpdateWeightedRRCommandUpdateWeightedRRBatchThese changes ensure consistent load balancer behavior across the cluster by maintaining state through Raft consensus.
Related PRs
Development Checklist
make gen-docscommand.Legal Checklist