Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] add read rate limter for compaction #8

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

HQebupt
Copy link
Owner

@HQebupt HQebupt commented May 31, 2023

Motivation

The purpose of this feature is to provide more flexibility and control over the compaction process in the Bookkeeper server by allowing users to configure the read rate of compaction according to their specific requirements.
It adds support for configuring the read rate of compaction in bytes per second.

In our production environment of Bookkeeper, we have set compactionRateByBytes=30M. However, we have observed that it only limits the write speed to disk and does not limit the read speed from disk. In our environment, the continuous saturation of disk IO for reads in compaction is causing issues with data consumption from the disk.

// Set the rate at which compaction will readd entries. The unit is bytes added per second.
 compactionRateByBytes=31457280

image

image

Changes

  • In the GarbageCollectorThread class, a new field compactionReadByteRateLimiter of type RateLimiter is added to control the read rate of compaction.
  • In the ServerConfiguration class, a new configuration property COMPACTION_READ_RATE_BY_BYTES is defined to specify the read rate of compaction in bytes per second.

Master Issue: #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant