Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor file backend to support multiple logs #48

Open
3 of 7 tasks
carns opened this issue Mar 17, 2021 · 0 comments
Open
3 of 7 tasks

refactor file backend to support multiple logs #48

carns opened this issue Mar 17, 2021 · 0 comments
Assignees

Comments

@carns
Copy link
Member

carns commented Mar 17, 2021

Testing has shown that the file backend will be significantly faster on small concurrent access sizes if there are multiple log files rather than just one, see #40 proof of concept. There are several steps to this (this will be an evolution/superset of the current backend rather than a new file backend).

  • make target specification for file backend refer to a subdirectory, rather than a singular file
  • modify log position tracking strategy: rather than extending log and sync'ing to record position on each write, track the next available log offset in memory and write it to the header on clean shutdown. If offset is not available on startup, pessimistically assume that next position is the end of the log.
  • add integer to region id to specify log index
  • add ability to initialize multiple log files
  • json parameter for size increments to expand log files by. This will be used by server for initial allocation when the target is attached as well.
  • add ability to extend log files once they reach a capacity threshold using an asynchronous fallocate()
  • experiment with signal handler as example for how to shutdown cleanly on ctrl-c (not just shutdown rpc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant