Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add queue based scheduler #1869

Closed

Conversation

erezrokah
Copy link
Member

@erezrokah erezrokah commented Aug 14, 2024

Summary

Mostly an experiment to deal with https://github.com/cloudquery/cloudquery-issues/issues/2227 as I couldn't think of a nice way to make singleNestedTableMaxConcurrency dynamic without making the code super complex.

This PR adds a scheduler that uses a worker pool pattern on top of a priority queue. This should ensure that as long as there's work to be done, all Go routines will be occupied. Also the concurrency setting is not only for the top level tables, as it's the number of workers so it's a fixed limit and simpler.
The more table client pairs in the queue the less priority they'll have, this should prevent a specific table from occupying all the workers.

Opening as draft since:

  1. I'm still testing this to see the impact
  2. There's still a lot of code duplication with current code and refactoring needed to avoid it

Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@github-actions github-actions bot added feat and removed feat labels Aug 14, 2024
@erezrokah
Copy link
Member Author

Closing in favor of #1914

@erezrokah erezrokah closed this Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant