Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize farm and cache identification #2900

Open
nazar-pc opened this issue Jul 9, 2024 · 5 comments · May be fixed by #3032
Open

Optimize farm and cache identification #2900

nazar-pc opened this issue Jul 9, 2024 · 5 comments · May be fixed by #3032
Assignees
@tediou5
Labels
farmer Farming library/app good first issue Good for newcomers improvement it is already working, but can be better performance Related to performance measurement or improvement
Milestone
Protocol UX Impro...

Comments

@nazar-pc
Copy link
Member

nazar-pc commented Jul 9, 2024

Currently farming cluster operates on farms and caches without any knowledge of "farmer".

This works and is flexible, but results in a bit too many notifications from identification messages, which #2899 improve a little bit.

It might be beneficial to also associate farms with farmers (and similarly for caches), just some ephemeral identifier, such that we can reduce identification broadcast from one per farm to one per farmer most of the time.

Farm identification as such should likely be replaced with a stream request such that farmer can send a stream of responses back to controller at a sustainable rate rather than bursting potentially thousands of messages all at once that might be dropped by NATS server due to buffer limits.
@nazar-pc nazar-pc added good first issue Good for newcomers farmer Farming library/app improvement it is already working, but can be better performance Related to performance measurement or improvement labels Jul 9, 2024
@nazar-pc nazar-pc added this to the Protocol UX Improvements milestone Jul 9, 2024
@tediou5
Copy link
Contributor

tediou5 commented Jul 14, 2024

let me work on this. Its looks easy

@tediou5 tediou5 mentioned this issue Jul 24, 2024
Closed
1 task
@tediou5
Copy link
Contributor

tediou5 commented Sep 7, 2024

@nazar-pc I have an experimental implementation repo, but here i have a problem.

In the #2945 (comment), we mentioned support quick farmer restarts. But while coding, I found out that the farm's fingerprints include randomly generated ids each time. So how do we implement this feature?
(I've already finished the rest of farmer, so you can take a look at it in advance

@nazar-pc
Copy link
Member Author

But while coding, I found out that the farm's fingerprints include randomly generated ids each time. So how do we implement this feature?

Check how we generate fingerprints right now. Essentially fingerprint of the farmer would be a generalization of the fingerprints of individual farms. The idea is to be able to tell whether it is the same farmer with the same set of farms in the same state or if some changes have happened to it during restart.

@tediou5
Copy link
Contributor

tediou5 commented Sep 11, 2024

@nazar-pc What do you think about me generating the fingerprint through the hash of plotted_sectors.

My current implementation is generate fingerprints from a randomly farmer ID for each startup and total_sectors_count for each farm , The concrete realisation is here.

Or I can support quick restarts in another pr?

@nazar-pc
Copy link
Member Author

Have you see what is currently used as fingerprint? You can literally do the same thing and hash it together.

@tediou5 tediou5 linked a pull request Sep 18, 2024 that will close this issue
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tediou5 tediou5

Labels
farmer Farming library/app good first issue Good for newcomers improvement it is already working, but can be better performance Related to performance measurement or improvement
Projects
Subspace core (node, farmer, etc.)
Status: In Progress
Milestone
Protocol UX Improvements
Development

Successfully merging a pull request may close this issue.

opt: optimize cluster identification tediou5/subspace
2 participants
@nazar-pc @tediou5