Balancing Network Diversity and Performance #6413
Comments
github-actions
bot
added
the
I10-unconfirmed
|
Afaik, reputation should only matter for collators, not validators, right? It might or might not exist for validators too, but in practice validators must talk to all other validators directly in availability, via fixed topologies in approvals, etc. I doubt this impact rewards right now, but.. Impact upon collators matters too. I'm afraid some collator tech like elastic scaling will always be massively unfair, but we could be more careful for parachains who do not require elastic scaling. It could massively impact validator rewards once we make vlaidator rewards make sense using #1811 We should explore information about latencies to different ISPs around the world. In theory, these all say under 300 ms but likely the results get worse once you look at say Thailand. https://wondernetwork.com/pings And maybe other bandwidth measures besides latency, ala https://www.speedtest.net/global-index Anyways we picked the approval (ELVES) parameters so that 500 ms round trip works fine. We could tweak these future, like a 1 second tranche time should reduce no-shows. |
|
This sounds like an issue that should be fixed. Can you elaborate more on what data are you basing this on please? Also if you have any logs from affected validators, that should be helpful. |
On staking.polkadot.cloud, you have an indicator of “validator performance” and similarly, there is apps.turboflakes.io to get more reporting on validators. This is what I alluded to when I referred to reputation for validators; it's not the only aspect that matters for reputation, but it matters a lot nonetheless.
Indeed, it generally works fine. My observation on validators not located right at the core of the Internet infrastructure is that the further away they get, the fewer peers they tend to be connected to, and the more votes they tend to miss. The missed votes numbers are not very high at this point, but I still wanted to report my observations because I am concerned that as the load on the network increases, this effect may increase as well. That would substantially disadvantage those who have invested in the decentralization of the network.
I may be mostly alone with this issue, and somehow it's my infrastructure that has an issue somewhere. I cannot see how at this point, which is why I have opened this ticket. To be more specific, I always have excellent performances & rewards in the Netherlands, but for example, Bulgarian or Polish validator performances would be slightly less. As a matter of fact, currently, I see 9 peers on the Kusama validator in Bulgaria, 18 in Poland, and 35 in the Netherlands. |
|
Your not alone, someone in bangkok mentioned have issues there. Jonas pointed out this map: https://www.certhum.com/polkadot-validator-map It's not that diverse in terms of validator locations really. We do want more validators in non-aligned or brics+ nations. I think dv/1kv has some affirmative action that selects such validators more. We do need limits though so really poor connections would miss rewards if they run validators, so no starlink or other craziness, but we need data on where to draw this line. After #1811 we should probably increase maximum commission in dv/1kv for validators in nations not well represented, so then maximum rewards would mean finding a fast ISP in a different nation. We should discuss the nato, west, brics+, or non-aligned classifications too. |
|
Is there a specific Github issue to follow on this matter? It's something I care a lot about and would gladly attempt to help where possible. The incentive for 1kv/dv is great, but I am particularly interested in ensuring that self-reliant validators also have the best possible incentives to run nodes as decentralized as possible. |
|
1kv/dv is an internal w3f thing. They do whatever they think benefits the network. I've suggested but we've never seriously considered location based affirmative action in actual rewards. It'd be done using median computaitons similar to #1811 which always scares people. Not impossible, but not politically the easest thing. Anyways like @eskimor said we're happy to have stats from validators who have latency problems, but ideally we'd love some information on selection of ISP there because a city might've good internet but a particularly inexpensive ISP might be bad or just non-commercial. As a related example, we'd trouble with too many validators being on Hetzner, who are cheap for Germany, but hate crpyto-currencies and cut off nodes without warning. |
|
Okay. I will provide some details while ensuring they are not getting into the specifics of the exact location of my validator(s). Hopefully this will be useful, if not you can just let me know and tell me what would be useful instead. We can also exchange privately, in which case I could share a bit more details than publicly. This is from a validator in Bulgaria, connected through ReTN, Cogent, HE, SOX. Does this look like a normal number of peers connected to the node? Likewise, Turboflakes says that this validator has about 60% "less than the average of Backing Points collected by all Para-Authorities of the last 192 sessions"; normal? Finally, I have done a speedtest and even though my server interface is 1Gbps, I seem to be getting substantially less. Running an iperf3 test seem to show better performance even if the testing server is located further away (France): |
|
What would be interesting is ping/round trip times. E.g. between the well performing and the not-so-well performing node. |
|
Sure thing. From a large DC in the Netherlands: 
From a large DC in Germany: 
|
FYI. Nothing has changed on my end, really no idea why I am getting such a low peer number. Also, on app.turboflakes, that validators is remaining A+. I can't say that I understand what is going on, but at least the validation aspect seems to be operating normally. |
|
Hmm. The data looks rather similar. How big is the difference in rewards? |
|
I'm starting one in Turkey and I'll try to get you some more useful details. For one thing so far, to complete the warp sync at current rate is about to take a week; very few peers. In AMS, UK, DE, US, etc... it would take a day at most. |
The number of peers reported is for sync peers, validation protocol has it's own peer set. So we should find out why the syncing is not able to establish enough connections. In any case, this is weird, I would expect poor networking to affect both syncing and validation at the same time. Is the peer count low from the very beginning after the node startup, or is it high first and than drops to 1-2 peers? Can you collect the logs with |
|
The Tor project puts Turkley in the top-10 countries by possible censorship events. Internet censorship often disrupts unrelated traffic using the same ISP, so polkadot infrastructure could be censored inadvertently, especially if the ISP criteria wind up similar for node operators of both polkadot and tor. It's good to know if we're hitting censorship, even if not targetted at us. Also, if someone want to ask questions about censorship, there are many people who study it not, both at Tor and elsewher, so one could ask questions at https://tor.stackexchange.com/ maybe. Also, the Tor matrix space has rooms for the "global south" and "relay operators" which maybe good places. |
|
#1 The server is staying behind on sync after about a week. It has a 1Gbps connection and is located in a top tier datacenter.
#2 @burdges
Sure I'll do so right away. But I would rather not share the output here as I don't know how to anonymize it. I'm not going to keep this server longer than January if it cannot even sync up to the tip. That is, unless, it can help figure out some useful things to the network. At this point I suggest taking this private with a Parity team member. If you agree please email me from a parity.io email address to 68jppbrc@addymail.com and from there we can either keep it over email or switch to Signal for faster back and forth. |
|
Ahh ooni-cli is a nice idea for this! :) |
There are no private information in these logs. Yeah, it will contain ip addresses, but I can also query those via the public network. |
|
Can you link those IPs with my name via the public network? My offer to work on this together stands, but I'm not sharing the private bits here. |
If you are running a validator or having any node name, that can be associated with you, it also works in a public network. I just wanted to say that there is not that much data in it, you need to afraid of. Especially if it is only about your ip address, which you could search&replace. But @dmitry-markin will reach out to you via mail and then you can share your logs there :) |
I have observed that nodes running at the core of the European and US internet infrastructure, are connected to a substantially higher number of peers and are getting better network performance. This comes with advantages such as a better reputation and, just as importantly, better rewards.
The wiki says, “A diverse network of nodes in varying different regions helps strengthen decentralized networks”, which I agree with. However, my experience has been that doing so, implies lower performance as well as lower rewards for a validator.
The Wiki for Validators should be more clear in terms of what regions would be beneficial for the network without expecting a negative impact on the validators performance and rewards. Likewise, it may also be worth documenting what regions would be, at this point, too far off and most likely incur performance and reward penalties.
The text was updated successfully, but these errors were encountered: