Investigate "tasks have lost their waker" and fix if needed

[conradoplg]

Motivation

tokio-console, for which we added support in #4519, is reporting a bunch os "tasks have lost their waker". We should investigate.

Most of them seem to be from the handshake code:

zebra/zebra-network/src/peer_set/initialize.rs

Lines 298 to 299 in ca0520b

	tokio::spawn(async move {
	let task = outbound_connector_future.await;

zebra/zebra-network/src/peer_set/initialize.rs

Lines 757 to 758 in 766dd93

	let hs_join = tokio::spawn(dial(
	candidate,

zebra/zebra-network/src/peer/handshake.rs

Lines 1016 to 1020 in 766dd93

	// Spawn a new task to drive this handshake.
	tokio::spawn(fut.instrument(negotiator_span))
	.map(|x: Result<Result<Client, HandshakeError>, JoinError>| Ok(x??))
	.boxed()
	}

Specifications

Designs

• Investigate the cause of the issue
• If it's an issue, fix it if possible. If it's too complex, create another issue for it.
• If it's a false positive, consider reporting it to tokio-console

Related Work

[teor2345]

I've had a look at all the lost wakers, and they seem pretty harmless.

If we're just dropping wakers for handshakes we're not using any more, that's probably a task cleanup bug. And we make new handshakes as needed anyway.

[teor2345]

The warning might also be incorrect or confusing, I'm still seeing the poll number for those tasks going up, even after the lost waker warning.

But handshakes should only last for 4 seconds, so there's definitely a bug here.

[teor2345]

This appears to be a known bug in tokio-console, related to tasks spawning other tasks:
tokio-rs/console#345