Metrics and Telemetry

[mzabaluev]

Technical details of implementation for https://github.com/movementlabsxyz/sre/discussions/63 will be discussed here.

Enablers

There are a few available solutions to expose metrics and timing information.

Timing information with the tracing framework

Already implemented in movement-timing.

Features:

• Fundamentally a logging system, produces a series of events that may have timing information.
• Can plug into integrations available for the tracing framework.
• Provides both busy time and total elapsed time for instrumented async futures.

Prometheus metrics

Features:

• Pull-based metrics API.
• Already in use in aptos-core.

OpenTelemetry metrics

Features:

• Pull (lossy) and push (lossless up to a limit) model possible with OLTP.
• Standard protocol with a wider choice of consumer tools.

Node Instrumentation

To support SRE, the node should provide metrics for:

1. Transaction rate per second

• Possible measurement points: transaction pipe/mempool, DA blob submission, execution, settlement (requires decoding of blocks, consider cost?).

2. Transaction latency

• An in-memory table of transactions in flight with timestamps.
• The metrics are aggregated across recent transactions in a time window, measuring latency at these possible points: DA blob submission, execution, settlement (requires decoding of blocks, consider cost?).

3. Number of transactions in flight

4. Gas per second

• Technical details of how to collect this need to be discussed. Execution as the tapping point?

Ad-hoc instrumentation for development and troubleshooting

The tracing framework could be used for more fine-grained instrumentation to obtain measurements of specific flows.

Infrastructure metrics

These common metrics should be collected by the deployment infrastructure.

• Network traffic per second
• Memory usage
• CPU utilization

Measurement challenges

Listing measurement tasks that may require some engineering effort.

• Transaction failure rate
  • Possible by tracking across node logs with tracing tags.
  • The built-in latency tracker can have a timeout, considering any transactions not confirmed within a deadline as failed. This is needed anyway to GC the transaction latency table.

[apenzk]

more metrics to consider

• b(yte)ps, network receive (Mbps)
• gas per second
• memory usage
• core thread utilization

[mzabaluev]

Thank you, I worked these into the head writeup.

[l-monninger]

So, @mzabaluev I think before we start talking about the tools and implementation too much, we need to cover what our "resources" are and where they are in code w.r.t USE/RED approach to monitoring: https://github.com/movementlabsxyz/sre/tree/main/monitoring

To me this looks like a discussion here where we identify the resource in the abstract, correlate it with code on main, and describe how USE/RED are applied.

Something like this:

RESOURCE: Memseq Builder
METRICS: TPS, Transaction Latency
CODE:
- At #L256 we have the ingress...
- At #L512 we have loading shedding...
- At #L1024 we have egress...
REFACTOR: 
- We might be able to unify all of the ingress and egress points by...
USE:
- USE is not particularly applicable here.
RED:
- We can trivially...
FOUR SIGNALS:
- Latency is amortized...

[l-monninger]

USE: not applicable

Explain why.

[l-monninger]

I get that this is high-level. But, yeah, we would need more detail. For example...

Rate: measure transactions per second.

We should be saying something like...

The rate we wish to compute is the transactions per second (TPS). To accomplish this, we will emit a message message TransactionIngress { } whenever a transaction crosses any of the ingress thresholds indicated above. We will, correspondingly, emit a message message TransactionEgress { }. The BLANK and BLANK fields of the ingress and egress methods respectively will be used to allow a sliding window analysis or similar to compute transactions per second.

[mzabaluev]

I actually thought of computing the rate metrics in the code and expose them via OpenTelemetry (or Prometheus, whichever we decide to use). But yes, I should expand on this. To limit the amount of work though, I think we should start from the top down and decide which metrics are we interested in, and then pick the measurement points at which to collect them.

[l-monninger]

Unless we actually want to affect dataflow with these metrics, I think doing this would be unnecessary and potentially risk performance detriments. It would also almost certainly add additional complexity to the code.

[mzabaluev]

Emitting an OLTP event for every transaction could be spammy, too.
The monitoring agent needs to be able to aggregate if we go with this approach.

[mzabaluev]

I have started discussions on specific metrics:

• Metrics: Transactions Per Second #422
• Metrics: Transaction Latency #423
• Metrics: Transaction Load #424
• Metrics: Gas per Second #425
• Metrics: Infrastructure #427
• Metric: Transaction Failure Rate #428

Much remain unsettled to progress with specific implementation, though.

[mzabaluev]

Should there be a metric for the rate of successfully settled (or, inversely, rejected) blocks?

[l-monninger]

Not for now.

[mzabaluev]

We can try using tokio-metrics for performance research.
It requires the tokio_unstable cfg flag, so it's not advisable for production or regular developer builds.
But we can put this functionality under the same flag; additionally, aptos-core currently requires this flag already.

[l-monninger]

We've already used this when looking for concurrency bugs I believe.

[mzabaluev]

Should we try to start with implementing OpenTelemetry push in the service binaries for some of the metrics, and see if it works for us? @radupopa369, @l-monninger, are there tools we can readily deploy to analyze e.g. TPS from an OpenTelemetry event series?