tune the collector for embedded devices (cpu consumption)

[cforce]

We have already implemented a memory limiter to ensure that we stay within a specific usage rate. However, when it comes to CPU usage, the factors that influence it include the choice of extensions, processors, and receivers. This is particularly crucial for the OTel collector base, which involves receiving traces and metrics via OTLP gRPC, handling host metrics, and exporting data via OTLP gRPC.

In the context of CPU consumption on embedded devices, it becomes essential to measure and optimize the relevant parameters. Understanding which factors influence CPU consumption, how to reduce it, and being aware of potential trade-offs is of utmost importance. Unfortunately, I haven't come across comprehensive guidance on this aspect yet.

In the case of single-core utilization, we need to ensure that we do not exceed a certain fraction of milli-cores. However, the challenge lies in how we can effectively achieve and maintain this without clear guidelines or tools in place.

related

• Degenerate collector performance when exporter has problems #4981
• Ensure reliable data delivery in erroneous situations #7460

[cforce]

Analysis on CPU spike for OTEL collector

Modified the config for OTEL collector and ran the cputop script "cputop" for 5 minutes (approx) and generated the report.
Host metrics scrapers list used in OTEL collector - cpu, memory, network, disk, load, paging, processes, process

Below table contains the config used to test OTEL collector CPU usage spike.

Run Number	Metrics	Traces enabled	Top value (approx) seen in graph for Load
1	Podman + all host metrics	Yes	35 to 39
2	Podman + prometheus + host metrics except network	No	40 to 45
3	Podman + prometheus + No host metrics	No	1.7
4	Podman + prometheus + host metrics except network, disk, load	No	35.8
5	Podman + prometheus + host metrics except network, disk, load, paging, processes, process	No	1.7
6	Podman + prometheus + host metrics except processes, process	No	8.4
7	Podman + prometheus + host metrics except processes, process, paging	No	8.3

From the above data we get to know that

1. Process, Processes -> create impact on CPU spike
2. cpu, memory, paging -> does not matter for CPU spike

[cforce]

Hostmetrics process(es) metrics are retrieved using the go lib gopstutil.
There are several known issues with cpu utilization peaks mainly connected to getting the "boot time each time" per process.
See

• BootTimeWithContext is being called for every pid causing high CPU consumption shirou/gopsutil#1283
• Processes() is extremely CPU intensive shirou/gopsutil#842
• get boot time each time cost too much cpu shirou/gopsutil#1070

A bit offtopic but relevant for hostmetrics "proc/net" network connection metrics is this one. The system.network.connections metric being disabled and not collecting the information from the host does waste less CPU cycles, especially through the flame graph query, it is found that gopsutil is used. This will scan files under the proc directory. The more links, the more CPU. The more resources are occupied, see if it can be adjusted.  However this needs otel collector >= v0.85.0 at least, as before there was a bug that disabling did still execucted the scraping. See receiver/hostmetricsreceiver/internal/scraper/networkscraper/network_scraper.go (line 160 , extr a"if !s.config.Metrics.SystemNetworkConnections.Enabled {").  If you try around with load/perf on a docker container env, be aware that you need to mount the host filesystem to simulate a proper /proc access .

• Use netlink API to get net connections for better performance shirou/gopsutil#695

[cforce]

Created open-telemetry/opentelemetry-collector-contrib#28849

[cforce]

Interesting projects using in context of eBPF collecting host metrics
https://github.com/pixie-io/pixie
https://github.com/iovisor/bpftrace
https://docs.px.dev/tutorials/custom-data/distributed-bpftrace-deployment/
https://docs.px.dev/about-pixie/pixie-ebpf/
https://docs.px.dev/about-pixie/data-sources/#data-sources