[Request]: inconsistencies in API Key description for APM Server usage

[endorama]

What documentation page is affected

• https://www.elastic.co/guide/en/observability/current/apm-configure-agent-config.html
• https://www.elastic.co/guide/en/observability/current/apm-elasticsearch-output.html
• https://www.elastic.co/guide/en/observability/current/apm-elasticsearch-output.html#apm-securing-communication-elasticsearch
• https://www.elastic.co/guide/en/observability/current/apm-elasticsearch-output.html#_api_key_2

What change would you like to see?

Either is clearly mentioned that the API key format required is the Beats/Logstash one or having an earlier link to the api_key section in Configure the Elasticsearch output.

An additional section about getting 401/API Key Invalid errors to Common problems in Configure APM agent configuration would also help troubleshoot this faster.

Additional info

APM Server supports using API key based authentication for its operations. The format required is not the newer base64 encoded, presented by ES in 8.15 interface as "Format used to make requests to Elasticsearch REST API."

It instead uses the Beats/Logstash format id:key. Our documentation is confusing about it and requires multiple hops to find confirmation of this requirement. Meanwhile what a customer using the default API Key value faces are "API key invalid" errors from APM server.

I entered the documentation at https://www.elastic.co/guide/en/observability/current/apm-configure-agent-config.html as I was searching how to configure APM server authentication for agent configurations. The section displays the Beats format, but is quite easy to miss:

I dug deeper, looking at the relevant config option, which does not mention the format but links to another page:

Opening that page presents a section where auth methods are described, Beats format is shown but not mentioned , with a link to another section:

This section increase confusion, as the API format is not the Beats one:

When finally reaching the "Grant access using API keys" page, which is not in the same hierarchy as the configuration section:

we get the information about the API key format:

All of this could have been avoided if I looked at the api_key section in Configure the Elasticsearch output, I think I missed due to links pointing me in other directions and not having APM Server displayed as a possible value in the Elasticsearch API key dropdown.