[BUG] Can not send logs using Elastic.Serilog.Sinks if running Elasticsearch behind an API gateway (or similar) on a non-root URL path

[gjermystreeva]

ECS integration/library project(s) (Elastic.Serilog.Sinks): (v8.11.1)

.NET framework / OS: net6.0 / Windows 10

Description of the problem, including expected versus actual behaviour:

Can not send logs using Elastic.Serilog.Sinks if running Elasticsearch behind an API gateway (or similar) on a non-root URL path.

So if Elasticsearch is running such that it is available on e.g. https://example.com/elastic/ then using the Elastic.Serilog.Sinks I would expect log lines to be published to correct elastic URL, but the root path part of the host URL is ignored when actually publishing logs documents to the DataStream.

Note that the bootstrapping phase (when using BootstrapMethod.Failure) does manage to install component and index templates to the elastic search instance correctly and is honouring the non-root part of the nodes' URL.

Steps to reproduce:
1.
Follow the setup guides but with a non-root URL e.g.

.WriteTo.Elasticsearch(new [] { new Uri("https://example.com/elastic" )}, opts =>
{
	opts.DataStream = new DataStreamName("logs", "example", "default");
	opts.BootstrapMethod = BootstrapMethod.Failure;
	opts.ConfigureChannel = channelOpts =>
	{
		channelOpts.BufferOptions = new BufferOptions 
		{ 
			ConcurrentConsumers = 10 
		};
	};
})

Run program and write to injected ILogger

Observe in Elasticsearch instance the index templates have been set up. (Actually I think template name for above would be logs-example-8.11.0 with an index-pattern logs-example-*)

Observe that no data stream logs-example-default or similar has been setup and no (hidden) indexes created either.

If add code similar to below to catch exceptions

 opts.ConfigureChannel = cfg =>
 {
   cfg.ExportExceptionCallback = ex =>
   {
     Serilog.Debugging.SelfLog.WriteLine("ExportExceptionCallback Exception: {0}", ex);
   };
 };

...we can see that the POST is being made but without the expected /elastic/ at the beginning of the path.

"Unsuccessful (403) low level call on POST: /logs-example-default/_bulk...

Note here I would expect the path to have been /elastic/logs-example-default/_bulk...

---

I'm raising in this repo as the Elastic.Serilog.Sinks is what I'm trying to use. However it is possible the issue is actually within Elastic.Ingest.Elasticsearch. In that repo looking at e.g. src\Elastic.Ingest.Elasticsearch\DataStreams\DataStreamChannel.cs and src\Elastic.Ingest.Elasticsearch\ElasticsearchChannelBase.cs the way BulkUrl is set to "/_bulk" rather than "_bulk" could explain why some parts of the sink (setting up the index template) can talk to elastic instance fine, whereas actually publishing the logging fails?

[Mpdreamz]

Thanks for raising and diagnosing this @gjermystreeva 👍

Opened elastic/elastic-ingest-dotnet#73 to address this.