Read Logs from application to filebeat and then export it into opentelemetry collector.

[PiyushSharma2812]

Configuration
filebeat version v8.11.2
opentelemetry collector version v0.89.0
Operationg system Windows 11 Pro

This is my filebeat configuration file
filebeat.inputs:

• type: log
  enabled: true
  paths:
  • C:/apache-tomcat-9.0.56/logs/localhost_access_log.*.txt

output.otlp:
hosts: ["localhost:11800"]

processors:

• add_host_metadata:
  when.not.contains.tags: forwarded
• add_cloud_metadata: ~
• add_docker_metadata: ~
• add_kubernetes_metadata: ~

Error getting when try to start the filebeat application
PS C:\Program Files\Filebeat> .\filebeat.exe -c "C:\Program Files\Filebeat\filebeat.yml" -e
{"log.level":"info","@timestamp":"2023-12-13T10:17:28.784+0530","log.origin":{"file.name":"instance/beat.go","file.line":806},"message":"Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-12-13T10:17:28.785+0530","log.origin":{"file.name":"instance/beat.go","file.line":814},"message":"Beat ID: e4c6cee2-71ba-411e-b6c0-f126ba44b262","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-12-13T10:17:28.987+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1332},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\Program Files\Filebeat","data":"C:\Program Files\Filebeat\data","home":"C:\Program Files\Filebeat","logs":"C:\Program Files\Filebeat\logs"},"type":"filebeat","uuid":"e4c6cee2-71ba-411e-b6c0-f126ba44b262"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-12-13T10:17:28.987+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1341},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"ce367ff5456dd8a1a93d6bae8fd600bb04816718","libbeat":"8.11.2","time":"2023-12-04T20:23:24.000Z","version":"8.11.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-12-13T10:17:28.989+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1344},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":8,"version":"go1.20.11"},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.988+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting huawei metadata: Get "http://169.254.169.254/openstack/latest/meta_data.json\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.991+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.992+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.992+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting openstack metadata: Get "https://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:443: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.992+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting hetzner metadata: Get "http://169.254.169.254/hetzner/v1/metadata/instance-id\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.993+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-13T10:17:28.994+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.","service.name":"filebeat","ecs.version":"1.6.0"}

This is my otelcol configuration file
receivers:
otlp:
protocols:
grpc:

exporters:
logging:
loglevel: debug

service:
pipelines:
logs:
receivers: [otlp]
exporters: [logging]

Getting this when try to run the otelcol application using opentelemetry collection configuration file.
C:>C:\otelcol.exe --config C:\opentelemetry-collector\otelcol\otelcol_configCheck.yaml
2023-12-12T16:41:17.659+0530 info service@v0.88.0/telemetry.go:84 Setting up own telemetry...
2023-12-12T16:41:17.660+0530 info service@v0.88.0/telemetry.go:201 Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
2023-12-12T16:41:17.662+0530 info exporter@v0.88.0/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-12-12T16:41:17.663+0530 warn common/factory.go:68 'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior. {"kind": "exporter", "data_type": "logs", "name": "logging", "loglevel": "debug", "equivalent verbosity level": "Detailed"}
2023-12-12T16:41:17.664+0530 info service@v0.88.0/service.go:143 Starting otelcol... {"Version": "0.88.0", "NumCPU": 8}
2023-12-12T16:41:17.664+0530 info extensions/extensions.go:33 Starting extensions...
2023-12-12T16:41:17.664+0530 warn internal@v0.88.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "logs", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-12-12T16:41:17.665+0530 info otlpreceiver@v0.88.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4317"}
2023-12-12T16:41:17.666+0530 info service@v0.88.0/service.go:169 Everything is ready. Begin running and processing data.

How to resolve this error so that i can read logs from application using filebeat and then push into opentelemetry collector

[florius88]

I have the same problem :(