Setup monitoring of ECS nodes

[hellais]

Currently we don't have observability into the container host of the ECS cluster. Moreover we are only able to scrape aggregate metrics from the services that are behind the balancer, which means we end up with the metrics "flapping".

Ideally we would have a way of scraping metrics for the container host, but also the per-servicer docker containers.

In summary we would like to collect two classes of metrics:

• Host container metrics (the ec2 nodes that run docker and we deploy docker containers to), using node_exporter
• Docker container application metrics, which are exposed using the instrumentator and we would like to scrape independently per each host container

[hellais]

Leaving here some links for things to look at:

• https://github.com/santosh-at-github/ecs-monitoring-with-prometheus-grafana
• https://github.com/google/cadvisor
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-ECS.html

[LDiazN]

I think this might be the way: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config

The problem I'm seeing right now is that the monitor server is not in AWS, so we have some issues with the connection between that server and the ec2 instances:

1. We have to set up IAM credentials for the server, as mentioned in the link above
2. The prometheus server needs a way to reach the ec2 instances (not the load balancer), but they're probably not open to internet traffic (and I don't think they should), what can we do about this?