AWS Distro for OpenTelemetry

Container Insights ECS Prometheus

GettingStarted / Container Insights / Container Insights for Prometheus Support

Overview

💡

You can also collect Prometheus metrics from Amazon ECS for Container Insights with the CloudWatch agent instead of the ADOT Collector. The CloudWatch User Guide documents this setup, including the pre-built dashboards.

Set up Prometheus metrics collection on Amazon ECS with the CloudWatch agent

The pre-built Container Insights dashboards for Prometheus workloads read metrics in the ECS/ContainerInsights/Prometheus namespace, which the collector doesn't write when it sends to the OTLP endpoint. You query the metrics with PromQL instead.

This page shows how to collect Prometheus metrics from tasks running on ECS with the AWS Distro for OpenTelemetry (ADOT) Collector and send them to the CloudWatch OTLP metrics endpoint. You can either run the collector as a sidecar or deploy the collector as its own ECS service for entire cluster. To attach task and service metadata to the metrics of every task in the cluster, run a single collector for the entire cluster. ECS tasks with Prometheus endpoints are discovered using extension ecsobserver. (Unlike EKS, there is no builtin discovery for ECS inside prometheus)

Steps

Follow these steps to run the ADOT Collector and Prometheus workload on ECS. You can skip some of them if you already have clusters or tasks running.

Create an ECS cluster based on tutorial or follow Appendix to Create ECS cluster with EC2 instances using ecs-cli
Create task IAM role and configure IAM policy. Discovery tasks requires extra ECS policies.
Create Configuration as an SSM parameter
Create ECS Task Definition
Deploy ADOT Collector as a replica service with 1 replica
Deploy sample workloads
View metrics in the CloudWatch console

Permission

For discovering ECS tasks, you need to config IAM policy. For scraping discovered targets, you need to configure network

IAM Policy

The task role needs cloudwatch:PutMetricData to send metrics to the CloudWatch OTLP metrics endpoint, and these extra permissions to discover ECS tasks with Prometheus metrics. The full document is included in the appendix IAM Policy Document. NOTE: The EC2 policy is required if you use ECS EC2 with bridge network mode. You can remove it if all your tasks are fargate or uses awsvpc network mode.

ec2:DescribeInstances
ecs:ListTasks
ecs:ListServices
ecs:DescribeContainerInstances
ecs:DescribeServices
ecs:DescribeTasks
ecs:DescribeTaskDefinition

Network

For the ADOT Collector to scrape metrics, connectivity between ECS tasks using private ip is required. Depends on how the cluster is created, you may need to configure security group to allow ingress within current VPC as prometheus pull metrics. The default VPC's security group normally allows ingress within VPC out of box, while some CLI tools such as ecs-cli creates new VPC with stricter rules.

When defining inbound rules, you can specify a strict port range if your task is using awsvpc as the network mode (ECS Fargate only use awsvpc). However, for ECS bridge network mode (only on EC2), the host port is random so you have to allow inbound on all TCP ports within the VPC.

# Enable all TCP inbound within same VPC
SG=sg-123456
aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 0-65535 --source-group ${SG}

Configuration

Configuring Container Insight Prometheus is similar to normal Prometheus pipeline.The overall flow of pipeline is:

Extension ecsobserver call AWS API and write discovered targets to a file in a folder that the collector can write to. The ADOT Collector image runs as the aoc user, so use its home folder, such as /home/aoc/ecs_sd_targets.yaml. A detailed explanation for ecsobserver config is in its source.
The Prometheus receiver includes the output file from ecsobserver using file_sd.
Prometheus relabeling copies the ECS metadata, such as the cluster, service, and task definition family, into metric labels.
The otlp_http exporter sends the metrics to the CloudWatch OTLP metrics endpoint, signing requests with the sigv4auth extension.

A full example can be found in examples/ecs/aws-cloudwatch/ecs-container-insights-prometheus.yaml. It contains discovery rules for all workloads. It sends the metrics with the awsemf exporter, so replace its exporters section and the exporters in its pipeline with the otlp_http exporter from the example below, and add the sigv4auth extension. NOTE: Remember to replace {{cluster_name}} in the example file.

Following is a small example for JMX and NGINX using service and docker label based discovery with inline comments.

extensions:
  sigv4auth: # signs requests to the CloudWatch OTLP metrics endpoint
    region: us-west-2
    service: monitoring
  ecs_observer: # extension type is ecs_observer
    cluster_name: 'my-ecs-cluster' # cluster name need to configured manually
    cluster_region: 'us-west-2' # region can be configured directly or use AWS_REGION env var
    result_file: '/home/aoc/ecs_sd_targets.yaml' # the directory for file must already exists
    refresh_interval: 60s # format is https://golang.org/pkg/time/#ParseDuration
    # custom name for 'job' so we can rename it back to 'job' using metricstransform processor
    job_label_name: prometheus_job
    # NGINX https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-nginx-ecs.html
    services:
      - name_pattern: '^.*nginx-service$' # NGINX
        metrics_ports:
          - 9113
        job_name: nginx-prometheus-exporter
    # JMX
    docker_labels:
      - port_label: 'ECS_PROMETHEUS_EXPORTER_PORT'

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "ecssd"
          file_sd_configs:
            - files:
                - '/home/aoc/ecs_sd_targets.yaml' # MUST match the file name in ecs_observer.result_file
          relabel_configs: # Relabel here because label with __ prefix will be dropped by receiver.
            - source_labels: [ __meta_ecs_cluster_name ] # ClusterName
              action: replace
              target_label: ClusterName
            - source_labels: [ __meta_ecs_service_name ] # ServiceName
              action: replace
              target_label: ServiceName
            - source_labels: [ __meta_ecs_task_definition_family ] # TaskDefinitionFamily
              action: replace
              target_label: TaskDefinitionFamily
            - source_labels: [ __meta_ecs_task_launch_type ] # LaunchType
              action: replace
              target_label: LaunchType
            - source_labels: [ __meta_ecs_container_name ] # container_name
              action: replace
              target_label: container_name
            - action: labelmap # Convert docker labels on container to metric labels
              regex: ^__meta_ecs_container_labels_(.+)$ # Capture the key using regex, e.g. __meta_ecs_container_labels_app -> app
              replacement: '$$1'

processors:
  metrics_transform:
    transforms:
      - include: ".*" # Rename customized job label back to job
        match_type: regexp
        action: update
        operations:
          - label: prometheus_job # must match the value configured in ecs_observer
            new_label: job
            action: update_label
  filter/cloudwatch: # The CloudWatch OTLP endpoint doesn't accept Prometheus summaries
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_SUMMARY
  cumulative_to_delta: # Prometheus counters have no start time, which the CloudWatch OTLP endpoint requires
  batch:

exporters:
  otlp_http:
    compression: gzip
    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
    auth:
      authenticator: sigv4auth

service:
  extensions: [ sigv4auth, ecs_observer ] # extensions are declared outside pipelines
  pipelines:
    metrics:
      receivers: [ prometheus ]
      processors: [ metrics_transform, filter/cloudwatch, cumulative_to_delta, batch ]
      exporters: [ otlp_http ]

Deploy ADOT Collector

ADOT Collector needs to run as single instance. We use the replica service because it handles task failure. Replica is 1 otherwise we will have duplicated metrics. If you have a large cluster, you can shard the scrape targets using hashmod or use different discovery configurations for multiple collectors.

# Run ADOT Collector as replica service on EC2
# https://docs.aws.amazon.com/cli/latest/reference/ecs/create-service.html
aws ecs create-service \
    --cluster my-ecs-cluster \
    --service-name adot-ecs-prometheus \
    --task-definition adot-ecs-prometheus:1 \
    --desired-count 1

Sample Workloads

There are four workloads with pre-built dashboards, JMX is the easiest to get started with.

JMX

Details about example JMX workload can be found from CloudWatch docs ECS JMX Sample and ECS JMX Metrics Collected

Our integration test also includes a sample java app.

NGINX

Detail about the example NGINX workload can be found from CloudWatch docs ECS Nginx Sample nginx-prometheus-exporter is an open source project.

NGINX Plus

NGINX Plus is similar to NGINX but requires subscription. Instruction for building image and run the workload can be found from CloudWatch docs Build Nginx Plus Image

AWS App Mesh

CloudWatch docs has a simple walk through ECS App Mesh Sample. Be aware it's not easy to set up Mesh. All the metrics (used in dashboard) are collected from Envoy.

View metrics in the CloudWatch console

In the CloudWatch console, open Query Studio and query the metrics with PromQL. For example, the active connections of the NGINX sample workload:

{"nginx_connections_active", "ClusterName"="my-ecs-cluster", "ServiceName"="nginx-service"}

The labels from the relabel rules, such as ClusterName, ServiceName, and TaskDefinitionFamily, are available as datapoint labels. Because of the cumulative_to_delta processor, counters such as nginx_http_requests_total are stored as deltas, so each datapoint is the increase since the previous scrape. For details, see Query metrics with PromQL.

Appendix

Create ECS Cluster with EC2 instances

Download ecs-cli.
Create a new cluster, take note of the security group and subnet it created.
If you missed it, you can still find it from CloudFormation console, ecs-cli generates CloudFormation and applies it.
To make testing easier, update security group to allow ingress on all TCP port within the cluster, this makes testing easier. However, you might want to only allow specific port when you use non bridge network mode. Bridge network mode assigns random host ports so you have to allow all.
Delete the cluster by either deleting the AWS CloudFormation stack or using ecs-cli down -f -cluster ${CLUSTER_NAME}

# Create ECS EC2 cluster with 2 EC2 instances

export CLUSTER_NAME=aoc-containerinsight-prometheus-example
ecs-cli up --capability-iam --size 2 --instance-type t2.medium --cluster ${CLUSTER_NAME} --region us-west-2

# Output is like
# INFO[0000] Using recommended Amazon Linux 2 AMI with ECS Agent 1.51.0 and Docker version 19.03.13-ce
# INFO[0001] Created cluster                               cluster=aoc-containerinsight-prometheus-example region=us-west-2
# INFO[0001] Waiting for your cluster resources to be created...
# INFO[0122] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
# VPC created: vpc-0fcxxxxx
# Security Group created: sg-04xxxxx
# Subnet created: subnet-03xxxx
# Subnet created: subnet-0exxxx
# Cluster creation succeeded.

# Update security group to allow ingress on all ports within security group
SG=sg-04xxxxx
aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 0-65535 --source-group ${SG}
aws ec2 describe-security-groups --group-ids ${SG}

IAM Policy Document

The task role needs permission to send metrics to the CloudWatch OTLP metrics endpoint and the extra permissions to discover ECS tasks:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ec2:DescribeInstances",
                "ecs:ListTasks",
                "ecs:ListServices",
                "ecs:DescribeContainerInstances",
                "ecs:DescribeServices",
                "ecs:DescribeTasks",
                "ecs:DescribeTaskDefinition"
            ],
            "Resource": "*"
        }
    ]
}

The task execution role reads the collector configuration from the SSM parameter, so it needs ssm:GetParameters on that parameter. It also needs logs:CreateLogGroup, because the task definition below sets awslogs-create-group. Attach these permissions along with the AmazonECSTaskExecutionRolePolicy managed policy, which allows pulling the image and writing the collector's own logs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:GetParameters",
            "Resource": "arn:aws:ssm:us-west-2:123456:parameter/aoc/ecs-prometheus"
        },
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-west-2:123456:log-group:/adot/collector"
        }
    ]
}

ECS Task Definition

NOTE: You need to replace placeholder values with actual value

The account id in the example is 123456 and is used in ARN for taskRole and executionRole, replace it with your own AWS account id.
public.ecr.aws/aws-observability/aws-otel-collector:v0.50.0 is the ADOT Collector image in Amazon ECR Public. Use a current release.
/aoc/ecs-prometheus is the name of SSM parameter.
networkMode is awsvpc, which both Fargate and EC2 support. On EC2 you can also use bridge or host, but then remove FARGATE from requiresCompatibilities.
logConfiguration is for logs from the ADOT Collector itself.
requiresCompatibilities includes both EC2 and Fargate because ADOT Collector is scraping metrics via HTTP and does not need special host level access.
cpu and memory must follow Fargate's requirement e.g. cpu 255 memory 511 is invalid pair.

{
  "family": "adot-ecs-prometheus",
  "taskRoleArn": "arn:aws:iam::123456:role/MyEcsTaskRole",
  "executionRoleArn": "arn:aws:iam::123456:role/MyEcsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "adot",
      "image": "public.ecr.aws/aws-observability/aws-otel-collector:v0.50.0",
      "secrets": [
        {
          "name": "AOT_CONFIG_CONTENT",
          "valueFrom": "/aoc/ecs-prometheus"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "True",
          "awslogs-group": "/adot/collector",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs-prometheus"
        }
      }
    }
  ],
  "requiresCompatibilities": [
    "EC2",
    "FARGATE"
  ],
  "cpu": "256",
  "memory": "512"
}

Container Insights ECS Prometheus

Container Insights ECS Prometheus

Overview

Steps

Permission

IAM Policy

Network

Configuration

Deploy ADOT Collector

Sample Workloads

JMX

NGINX

NGINX Plus

AWS App Mesh

View metrics in the CloudWatch console

Appendix

Create ECS Cluster with EC2 instances

IAM Policy Document

ECS Task Definition

On this page