Getting started using AWS ECS container metrics receiver in AWS Distro for OpenTelemetry Collector

The receiver is also documented upstream in the awsecscontainermetricsreceiver README.

The Amazon ECS container agent provides a method for customers to retrieve various task metadata and Docker stats by using ECS Task Metadata Endpoint. The AWS Container Observability team developed a receiver in the OpenTelemetry Collector that scrapes this endpoint and collects container metrics (such as CPU, memory, network, and disk). Customers can enable awsecscontainermetrics receiver in their OpenTelemetry configuration file to collect specific task- and container-level metrics and send the data to desired monitoring tools such as Amazon CloudWatch.

This receiver works with ECS Task Metadata Endpoint V4, which means Amazon ECS tasks with Fargate launch type with platform version 1.4.0 or later or Amazon ECS tasks with Amazon EC2 launch type with ECS agent version 1.39.0+. For more information, see Amazon ECS Container Agent Versions.

Enabling the AWS ECS Container Metrics Receiver

To enable the awsecscontainermetrics receiver, add the name under receiver section in the config file (local/config.yaml). By default, the receiver scrapes the ECS task metadata endpoint every 20s and collects all metrics (For the full list of metrics, see Available Metrics).

The following configuration collects AWS ECS resource usage metrics by using awsecscontainermetrics receiver and sends them to the CloudWatch OTLP metrics endpoint with the otlp_http exporter. The sigv4auth extension signs the requests with the task role's credentials, so the task role needs the cloudwatch:PutMetricData permission. Replace us-west-2 with your Region.

The CloudWatch OTLP metrics endpoint requires a start time on cumulative datapoints. The receiver reports its CPU usage time, network I/O, and storage metrics as cumulative sums without a start time, so the endpoint rejects them and the collector logs a Partial success response warning. The filter/cloudwatch processor drops these metrics; the other metrics are gauges, which the endpoint accepts. For the full list, see Available Metrics. This is tracked upstream in opentelemetry-collector-contrib#52512.

Copy
1extensions:
2  sigv4auth:
3    region: us-west-2
4    service: monitoring
5
6receivers:
7  awsecscontainermetrics:
8
9processors:
10  filter/cloudwatch: # drop the cumulative sums, which the CloudWatch OTLP endpoint rejects
11    metrics:
12      exclude:
13        match_type: regexp
14        metric_names:
15          - .*\.cpu\.usage\.(total|kernelmode|usermode|system)$
16          - .*\.network\.io\.usage\..*
17          - .*\.storage\.(read|write)_bytes$
18
19exporters:
20  otlp_http:
21    compression: gzip
22    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
23    auth:
24      authenticator: sigv4auth
25
26service:
27  extensions: [sigv4auth]
28  pipelines:
29    metrics:
30      receivers: [awsecscontainermetrics]
31      processors: [filter/cloudwatch]
32      exporters: [otlp_http]

To query the metrics, open Query Studio in the CloudWatch console and use PromQL. For example, the memory that each task in a cluster uses:

Copy
{"ecs.task.memory.utilized", "@resource.aws.ecs.cluster.name"="my-cluster"}

For details, see Query metrics with PromQL. To get Container Insights metrics for ECS without running a collector, see Container Insights for Amazon ECS.

Set Metrics Collection Interval

Customers can configure collection_interval under awsecscontainermetrics receiver to scrape and gather metrics at a specific interval. The following example configuration will collect metrics every 40 seconds.

Copy
1extensions:
2  sigv4auth:
3    region: us-west-2
4    service: monitoring
5
6receivers:
7  awsecscontainermetrics:
8    collection_interval: 40s
9
10processors:
11  filter/cloudwatch: # drop the cumulative sums, which the CloudWatch OTLP endpoint rejects
12    metrics:
13      exclude:
14        match_type: regexp
15        metric_names:
16          - .*\.cpu\.usage\.(total|kernelmode|usermode|system)$
17          - .*\.network\.io\.usage\..*
18          - .*\.storage\.(read|write)_bytes$
19
20exporters:
21  otlp_http:
22    compression: gzip
23    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
24    auth:
25      authenticator: sigv4auth
26
27service:
28  extensions: [sigv4auth]
29  pipelines:
30    metrics:
31      receivers: [awsecscontainermetrics]
32      processors: [filter/cloudwatch]
33      exporters: [otlp_http]

Collect specific metrics and update metric names

The previous configuration collects all the metrics and sends them to Amazon CloudWatch using default names. Customers can use filter and metrics_transform processors to send specific metrics and rename them respectively.

The following configuration example collects only the ecs.task.memory.utilized metric and renames it to MemoryUtilized before sending to CloudWatch.

Copy
1extensions:
2  sigv4auth:
3    region: us-west-2
4    service: monitoring
5
6receivers:
7  awsecscontainermetrics:
8
9processors:
10  filter:
11    metrics:
12      include:
13        match_type: strict
14        metric_names:
15          - ecs.task.memory.utilized
16  metrics_transform:
17    transforms:
18      - include: ecs.task.memory.utilized
19        action: update
20        new_name: MemoryUtilized
21
22exporters:
23  otlp_http:
24    compression: gzip
25    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
26    auth:
27      authenticator: sigv4auth
28
29service:
30  extensions: [sigv4auth]
31  pipelines:
32    metrics:
33      receivers: [awsecscontainermetrics]
34      processors: [filter, metrics_transform]
35      exporters: [otlp_http]

Available Metrics

The following table lists all metrics emitted by AWS ECS container metrics receiver.

The cpu.usage.total, cpu.usage.kernelmode, cpu.usage.usermode, cpu.usage.system, network.io.usage.*, and storage.* metrics are cumulative sums without a start time, which the CloudWatch OTLP metrics endpoint rejects. The other metrics are gauges.

Resource Attributes and Metrics Labels

Metrics emitted by this receiver come with a set of resource attributes. The CloudWatch OTLP metrics endpoint keeps them, and you can filter and group by them in PromQL as @resource. labels, for example "@resource.aws.ecs.cluster.name"="my-cluster". The following table lists the resource attributes for task-level and container-level metrics. Container-level metrics have seven more attributes than task-level metrics.

Full configuration for task level metrics

The following example shows a full configuration to get the most useful task-level metrics. It uses the awsecscontainermetrics receiver to collect all the resource usage metrics from the ECS task metadata endpoint, applies the filter processor to select only 6 task-level gauges, batches them, and sends them to the CloudWatch OTLP metrics endpoint. The resource attributes, such as aws.ecs.cluster.name and aws.ecs.task.family, stay on the metrics, so you don't need to rename them to use them in PromQL queries.

Copy
1extensions:
2  sigv4auth:
3    region: us-west-2
4    service: monitoring
5
6receivers:
7  awsecscontainermetrics: # collect 52 metrics
8
9processors:
10  filter: # filter metrics
11    metrics:
12      include:
13        match_type: strict
14        metric_names: # select only 6 task level gauges
15          - ecs.task.memory.reserved
16          - ecs.task.memory.utilized
17          - ecs.task.cpu.reserved
18          - ecs.task.cpu.utilized
19          - ecs.task.network.rate.rx
20          - ecs.task.network.rate.tx
21  batch:
22
23exporters:
24  otlp_http:
25    compression: gzip
26    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
27    auth:
28      authenticator: sigv4auth
29
30service:
31  extensions: [sigv4auth]
32  pipelines:
33    metrics:
34      receivers: [awsecscontainermetrics]
35      processors: [filter, batch]
36      exporters: [otlp_http]

Full configuration for task- and container-level metrics

The following example collects the same 6 metrics at both the task level and the container level. The filter processor matches the metric names with regular expressions, so it selects both the ecs.task.* and the container.* metrics. Container-level metrics also carry the container.name resource attribute.

Copy
1extensions:
2  sigv4auth:
3    region: us-west-2
4    service: monitoring
5
6receivers:
7  awsecscontainermetrics:
8
9processors:
10  filter:
11    metrics:
12      include:
13        match_type: regexp
14        metric_names:
15          - .*memory.reserved
16          - .*memory.utilized
17          - .*cpu.reserved
18          - .*cpu.utilized
19          - .*network.rate.rx
20          - .*network.rate.tx
21  batch:
22
23exporters:
24  otlp_http:
25    compression: gzip
26    metrics_endpoint: https://monitoring.us-west-2.amazonaws.com/v1/metrics
27    auth:
28      authenticator: sigv4auth
29
30service:
31  extensions: [sigv4auth]
32  pipelines:
33    metrics:
34      receivers: [awsecscontainermetrics]
35      processors: [filter, batch]
36      exporters: [otlp_http]