AWS Distro for OpenTelemetry

Setting up the ADOT Collector on Amazon ECS

Setting up the ADOT Collector on Amazon ECS

Run the AWS Distro for OpenTelemetry (ADOT) Collector as a sidecar container in your Amazon ECS tasks, on AWS Fargate or Amazon EC2. Your applications send telemetry to the collector over OTLP. The collector adds ECS and EC2 resource attributes and sends traces and metrics to the Amazon CloudWatch OTLP endpoints. You can search the traces with Transaction Search and query the metrics with PromQL.

💡
This guide is for Amazon ECS. On Amazon EKS, use the Amazon CloudWatch Observability EKS add-on, which runs the CloudWatch agent and provides Container Insights and Application Signals.



Prerequisites

  • An Amazon ECS cluster. To create one, see Creating an Amazon ECS cluster.
  • Transaction Search enabled in CloudWatch. The OTLP traces endpoint requires it.
  • A task IAM role that allows the collector to send telemetry, such as one with the CloudWatchAgentServerPolicy managed policy, or a policy with these actions:
1{
2 "Version": "2012-10-17",
3 "Statement": [
4 {
5 "Effect": "Allow",
6 "Action": [
7 "cloudwatch:PutMetricData",
8 "xray:PutTraceSegments",
9 "xray:PutTelemetryRecords",
10 "xray:GetSamplingRules",
11 "xray:GetSamplingTargets",
12 "xray:GetSamplingStatisticSummaries"
13 ],
14 "Resource": "*"
15 }
16 ]
17}



Option 1: Deploy with AWS CloudFormation

The CloudFormation templates create a task definition and service that run the ADOT Collector next to a sample Java application, which is instrumented with the ADOT Java agent, and a traffic generator. With CreateIAMRoles=True, the templates also create the task role and task execution role.

Amazon ECS on AWS Fargate

1curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-fargate-sidecar-deployment-cfn.yaml
2
3ClusterName=<cluster-name>
4Region=<region>
5SecurityGroups=<security-group-id>
6Subnets=<subnet-id-1>\\,<subnet-id-2>
7
8aws cloudformation create-stack --stack-name ADOT-ECS-Fargate-${ClusterName} \
9 --template-body file://aws-otel-fargate-sidecar-deployment-cfn.yaml \
10 --parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
11 ParameterKey=CreateIAMRoles,ParameterValue=True \
12 ParameterKey=SecurityGroups,ParameterValue=${SecurityGroups} \
13 ParameterKey=Subnets,ParameterValue=${Subnets} \
14 --capabilities CAPABILITY_NAMED_IAM \
15 --region ${Region}

The task needs a route to the CloudWatch OTLP endpoints and to Amazon ECR Public, for example through a public subnet with a public IP or a NAT gateway.

Amazon ECS on Amazon EC2

1curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-ec2-sidecar-deployment-cfn.yaml
2
3ClusterName=<cluster-name>
4Region=<region>
5
6aws cloudformation create-stack --stack-name ADOT-ECS-EC2-${ClusterName} \
7 --template-body file://aws-otel-ec2-sidecar-deployment-cfn.yaml \
8 --parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \
9 ParameterKey=CreateIAMRoles,ParameterValue=True \
10 --capabilities CAPABILITY_NAMED_IAM \
11 --region ${Region}

Leave the command parameter empty to use the CloudWatch OTLP configuration in the template. To use a configuration that is bundled in the collector image instead, set command, for example to --config=/etc/ecs/ecs-default-config.yaml. See Moving from the EMF configuration.




Option 2: Add the collector to your task definition

1. Store the collector configuration

Save this configuration as adot-collector-config.yaml:

1extensions:
2 health_check:
3 sigv4auth/traces:
4 region: ${env:AWS_REGION}
5 service: xray
6 sigv4auth/metrics:
7 region: ${env:AWS_REGION}
8 service: monitoring
9
10receivers:
11 otlp:
12 protocols:
13 grpc:
14 endpoint: 0.0.0.0:4317
15 http:
16 endpoint: 0.0.0.0:4318
17
18processors:
19 # Adds aws.ecs.*, cloud.*, and, on EC2, host.* resource attributes.
20 resource_detection:
21 detectors: [env, ecs, ec2]
22 batch/traces:
23 timeout: 1s
24 send_batch_size: 50
25 batch/metrics:
26 timeout: 60s
27
28exporters:
29 otlp_http/traces:
30 compression: gzip
31 traces_endpoint: https://xray.${env:AWS_REGION}.amazonaws.com/v1/traces
32 auth:
33 authenticator: sigv4auth/traces
34 otlp_http/metrics:
35 compression: gzip
36 metrics_endpoint: https://monitoring.${env:AWS_REGION}.amazonaws.com/v1/metrics
37 auth:
38 authenticator: sigv4auth/metrics
39
40service:
41 extensions: [health_check, sigv4auth/traces, sigv4auth/metrics]
42 pipelines:
43 traces:
44 receivers: [otlp]
45 processors: [resource_detection, batch/traces]
46 exporters: [otlp_http/traces]
47 metrics:
48 receivers: [otlp]
49 processors: [resource_detection, batch/metrics]
50 exporters: [otlp_http/metrics]

Store it in an AWS Systems Manager parameter:

aws ssm put-parameter --name adot-collector-config --type String \
--value file://adot-collector-config.yaml --region <region>

The task execution role needs ssm:GetParameters on this parameter.

2. Add the collector container

Add this container to your task definition. The collector reads its configuration from the AOT_CONFIG_CONTENT environment variable.

1{
2 "name": "aws-otel-collector",
3 "image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
4 "essential": true,
5 "environment": [
6 { "name": "AWS_REGION", "value": "<region>" }
7 ],
8 "secrets": [
9 { "name": "AOT_CONFIG_CONTENT", "valueFrom": "adot-collector-config" }
10 ],
11 "healthCheck": {
12 "command": ["/healthcheck"],
13 "interval": 5,
14 "timeout": 3,
15 "retries": 2
16 },
17 "logConfiguration": {
18 "logDriver": "awslogs",
19 "options": {
20 "awslogs-create-group": "true",
21 "awslogs-group": "/ecs/aws-otel-collector",
22 "awslogs-region": "<region>",
23 "awslogs-stream-prefix": "ecs"
24 }
25 }
26}

To start your application after the collector, add a dependsOn entry with "condition": "START" for the aws-otel-collector container.




Send telemetry from your application

ProtocolCollector portUse for
OTLP/HTTP4318OpenTelemetry and ADOT SDKs
OTLP/gRPC4317OpenTelemetry and ADOT SDKs

Point your SDK at the collector with OTEL_EXPORTER_OTLP_ENDPOINT:

  • With the awsvpc network mode (always used on Fargate), containers in a task share localhost: http://localhost:4318.
  • With the bridge network mode, add a link to the collector container and use its name: http://aws-otel-collector:4318.

The collector only accepts OTLP traces and metrics. Set OTEL_LOGS_EXPORTER=none in your application, or the SDK reports errors when it tries to export logs.




View your telemetry

  • Traces: In the CloudWatch console, use Transaction Search to search spans and the trace map to see how your services connect. Spans are stored in the aws/spans log group.

  • Metrics: In the CloudWatch console, open Query Studio and query with PromQL. For example, the JVM memory of the sample application from the CloudFormation templates:

    {"jvm.memory.used", "@resource.service.name"="ecs-otel-sample-app"}

    Resource attributes that the collector adds, such as aws.ecs.cluster.arn and aws.ecs.task.family, are available as @resource. labels. For details, see Query metrics with PromQL.




Container and task metrics

For CPU, memory, network, and storage metrics for your tasks and containers, turn on Container Insights with enhanced observability for the cluster. Amazon ECS publishes these metrics without a collector.

If you need task and container metrics as OpenTelemetry metrics in the CloudWatch OTLP endpoint, add the awsecscontainermetrics receiver to the metrics pipeline. Each collector reports only the task it runs in, so add the collector to every task that you want metrics for. The receiver uses CloudWatch-style units, such as Bytes, MiB, and Count, rather than OpenTelemetry (UCUM) units.

1receivers:
2 awsecscontainermetrics:
3 collection_interval: 60s
4
5service:
6 pipelines:
7 metrics:
8 receivers: [otlp, awsecscontainermetrics]



Moving from the EMF configuration

Earlier versions of these templates and of this guide used --config=/etc/ecs/ecs-default-config.yaml, which sends metrics as CloudWatch embedded metric format (EMF) to the ECS/AWSOTel/Application namespace and traces through the awsxray exporter. With the configuration in this guide, metrics go to the CloudWatch OTLP endpoint instead, so alarms and dashboards on the ECS/AWSOTel/Application metrics stop receiving data. Move them to PromQL queries, or keep the EMF configuration by setting the template's command parameter to --config=/etc/ecs/ecs-default-config.yaml.

The configuration in this guide receives OTLP only. If your applications send StatsD metrics or use the X-Ray SDK, keep --config=/etc/ecs/ecs-default-config.yaml, which also receives StatsD on UDP port 8125 and X-Ray segments on UDP port 2000, or move them to an OpenTelemetry SDK.




Clean up

Delete the CloudFormation stack, and the SSM parameter if you created one:

aws cloudformation delete-stack --stack-name <stack-name> --region <region>
aws ssm delete-parameter --name adot-collector-config --region <region>