AWS Distro for OpenTelemetry
Setting up the ADOT Collector on Amazon ECS
Setting up the ADOT Collector on Amazon ECS
Run the AWS Distro for OpenTelemetry (ADOT) Collector as a sidecar container in your Amazon ECS tasks, on AWS Fargate or Amazon EC2. Your applications send telemetry to the collector over OTLP. The collector adds ECS and EC2 resource attributes and sends traces and metrics to the Amazon CloudWatch OTLP endpoints. You can search the traces with Transaction Search and query the metrics with PromQL.
Prerequisites
- An Amazon ECS cluster. To create one, see Creating an Amazon ECS cluster.
- Transaction Search enabled in CloudWatch. The OTLP traces endpoint requires it.
- A task IAM role that allows the collector to send telemetry, such as one with the
CloudWatchAgentServerPolicymanaged policy, or a policy with these actions:
1{2 "Version": "2012-10-17",3 "Statement": [4 {5 "Effect": "Allow",6 "Action": [7 "cloudwatch:PutMetricData",8 "xray:PutTraceSegments",9 "xray:PutTelemetryRecords",10 "xray:GetSamplingRules",11 "xray:GetSamplingTargets",12 "xray:GetSamplingStatisticSummaries"13 ],14 "Resource": "*"15 }16 ]17}Option 1: Deploy with AWS CloudFormation
The CloudFormation templates create a task definition and service that run the ADOT Collector next to a sample Java
application, which is instrumented with the ADOT Java agent, and a traffic generator. With
CreateIAMRoles=True, the templates also create the task role and task execution role.
Amazon ECS on AWS Fargate
1curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-fargate-sidecar-deployment-cfn.yaml2
3ClusterName=<cluster-name>4Region=<region>5SecurityGroups=<security-group-id>6Subnets=<subnet-id-1>\\,<subnet-id-2>7
8aws cloudformation create-stack --stack-name ADOT-ECS-Fargate-${ClusterName} \9 --template-body file://aws-otel-fargate-sidecar-deployment-cfn.yaml \10 --parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \11 ParameterKey=CreateIAMRoles,ParameterValue=True \12 ParameterKey=SecurityGroups,ParameterValue=${SecurityGroups} \13 ParameterKey=Subnets,ParameterValue=${Subnets} \14 --capabilities CAPABILITY_NAMED_IAM \15 --region ${Region}The task needs a route to the CloudWatch OTLP endpoints and to Amazon ECR Public, for example through a public subnet with a public IP or a NAT gateway.
Amazon ECS on Amazon EC2
1curl -O https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/ecs/aws-otel-ec2-sidecar-deployment-cfn.yaml2
3ClusterName=<cluster-name>4Region=<region>5
6aws cloudformation create-stack --stack-name ADOT-ECS-EC2-${ClusterName} \7 --template-body file://aws-otel-ec2-sidecar-deployment-cfn.yaml \8 --parameters ParameterKey=ClusterName,ParameterValue=${ClusterName} \9 ParameterKey=CreateIAMRoles,ParameterValue=True \10 --capabilities CAPABILITY_NAMED_IAM \11 --region ${Region}Leave the command parameter empty to use the CloudWatch OTLP configuration in the template. To use a configuration
that is bundled in the collector image instead, set command, for example to --config=/etc/ecs/ecs-default-config.yaml.
See Moving from the EMF configuration.
Option 2: Add the collector to your task definition
1. Store the collector configuration
Save this configuration as adot-collector-config.yaml:
1extensions:2 health_check:3 sigv4auth/traces:4 region: ${env:AWS_REGION}5 service: xray6 sigv4auth/metrics:7 region: ${env:AWS_REGION}8 service: monitoring9
10receivers:11 otlp:12 protocols:13 grpc:14 endpoint: 0.0.0.0:431715 http:16 endpoint: 0.0.0.0:431817
18processors:19 # Adds aws.ecs.*, cloud.*, and, on EC2, host.* resource attributes.20 resource_detection:21 detectors: [env, ecs, ec2]22 batch/traces:23 timeout: 1s24 send_batch_size: 5025 batch/metrics:26 timeout: 60s27
28exporters:29 otlp_http/traces:30 compression: gzip31 traces_endpoint: https://xray.${env:AWS_REGION}.amazonaws.com/v1/traces32 auth:33 authenticator: sigv4auth/traces34 otlp_http/metrics:35 compression: gzip36 metrics_endpoint: https://monitoring.${env:AWS_REGION}.amazonaws.com/v1/metrics37 auth:38 authenticator: sigv4auth/metrics39
40service:41 extensions: [health_check, sigv4auth/traces, sigv4auth/metrics]42 pipelines:43 traces:44 receivers: [otlp]45 processors: [resource_detection, batch/traces]46 exporters: [otlp_http/traces]47 metrics:48 receivers: [otlp]49 processors: [resource_detection, batch/metrics]50 exporters: [otlp_http/metrics]Store it in an AWS Systems Manager parameter:
aws ssm put-parameter --name adot-collector-config --type String \ --value file://adot-collector-config.yaml --region <region>The task execution role needs ssm:GetParameters on this parameter.
2. Add the collector container
Add this container to your task definition. The collector reads its configuration from the AOT_CONFIG_CONTENT
environment variable.
1{2 "name": "aws-otel-collector",3 "image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",4 "essential": true,5 "environment": [6 { "name": "AWS_REGION", "value": "<region>" }7 ],8 "secrets": [9 { "name": "AOT_CONFIG_CONTENT", "valueFrom": "adot-collector-config" }10 ],11 "healthCheck": {12 "command": ["/healthcheck"],13 "interval": 5,14 "timeout": 3,15 "retries": 216 },17 "logConfiguration": {18 "logDriver": "awslogs",19 "options": {20 "awslogs-create-group": "true",21 "awslogs-group": "/ecs/aws-otel-collector",22 "awslogs-region": "<region>",23 "awslogs-stream-prefix": "ecs"24 }25 }26}To start your application after the collector, add a dependsOn entry with "condition": "START" for the
aws-otel-collector container.
Send telemetry from your application
| Protocol | Collector port | Use for |
|---|---|---|
| OTLP/HTTP | 4318 | OpenTelemetry and ADOT SDKs |
| OTLP/gRPC | 4317 | OpenTelemetry and ADOT SDKs |
Point your SDK at the collector with OTEL_EXPORTER_OTLP_ENDPOINT:
- With the
awsvpcnetwork mode (always used on Fargate), containers in a task sharelocalhost:http://localhost:4318. - With the
bridgenetwork mode, add a link to the collector container and use its name:http://aws-otel-collector:4318.
The collector only accepts OTLP traces and metrics. Set OTEL_LOGS_EXPORTER=none in your application, or the SDK
reports errors when it tries to export logs.
View your telemetry
Traces: In the CloudWatch console, use Transaction Search to search spans and the trace map to see how your services connect. Spans are stored in the
aws/spanslog group.Metrics: In the CloudWatch console, open Query Studio and query with PromQL. For example, the JVM memory of the sample application from the CloudFormation templates:
{"jvm.memory.used", "@resource.service.name"="ecs-otel-sample-app"}Resource attributes that the collector adds, such as
aws.ecs.cluster.arnandaws.ecs.task.family, are available as@resource.labels. For details, see Query metrics with PromQL.
Container and task metrics
For CPU, memory, network, and storage metrics for your tasks and containers, turn on Container Insights with enhanced observability for the cluster. Amazon ECS publishes these metrics without a collector.
If you need task and container metrics as OpenTelemetry metrics in the CloudWatch OTLP endpoint, add the
awsecscontainermetrics
receiver to the metrics pipeline. Each collector reports only the task it runs in, so add the collector to every task
that you want metrics for. The receiver uses CloudWatch-style units, such as Bytes, MiB, and Count, rather than
OpenTelemetry (UCUM) units.
1receivers:2 awsecscontainermetrics:3 collection_interval: 60s4
5service:6 pipelines:7 metrics:8 receivers: [otlp, awsecscontainermetrics]Moving from the EMF configuration
Earlier versions of these templates and of this guide used --config=/etc/ecs/ecs-default-config.yaml, which sends
metrics as CloudWatch embedded metric format (EMF) to the ECS/AWSOTel/Application namespace and traces through the
awsxray exporter. With the configuration in this guide, metrics go to the CloudWatch OTLP endpoint instead, so
alarms and dashboards on the ECS/AWSOTel/Application metrics stop receiving data. Move them to PromQL queries, or keep
the EMF configuration by setting the template's command parameter to --config=/etc/ecs/ecs-default-config.yaml.
The configuration in this guide receives OTLP only. If your applications send StatsD metrics or use the X-Ray SDK, keep
--config=/etc/ecs/ecs-default-config.yaml, which also receives StatsD on UDP port 8125 and X-Ray segments on UDP
port 2000, or move them to an OpenTelemetry SDK.
Clean up
Delete the CloudFormation stack, and the SSM parameter if you created one:
aws cloudformation delete-stack --stack-name <stack-name> --region <region>aws ssm delete-parameter --name adot-collector-config --region <region>