AWS Distro for OpenTelemetry

Getting Started with the AWS X-Ray Exporter in the Collector

The OpenTelemetry Collector has three key components: the receiver, exporter, and processor. The receiver in the OpenTelemetry Collector is responsible for receiving data via gRPC or HTTP using the OpenTelemetry protocol (OTLP). The AWS X-Ray exporter available in the OpenTelemetry Collector converts OTLP formatted trace data to the AWS X-Ray format and then exports this data to the AWS X-Ray service.

In this guide, we will demonstrate how to configure the AWS Distro for the OpenTelemetry (ADOT) Collector for use with AWS X-Ray. To learn more about AWS X-Ray, see the X-Ray Developer Guide.

Prerequisites

To get started using the ADOT Collector to connect with AWS X-Ray to analyze distributed traces, you have to first install AWS Distro for the OpenTelemetry Collector. Make sure you have your access key ID and secret access key properly configured in the docker file. You can generate access key ID and secret access key with the information here.

In order to set up the ADOT Collector on a particular platform such as EKS, ECS, EC2 or on-premises, please see the Getting Started guide for

ADOT also includes fully managed AWS Lambda Layers, which include an OpenTelemetry SDK and the ADOT Collector to auto-instrument your function for tracing with X-Ray. See AWS Distro for OpenTelemetry Lambda to get started.

Configuring the OTLP Receiver

To enable the OTLP receiver, we include the “otlp” flag in the definitions. You will have to set the “endpoint” consisting of <host>:<port> for the gRPC or HTTP protocols. You can change the endpoints depending on how you set up your instrumentation with OpenTelemetry.

Example:

1receivers:
2  otlp:
3    protocols:
4      grpc:
5        endpoint: 0.0.0.0:4317
6      http:
7        endpoint: 0.0.0.0:4318

Note: A protocol can be disabled by not specifying it in the list of protocols.

Configuring the AWS X-Ray Exporter

By default, the AWS Distribution for the OpenTelemetry Collector has enabled exporting to AWS X-Ray without any additional configurations - converting the AWS X-Ray OTLP formatted trace data to the AWS X-Ray format and then exports this data to the AWS X-Ray service. For additional configuration options of the AWS X-Ray exporter in the Collector, you can find the comprehensive list here.

Example:

1exporters:
2  awsxray:
3    # you can add additional configuration here
4    # for example configure sending traces to AWS X-Ray in a specific region
5    region: us-west-2

Note: These configurations are optional

OTel to X-Ray data model translation behavior of AWS X-Ray Exporter

OTel span attributes translation

OTel data model field/Attribute keys	X-Ray data model field	Notes
Attributes	annotations/metadata	index_all_attributes is set to false by default. By default the exporter will translate all the OpenTelemetry span attributes to metadata. If set to true, then it will convert all span attributes to annotations. indexed_attributes converts a list of attributes to annotations, and concerts remaining OpenTelemetry span attributes to metadata.
SpanId	id	X-Ray (sub)segment ID is a 64-bit identifier which is unique within a single trace, and consists of 16 hexadecimal digits. OpenTelemetry SpanId is a valid span identifier consisting of an 8-byte array, with at least one non-zero byte.
TraceId	trace_id	An X-Ray trace_id consists of three values separated by hyphens. For example, the trace_id {1-5759e988-bd862e3fe1be46a994272793} includes three values: the version number 1, the time of the original request in Unix epoch time, and finally a 96-bit identifier for the trace which is globally unique. OpenTelemetry TraceID is a valid trace identifier consisting of a 16-byte array with at least one non-zero byte. 3. X-Ray accepts trace IDs generated within the past 30 days. If the epoch time portion of the trace ID is outside 30 days, the exporter returns an error.
StartTime	start_time
EndTime	end_time
ParentSpanId	parent_id
Status.StatusCode	fault	When StatusCode = Error then fault is set to true for a 500-class HTTP response. The exception stack trace is captured in the exception field. Any other values of StatusCode result in fault being set to false.
Status.StatusCode	error	When StatusCode = Error then error is set to true for a 400-class HTTP response. The exception stack trace is captured in the exception field. Any other values of StatusCode result in error being set to false.
Status.StatusCode	throttle	When StatusCode = Error then error is set to true for a 429 HTTP response. The exception stack trace is captured in the exception field. Any other values of StatusCode result in throttle being set to false.
Event	exception	This field is populated by span exception events when the StatusCode = Error ; all other span events are dropped. Note: Because stack traces are recorded in a language-dependent format, the `telemetry.sdk.language` attribute must be set for stack traces to be recorded.
Link	N/A	Not currently supported by the exporter.
enduser.id	user	This field is only populated on X-Ray segments, not subsegments.
cloud.provider and cloud.platform	origin	If value of `cloud.provider` attribute key within span resource attributes = “aws", then the exporter copies the value from cloud.platform attribute key to the origin field, if the cloud.platform key is one of the following supported values: `aws_app_runner`, `aws_eks`, `aws_elastic_beanstalk`, `aws_ecs` and `aws_ec2`. Any other cloud.platform key values then origin will not be set.
rpc.system and aws.service	namespace	If namespace has not yet been set, and one of the following conditions is true, then namespace will be set to "aws": - rpc.system attribute key = "aws-api" - aws.service attribute key is not empty. If namespace has not yet been set, and spankind = "client", then namespace is set to "remote". In all other cases, namespace is not set.
peer.service, aws.service, db.service, service.name, span.kind, and span name	name	Name field is set to peer.service if not empty. If peer.service is empty and aws.service attribute key is not empty, name is set to aws.service. If aws.service is empty and db.service attribute key is not empty, name is set to db.service. If none of these attribute keys has a value, and span.kind = "Server", then name is set to value of service.name attribute key. If none of the prior conditions are met, name is set to the name of the span.
pdata.SpanKindServer	type	If span.kind = "Server" then the exporter creates an X-Ray segment. Otherwise a subsegment is created.

OTel span HTTP attributes translation

OTel Attributes keys	X-Ray data model field	Notes
http.method, http.request.method	http.request.method
http.client_ip	http.request.client_ip
http.client_ip	http.request.x_forwarded_for
http.user_agent, user_agent.original	http.request.user_agent
http.status_code, http.response.status	http.response.status
http.url, http.scheme, http.host, http.target, http.server_name, net.host.port, host.name, net.host.name, net.peer.name, net.peer.port, net.peer.ip, server.address, client.address	http.request.url	if span.kind = "Server" then construct server URL. Otherwise contruct a client URL.
message.type	http.response.content_length	If message.type attribute key exists and value = "RECEIVED" then http.content_length is set to messaging.message_payload_size_bytes if it is set; otherwise http.content_length is set to 0.

OTel span SQL attributes translation

OTel Attributes env var value	X-Ray data model field	Notes
db.connection_string, db.name	sql.url	If db.connection_string attribute key exists and is not empty, then sql.url is created from the values of the db.connection_string and db.name key values.
db.system	sql.database_type
db.user	sql.user
db.statement	sql.sanatized_query

OTel span AWS attributes translation

OTel Attributes env var value	X-Ray data model field	Notes
cloud.account.id	aws.account_id
aws.operation, rpc.method	aws.operation	aws.operation is set to value of aws.operation attribute key if it exists. Otherwise it is set to the value of rpc.method attribute key.
aws.region	aws.region
aws.requestId	aws.request_id
aws.queue.url	aws.queue_url
aws.table.name	aws.table_name

OTel span metadata translation

OTel Attributes env var value	X-Ray data model field	Notes
telemetry.sdk.name	xray.sdk
telemetry.sdk.version	xray.sdk_version
telemetry.auto.version	xray.auto_instrumentation

OTel span EC2 metadata translation

OTel Attributes env var value	X-Ray data model field	Notes
host.id	ec2.instance_id
cloud.availability_zone	ec2.availability_zone
host.type	ec2.instance_size
host.image.id	ec2.ami_id

OTel span ECS metadata translation

OTel Attributes env var value	X-Ray data model field	Notes
container.name	ecs.container
container.id	ecs.container_id
cloud.availability_zone	ecs.availability_zone
aws.ecs.container.arn	ecs.container_arn
aws.ecs.cluster.arn	ecs.cluster_arn
aws.ecs.task.arn	ecs.task_arn
aws.ecs.task.family	ecs.task_family
aws.ecs.launchtype	ecs.launch_type

OTel span EKS metadata translation

OTel Attributes env var	X-Ray data model field	Notes
k8s.cluster.name	eks.cluster_name
k8s.pod.name	eks.pod
k8s.pod.uid	eks.container_id

OTel span EB metadata translation

OTel Attributes env var	X-Ray data model field	Notes
service.namespace	elastic_beanstalk.environment_name
service.instance.id	elastic_beanstalk.deployment_id	If AttributeCloudPlatformAWSElasticBeanstalk attribute key = "aws_elastic_beanstalk" and service.instance.id attribute key exists and is not empty, then elastic_beanstalk_deployment_id field is set to value of service.instance.id attribute key.
service.version	elastic_beanstalk.version_label

OTel span CW Logs metadata translation

The attributes in this section are translated into the AWS resource field cloudwatch_logs. This resource field is used to correlate Cloudwatch Log Groups with the segments associated with it. This correlation can be explored when analyzing individual traces in the X-Ray console.

If an attribute is not specified, then the exporter's aws_log_groups field will be used if available.

OTel Attributes env var	X-Ray data model field	Notes
aws.log.group.arns	cloudwatch_logs.arn
aws.log.group.names	cloudwatch_logs.log_group	Precedence is given in the order of `aws.log.group.arns`, `aws.log.group.names`, then the exporter config's `aws_log_groups`.

Using Config to set Cloud Watch Log Group Names

1awsxray:
2  aws_log_groups: ["group1", "group2"]

With the ADOT Collector v0.26.0 and newer, it is also possible to set the resource attributes aws.log.group.names and aws.log.group.arns using the environment variable OTEL_RESOURCE_ATTRIBUTES. This is useful for the case you are using auto-instrumentation in your application. The following is an example of value that you could use in that environment variable.

OTEL_RESOURCE_ATTRIBUTES=aws.log.group.names=group1

See AWS X-Ray Segment Documents for more information on the AWS X-Ray data model. See OpenTelemetry data model spec for more information on OpenTelemetry data model. See X-Ray Logging Configuration for references of expected log pattern. See AWS X-Ray Tracing Exporter for more information on the exporter's config.

Enable The Customized X-Ray Annotations

By using OpenTelemetry SDK, we can add the customized attributes for each trace subsegment. By default, these OpenTelemetry attributes will be converted into metadata attributes in X-Ray raw data. If we need to convert some or all OpenTelemetry attributes into X-Ray annotation, we can follow the steps below for enabling annotation conversion.

Enabling X-Ray annotations via the otel collector xray exporter config

Step 1: create an attribute by using OpenTelemetry SDK

For example:

setAttribute(“TransactionId”, <value>)
setAttribute(“AccountId”, <value>)

Note: More detailed instructions for adding attributes can be found in individual languages' guides

Step 2: In the collector, set the config in one of two ways

For specific attributes to be indexed:

1awsxray:
2  indexed_attributes: ["TransactionId", "AccountId"]

For all attributes to be indexed:

1awsxray:
2  index_all_attributes: true

Enabling X-Ray annotations via an otel sdk span attribute

We can direct X-ray to annotate customized attributes of a span via otel span attributes.

Step 1: create an attribute by using OpenTelemetry SDK

For example:

setAttribute(“TransactionId”, <value>)
setAttribute(“AccountId”, <value>)

Note: More detailed instructions for adding attributes can be found in individual languages' guides

Step 2: direct xray to annotate these attributes

For example:

setAttribute("aws.xray.annotations", List.of(“TransactionId”, “AccountId”))

*Note: More detailed instructions can be found here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awsxrayexporter

Configuring the OpenTelemetry Collector for X-Ray remote Sampling

AWS X-Ray remote sampling is supported with OpenTelemetry by configuring the OpenTelemetry Collector to proxy sampling requests to AWS X-Ray using the awsproxy extension. Check out Configuring the OpenTelemetry Collector for X-Ray remote Sampling for more details.

Sample Collector Configuration (Putting it Together)

This is a sample configuration without additional configurations in the exporter:

1receivers:
2  otlp:
3    protocols:
4      grpc:
5        endpoint: 0.0.0.0:4317
6      http:
7        endpoint: 0.0.0.0:4318
8
9exporters:
10  awsxray:
11
12processors:
13  memory_limiter:
14    limit_mib: 100
15    check_interval: 5s
16
17service:
18  pipelines:
19    traces:
20      processors:
21        - memory_limiter
22      receivers:
23        - otlp
24      exporters:
25        - awsxray

Using the ADOT SDKs

AWS Distro for OpenTelemetry currently supports applications written in Go, Java, JavaScript, .NET, and Python. Check out the Getting Started guides for the respective languages.

If the OpenTelemetry Collector is configured correctly and an application is instrumented to generate traces, we should be able see the traces on the AWS X-Ray console.

Questions, Issues, Missing Documentation

If you have questions or issues with this guide, you can submit an issue on the AWS Observability community page to let us know. This website is also open source, so you can submit a pull request directly to update this guide on GitHub.

Getting Started with the AWS X-Ray Exporter in the Collector

Getting Started with the AWS X-Ray Exporter in the Collector

Prerequisites

Configuring the OTLP Receiver

Example:

Configuring the AWS X-Ray Exporter

Example:

OTel to X-Ray data model translation behavior of AWS X-Ray Exporter

OTel span attributes translation

OTel span HTTP attributes translation

OTel span SQL attributes translation

OTel span AWS attributes translation

OTel span metadata translation

OTel span EC2 metadata translation

OTel span ECS metadata translation

OTel span EKS metadata translation

OTel span EB metadata translation

OTel span CW Logs metadata translation

Using Config to set Cloud Watch Log Group Names

Enable The Customized X-Ray Annotations

Enabling X-Ray annotations via the otel collector xray exporter config

Step 1: create an attribute by using OpenTelemetry SDK

Step 2: In the collector, set the config in one of two ways

Enabling X-Ray annotations via an otel sdk span attribute

Step 1: create an attribute by using OpenTelemetry SDK

Step 2: direct xray to annotate these attributes

Configuring the OpenTelemetry Collector for X-Ray remote Sampling

Sample Collector Configuration (Putting it Together)

Using the ADOT SDKs

Questions, Issues, Missing Documentation

On this page