AWS Open Distro for OpenTelemetry

Tracing with the AWS Distro for OpenTelemetry Java SDK and X-Ray

Tracing with the AWS Distro for OpenTelemetry Java SDK and X-Ray

Introduction

The OpenTelemetry Java SDK can be compiled into any Java 8+ application to gather telemetry data from a diverse set of libraries and frameworks. Library instrumentation can be registered to quickly gather data on popular frameworks and the OpenTelemetry API can be used to customize tracing for your application.

For integration with X-Ray, OpenTelemetry provides extension modules for configuring the X-Ray ID generator, X-Ray propagator, and AWS resource detectors.

If you are using the Auto-Instrumentation Java Agent, refer to the documentation on auto-instrumentation.




Requirements

Java 8 (or later) is required to run an application using OpenTelemetry.

Note: You’ll also need to have the AWS Distro for OpenTelemetry (ADOT) Collector running to export traces to X-Ray.




Installation

Several components provide the functionality for using OpenTelemetry SDK with X-Ray. You must use the OpenTelemetry BOM to align dependency versions for non-contrib components.

For Gradle:
1dependencies {
2 api(platform("io.opentelemetry:opentelemetry-bom:1.6.0"))
3
4 implementation("io.opentelemetry:opentelemetry-api")
5 implementation("io.opentelemetry:opentelemetry-exporter-otlp")
6 implementation("io.opentelemetry:opentelemetry-sdk")
7
8
9 implementation("io.opentelemetry:opentelemetry-extension-aws")
10 implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
11 implementation("io.opentelemetry.contrib:opentelemetry-aws-xray:1.6.0")
12}
For Maven:
1<dependencyManagement>
2 <dependencies>
3 <dependency>
4 <groupId>io.opentelemetry</groupId>
5 <artifactId>opentelemetry-bom</artifactId>
6 <version>1.6.0</version>
7 <type>pom</type>
8 <scope>import</scope>
9 <dependency>
10 </dependencies>
11</dependencyManagement>
12<dependencies>
13 <dependency>
14 <groupId>io.opentelemetry</groupId>
15 <artifactId>opentelemetry-api</artifactId>
16 </dependency>
17 <dependency>
18 <groupId>io.opentelemetry</groupId>
19 <artifactId>opentelemetry-exporter-otlp</artifactId>
20 </dependency>
21 <dependency>
22 <groupId>io.opentelemetry</groupId>
23 <artifactId>opentelemetry-sdk</artifactId>
24 </dependency>
25 <dependency>
26 <groupId>io.opentelemetry</groupId>
27 <artifactId>opentelemetry-extension-aws</artifactId>
28 </dependency>
29 <dependency>
30 <groupId>io.opentelemetry</groupId>
31 <artifactId>opentelemetry-sdk-extension-aws</artifactId>
32 </dependency>
33 <dependency>
34 <groupId>io.opentelemetry.contrib</groupId>
35 <artifactId>opentelemetry-aws-xray</artifactId>
36 <version>1.6.0</version>
37 </dependency>
38</dependencies>



Setting up the SDK

Sending Traces to AWS X-Ray

Initialize the OpenTelemetry SDK with AWS components for exporting to X-Ray as follows.

OpenTelemetrySdk.builder()
// This will enable your downstream requests to include the X-Ray trace header
.setPropagators(
ContextPropagators.create(
TextMapPropagator.composite(
W3CTraceContextPropagator.getInstance(), AwsXrayPropagator.getInstance())))
// This provides basic configuration of a TracerProvider which generates X-Ray compliant IDs
.setTracerProvider(
SdkTracerProvider.builder()
.addSpanProcessor(
BatchSpanProcessor.builder(OtlpGrpcSpanExporter.getDefault()).build())
.setIdGenerator(AwsXrayIdGenerator.getInstance())
.build())
.buildAndRegisterGlobal();

Using the AWS resource detectors

AWS resource detectors for enriching traces with AWS infrastructure information is available in the opentelemetry-sdk-extension-aws artifact.

For Gradle:
1dependencies {
2 implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
3}
For Maven:
1<dependencies>
2 <dependency>
3 <groupId>io.opentelemetry</groupId>
4 <artifactId>opentelemetry-sdk-extension-aws</artifactId>
5 </dependency>
6</dependencies>

Register the detectors you would like to use when initializing the SDK.

OpenTelemetrySdk.builder()
...
.setTracerProvider(
SdkTracerProvider.builder()
...
.setResource(
Resource.getDefault()
.merge(BeanstalkResource.get())
.merge(Ec2Resource.get())
.merge(EcsResource.get()
.merge(EksResource.get())))
.build())
.buildAndRegisterGlobal();

Debug Logging

The SDK uses java.util.logging to log messages at FINE level - logging frameworks like Logback or Log4J map this to debug level. To view debug statements, configure your logging framework to output io.opentelemetry with debug level.




Instrumenting an application

OpenTelemetry provides a wide range of instrumentations for popular Java libraries such as Spring, gRPC, OkHttp, and JDBC. Instrumenting a library means that every time the library is used to make or handle a request is automatically wrapped with a populated span.

View the full list of instrumented libraries.

Note that library instrumentation is currently alpha and some APIs may change before a stable release. You must use the io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha BOM to manage versions when adding library instrumentation. When using this, do not include opentelemetry-bom.

For Gradle:
1dependencies {
2 api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.6.0-alpha"))
3
4 implementation("io.opentelemetry:opentelemetry-api")
5 implementation("io.opentelemetry:opentelemetry-exporter-otlp")
6 implementation("io.opentelemetry:opentelemetry-sdk")
7
8 implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-<framework>")
9
10 ...
11}
For Maven:
1<dependencyManagement>
2 <dependencies>
3 <dependency>
4 <groupId>io.opentelemetry.instrumentation</groupId>
5 <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
6 <version>1.6.0</version>
7 <type>pom</type>
8 <scope>import</scope>
9 <dependency>
10 </dependencies>
11</dependencyManagement>
12<dependencies>
13 <dependency>
14 <groupId>io.opentelemetry</groupId>
15 <artifactId>opentelemetry-api</artifactId>
16 </dependency>
17 <dependency>
18 <groupId>io.opentelemetry</groupId>
19 <artifactId>opentelemetry-exporter-otlp</artifactId>
20 </dependency>
21 <dependency>
22 <groupId>io.opentelemetry</groupId>
23 <artifactId>opentelemetry-sdk</artifactId>
24 </dependency>
25 <dependency>
26 <groupId>io.opentelemetry.instrumentation</groupId>
27 <artifactId>opentelemetry-instrumentation-<framework></artifactId>
28 </dependency>
29 ...
30</dependencies>

Instrumenting the AWS SDK

The opentelemetry-instrumentation-aws-sdk-2.2 artifact provides instrumentation for the AWS SDK v2.

For Gradle:
1dependencies {
2 api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.5.0-alpha"))\
3
4 implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-aws-sdk-2.2")
5
6 ...
7}
For Maven:
1<dependencyManagement>
2 <dependencies>
3 <dependency>
4 <groupId>io.opentelemetry.instrumentation</groupId>
5 <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
6 <version>1.5.0</version>
7 <type>pom</type>
8 <scope>import</scope>
9 <dependency>
10 </dependencies>
11</dependencyManagement>
12<dependencies>
13 <dependency>
14 <groupId>io.opentelemetry.instrumentation</groupId>
15 <artifactId>opentelemetry-instrumentation-aws-sdk-2.2</artifactId>
16 </dependency>
17 ...
18</dependencies>

And when initializing an AWS SDK, add the ExecutionInterceptor which enables tracing.

1DynamoDbClient.builder()
2 .overrideConfiguration(ClientOverrideConfiguration.builder()
3 .addExecutionInterceptor(AwsSdkTracing.create(openTelemetry).newExecutionInterceptor())
4 .build())
5 .build();

This will enable tracing for all DynamoDB calls using this client.

Using X-Ray Remote Sampling

The opentelemetry-aws-xray artifact provides a Sampler implementation for use with X-Ray remote sampling.

When initializing the OpenTelemetry SDK, register the AwsXrayRemoteSampler. Moreover, You can configure the following attributes.

AttributeTypeDescriptionDefault
pollingIntervalDurationDuration between polling the GetSamplingRules API5 minutes
endpointstringEndpoint used to communicate with the awsproxy collector extensionhttp://localhost:2000
1Resource resource = Resource.builder()
2 ...
3 .build();
4
5OpenTelemetrySdk.builder()
6 .setTracerProvider(SdkTracerProvider.builder()
7 .setResource(resource)
8 .setSampler(AwsXrayRemoteSampler.newBuilder(resource).setEndpoint("http://localhost:2000")
9 .setPollingInterval(Duration.ofSeconds(300))
10 .build())
11 ...
12 .build())
13 .build();

You will also need to configure the OpenTelemetry collector to allow the application to fetch sampling configuration.




Custom Instrumentation

Creating Custom Spans

You can use custom spans to monitor the performance of internal activities that are not captured by instrumentation libraries. Note that only spans of kind Server are converted into X-Ray segments, all other spans are converted into X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.

First, create a Tracer to associate with generated spans. It is common to have one Tracer for the entire application, often available via dependency injection.

Tracer tracer = openTelemetry.getTracer("my-app");

Then to create spans:

// SERVER span will become an X-Ray segment
Span span = tracer.spanBuilder("get-token")
.setKind(SpanKind.SERVER)
.setAttribute(USER_ID, "user")
.startSpan();
try (Scope ignored = span.makeCurrent()) {
doGetToken();
}
// Default span of type INTERNAL will become an X-Ray subsegment
Span span = tracer.spanBuilder("process-header")
.startSpan();
try (Scope ignored = span.makeCurrent()) {
doProcessHeader();
}

Adding custom attributes

You can also add custom key-value pairs as attributes onto your spans. Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray Developer Guide.

class RequestHandler {
// Not storing AttributeKey as a constant will result in significantly degraded performance.
private static final AttributeKey<String> USER_ID_KEY = AttributeKey.stringKey("user.id");
Response handle(Request request) {
// Library instrumentation, for example for Spring, has already created a span for this request. We access it with
// Span.current() and can add any attributes we define ourselves.
Span.current().setAttribute(USER_ID_KEY, request.getUserId());
}
}