AWS Distro for OpenTelemetry

Manual Instrumentation for Traces and Metrics with the Java SDK

Manual Instrumentation for Traces and Metrics with the Java SDK

Introduction

The OpenTelemetry Java SDK can be compiled into any Java 8+ application to gather telemetry data from a diverse set of libraries and frameworks. Library instrumentation can be registered to quickly gather data on popular frameworks and the OpenTelemetry API can be used to customize tracing for your application.

For integration with X-Ray, OpenTelemetry provides extension modules for configuring the X-Ray ID generator, X-Ray propagator, and AWS resource detectors.

If you are using the Auto-Instrumentation Java Agent, refer to the documentation on auto-instrumentation.



Requirements

Java 8 (or later) is required to run an application using OpenTelemetry.

Note: You’ll also need to have the AWS Distro for OpenTelemetry (ADOT) Collector running to export traces to X-Ray.



Installation

Several components provide the functionality for using OpenTelemetry SDK with X-Ray. You must use the OpenTelemetry BOM to align dependency versions for non-contrib components.

For Gradle:
1dependencies {
2    api(platform("io.opentelemetry:opentelemetry-bom:1.32.0"))
3

4    implementation("io.opentelemetry:opentelemetry-api")
5    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
6    implementation("io.opentelemetry:opentelemetry-sdk")
7

8

9    implementation("io.opentelemetry:opentelemetry-extension-aws")
10    implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
11    implementation("io.opentelemetry.contrib:opentelemetry-aws-xray:1.32.0")
12}
For Maven:
1<dependencyManagement>
2  <dependencies>
3    <dependency>
4      <groupId>io.opentelemetry</groupId>
5      <artifactId>opentelemetry-bom</artifactId>
6      <version>1.32.0</version>
7      <type>pom</type>
8      <scope>import</scope>
9    <dependency>
10  </dependencies>
11</dependencyManagement>
12<dependencies>
13  <dependency>
14    <groupId>io.opentelemetry</groupId>
15    <artifactId>opentelemetry-api</artifactId>
16  </dependency>
17  <dependency>
18    <groupId>io.opentelemetry</groupId>
19    <artifactId>opentelemetry-exporter-otlp</artifactId>
20  </dependency>
21  <dependency>
22    <groupId>io.opentelemetry</groupId>
23    <artifactId>opentelemetry-sdk</artifactId>
24  </dependency>
25  <dependency>
26    <groupId>io.opentelemetry</groupId>
27    <artifactId>opentelemetry-extension-aws</artifactId>
28  </dependency>
29  <dependency>
30    <groupId>io.opentelemetry</groupId>
31    <artifactId>opentelemetry-sdk-extension-aws</artifactId>
32  </dependency>
33  <dependency>
34    <groupId>io.opentelemetry.contrib</groupId>
35    <artifactId>opentelemetry-aws-xray</artifactId>
36    <version>1.32.0</version>
37  </dependency>
38</dependencies>


Setting up the SDK

Sending Traces to AWS X-Ray

Initialize the OpenTelemetry SDK with AWS components for exporting to X-Ray as follows.

OpenTelemetrySdk.builder()


        // This will enable your downstream requests to include the X-Ray trace header
        .setPropagators(
            ContextPropagators.create(
                TextMapPropagator.composite(
                    W3CTraceContextPropagator.getInstance(), AwsXrayPropagator.getInstance())))


        // This provides basic configuration of a TracerProvider which generates X-Ray compliant IDs
        .setTracerProvider(
            SdkTracerProvider.builder()
                .addSpanProcessor(
                    BatchSpanProcessor.builder(OtlpGrpcSpanExporter.getDefault()).build())
                .setIdGenerator(AwsXrayIdGenerator.getInstance())
                .build())
        .buildAndRegisterGlobal();

Using the AWS resource detectors

AWS resource detectors for enriching traces with AWS infrastructure information is available in the opentelemetry-sdk-extension-aws artifact.

For Gradle:
1dependencies {
2    implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
3}
For Maven:
1<dependencies>
2  <dependency>
3    <groupId>io.opentelemetry</groupId>
4    <artifactId>opentelemetry-sdk-extension-aws</artifactId>
5  </dependency>
6</dependencies>

Register the detectors you would like to use when initializing the SDK.

OpenTelemetrySdk.builder()
        ...
        .setTracerProvider(
            SdkTracerProvider.builder()
                ...
                .setResource(
                    Resource.getDefault()
                        .merge(BeanstalkResource.get())
                        .merge(Ec2Resource.get())
                        .merge(EcsResource.get()
                        .merge(EksResource.get())))
                .build())
        .buildAndRegisterGlobal();

Adding support for Metrics

The API and SDK for Metrics became stable in v1.15.0 of OpenTelemetry for Java. The following piece of code initialize the OpenTelemetry SDK to use Metrics and Traces.

MetricReader metricReader = PeriodicMetricReader.builder(
        OtlpGrpcMetricExporter.getDefault())
        .build();
OpenTelemetry opentelemetry = OpenTelemetrySdk.builder()
        // Traces configuration
        .setPropagators(
            ContextPropagators.create(
                TextMapPropagator.composite(
                    W3CTraceContextPropagator.getInstance(), AwsXrayPropagator.getInstance())))


        .setTracerProvider(
            SdkTracerProvider.builder()
                .addSpanProcessor(
                    BatchSpanProcessor.builder(OtlpGrpcSpanExporter.getDefault()).build())
                .setIdGenerator(AwsXrayIdGenerator.getInstance())
                .build()
        // Metrics Configuration
        .setMeterProvider(
                SdkMeterProvider.builder()
                        .registerMetricReader(metricReader)
                        .build())
        .buildAndRegisterGlobal();

Debug Logging

The SDK uses java.util.logging to log messages at FINE level - logging frameworks like Logback or Log4J map this to debug level. To view debug statements, configure your logging framework to output io.opentelemetry with debug level.



Instrumenting an application

OpenTelemetry provides a wide range of instrumentations for popular Java libraries such as Spring, gRPC, OkHttp, and JDBC. Instrumenting a library means that every time the library is used to make or handle a request is automatically wrapped with a populated span.

View the full list of instrumented libraries.

Note that library instrumentation is currently alpha and some APIs may change before a stable release. You must use the io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha BOM to manage versions when adding library instrumentation. When using this, do not include opentelemetry-bom.

For Gradle:
1dependencies {
2    api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.32.1-alpha"))
3

4    implementation("io.opentelemetry:opentelemetry-api")
5    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
6    implementation("io.opentelemetry:opentelemetry-sdk")
7

8    implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-<framework>")
9

10    ...
11}
For Maven:
1<dependencyManagement>
2  <dependencies>
3    <dependency>
4      <groupId>io.opentelemetry.instrumentation</groupId>
5      <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
6      <version>1.32.1-alpha</version>
7      <type>pom</type>
8      <scope>import</scope>
9    <dependency>
10  </dependencies>
11</dependencyManagement>
12<dependencies>
13  <dependency>
14    <groupId>io.opentelemetry</groupId>
15    <artifactId>opentelemetry-api</artifactId>
16  </dependency>
17  <dependency>
18    <groupId>io.opentelemetry</groupId>
19    <artifactId>opentelemetry-exporter-otlp</artifactId>
20  </dependency>
21  <dependency>
22    <groupId>io.opentelemetry</groupId>
23    <artifactId>opentelemetry-sdk</artifactId>
24  </dependency>
25  <dependency>
26    <groupId>io.opentelemetry.instrumentation</groupId>
27    <artifactId>opentelemetry-instrumentation-<framework></artifactId>
28  </dependency>
29  ...
30</dependencies>

Instrumenting the AWS SDK

The opentelemetry-instrumentation-aws-sdk-2.2 artifact provides instrumentation for the AWS SDK v2.

For Gradle:
1dependencies {
2    api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.32.1-alpha"))\
3

4    implementation("io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2")
5

6    ...
7}
For Maven:
1<dependencyManagement>
2  <dependencies>
3    <dependency>
4      <groupId>io.opentelemetry.instrumentation</groupId>
5      <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
6      <version>1.32.1-alpha</version>
7      <type>pom</type>
8      <scope>import</scope>
9    <dependency>
10  </dependencies>
11</dependencyManagement>
12<dependencies>
13  <dependency>
14    <groupId>io.opentelemetry.instrumentation</groupId>
15    <artifactId>opentelemetry-instrumentation-aws-sdk-2.2</artifactId>
16  </dependency>
17  ...
18</dependencies>

And when initializing an AWS SDK, add the ExecutionInterceptor which enables tracing.

1DynamoDbClient.builder()
2  .overrideConfiguration(ClientOverrideConfiguration.builder()
3    .addExecutionInterceptor(AwsSdkTracing.create(openTelemetry).newExecutionInterceptor())
4    .build())
5  .build();

This will enable tracing for all DynamoDB calls using this client.

Using X-Ray Remote Sampling

The opentelemetry-aws-xray artifact provides a Sampler implementation for use with X-Ray remote sampling.

When initializing the OpenTelemetry SDK, register the AwsXrayRemoteSampler. Moreover, You can configure the following attributes.

AttributeTypeDescriptionDefault
pollingIntervalDurationDuration between polling the GetSamplingRules API5 minutes
endpointstringEndpoint used to communicate with the awsproxy collector extensionhttp://localhost:2000
1Resource resource = Resource.builder()
2  ...
3  .build();
4

5OpenTelemetrySdk.builder()
6  .setTracerProvider(SdkTracerProvider.builder()
7    .setResource(resource)
8    .setSampler(AwsXrayRemoteSampler.newBuilder(resource).setEndpoint("http://localhost:2000")
9                                                                      .setPollingInterval(Duration.ofSeconds(300))
10                                                                      .build())
11    ...
12    .build())
13  .build();

You will also need to configure the OpenTelemetry collector to allow the application to fetch sampling configuration.



Custom Instrumentation

Creating Custom Spans

You can use custom spans to monitor the performance of internal activities that are not captured by instrumentation libraries. Note that only spans of kind Server are converted into X-Ray segments, all other spans are converted into X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.

First, create a Tracer to associate with generated spans. It is common to have one Tracer for the entire application, often available via dependency injection.

Tracer tracer = openTelemetry.getTracer("my-app");

Then to create spans:

// SERVER span will become an X-Ray segment
Span span = tracer.spanBuilder("get-token")
  .setKind(SpanKind.SERVER)
  .setAttribute(USER_ID, "user")
  .startSpan();
try (Scope ignored = span.makeCurrent()) {
  doGetToken();
}
// Default span of type INTERNAL will become an X-Ray subsegment
Span span = tracer.spanBuilder("process-header")
  .startSpan();
try (Scope ignored = span.makeCurrent()) {
  doProcessHeader();
}

Adding custom attributes

You can also add custom key-value pairs as attributes onto your spans. Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray Developer Guide.

class RequestHandler {
  // Not storing AttributeKey as a constant will result in significantly degraded performance.
  private static final AttributeKey<String> USER_ID_KEY = AttributeKey.stringKey("user.id");


  Response handle(Request request) {
    // Library instrumentation, for example for Spring, has already created a span for this request. We access it with
    // Span.current() and can add any attributes we define ourselves.
    Span.current().setAttribute(USER_ID_KEY, request.getUserId());
  }
}

Creating Metrics

Similarly to Traces, you can create custom metrics in your application using the OpenTelemetry API and SDK.

In the following example application we demonstrate how to use the three types of metric instruments that are available to record metrics: Counters, Gauges and Histograms.

The theoretic application being depicted is a worker that process messages from 2 different queues.

Counters:

Meter meter = opentelemetry.getMeter("consumer-application");


LongCounter counter = meter.counterBuilder("messages_consumed")
        .setDescription("Number of messages consumed")
        .setUnit("n")
        .build();


Attributes attributes1 = Attributes.of(AttributeKey.stringKey("processing_place"), "Place1");
Attributes attributes2 = Attributes.of(AttributeKey.stringKey("processing_place"), "Place2");


// Counters can be synchronous
counter.record(getProcessedMessagesQueue1(), attributes1);


// Different attributes can be associated with the value
counter.record(getProcessedMessagesQueue2(), attributes2);


// Counters also have the asynchronous form
LongCounter messagesDroppedCounter = meter.counterBuilder("messages_dropped")
        .setDescription("Number of messages dropped")
        .buildWithCallback( (consumer) -> consumer.record(getTotalMessagesDropped()));

Gauges:

Meter meter = opentelemetry.getMeter("consumer-application");


Attributes attributes1 = Attributes.of(AttributeKey.stringKey("queue_name"), "Queue1");
Attributes attributes2 = Attributes.of(AttributeKey.stringKey("queue_name"), "Queue2");


Gauge gauge = meter
    .gaugeBuilder("consumer_queue_size")
    .setDescription("The size of the queue that is being consumed")
    .setUnit("1")
    .ofLongs()
    // Gauges are asynchronous
    .buildWithCallback(
        measurement -> {
          measurement.record(getQueueSize1(), attributes1);
          measurement.record(getQueueSize2(), attributes2);
        });

Histograms:

Meter meter = opentelemetry.getMeter("consumer-application");


// Histograms metric data points convey a population of recorded measurements in a compressed format.
// A histogram bundles a set of events into divided populations with an overall event count and aggregate sum for all events.
// Histograms are useful to record measurements such as latency. With histograms we can extract the min, max and percentiles.
LongHistogram histogram = meter.histogramBuilder("processing_time")
      .setUnit("ms")
      .setDescription("Amount of time it takes to process a message")
      .ofLongs()
      .build();


histogram.record(messageProcessingTime)

There are more examples in the OpenTelemetry Java Manual.

On this page