AWS Distro for OpenTelemetry
Manual Instrumentation for Traces and Metrics with the Java SDK
Manual Instrumentation for Traces and Metrics with the Java SDK
Introduction
The OpenTelemetry Java SDK can be compiled into any Java 8+ application to gather telemetry data from a diverse set of libraries and frameworks. Library instrumentation can be registered to quickly gather data on popular frameworks and the OpenTelemetry API can be used to customize tracing for your application.
For integration with X-Ray, OpenTelemetry provides extension modules for configuring the X-Ray ID generator, X-Ray propagator, and AWS resource detectors.
If you are using the Auto-Instrumentation Java Agent, refer to the documentation on auto-instrumentation.
Requirements
Java 8 (or later) is required to run an application using OpenTelemetry.
Note: You’ll also need to have the AWS Distro for OpenTelemetry (ADOT) Collector running to export traces to X-Ray.
Installation
Several components provide the functionality for using OpenTelemetry SDK with X-Ray. You must use the OpenTelemetry BOM to align dependency versions for non-contrib components.
For Gradle:
1dependencies {2 api(platform("io.opentelemetry:opentelemetry-bom:1.32.0"))3
4 implementation("io.opentelemetry:opentelemetry-api")5 implementation("io.opentelemetry:opentelemetry-exporter-otlp")6 implementation("io.opentelemetry:opentelemetry-sdk")7
8
9 implementation("io.opentelemetry:opentelemetry-extension-aws")10 implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")11 implementation("io.opentelemetry.contrib:opentelemetry-aws-xray:1.32.0")12}
For Maven:
1<dependencyManagement>2 <dependencies>3 <dependency>4 <groupId>io.opentelemetry</groupId>5 <artifactId>opentelemetry-bom</artifactId>6 <version>1.32.0</version>7 <type>pom</type>8 <scope>import</scope>9 <dependency>10 </dependencies>11</dependencyManagement>12<dependencies>13 <dependency>14 <groupId>io.opentelemetry</groupId>15 <artifactId>opentelemetry-api</artifactId>16 </dependency>17 <dependency>18 <groupId>io.opentelemetry</groupId>19 <artifactId>opentelemetry-exporter-otlp</artifactId>20 </dependency>21 <dependency>22 <groupId>io.opentelemetry</groupId>23 <artifactId>opentelemetry-sdk</artifactId>24 </dependency>25 <dependency>26 <groupId>io.opentelemetry</groupId>27 <artifactId>opentelemetry-extension-aws</artifactId>28 </dependency>29 <dependency>30 <groupId>io.opentelemetry</groupId>31 <artifactId>opentelemetry-sdk-extension-aws</artifactId>32 </dependency>33 <dependency>34 <groupId>io.opentelemetry.contrib</groupId>35 <artifactId>opentelemetry-aws-xray</artifactId>36 <version>1.32.0</version>37 </dependency>38</dependencies>
Setting up the SDK
Sending Traces to AWS X-Ray
Initialize the OpenTelemetry SDK with AWS components for exporting to X-Ray as follows.
OpenTelemetrySdk.builder()
// This will enable your downstream requests to include the X-Ray trace header .setPropagators( ContextPropagators.create( TextMapPropagator.composite( W3CTraceContextPropagator.getInstance(), AwsXrayPropagator.getInstance())))
// This provides basic configuration of a TracerProvider which generates X-Ray compliant IDs .setTracerProvider( SdkTracerProvider.builder() .addSpanProcessor( BatchSpanProcessor.builder(OtlpGrpcSpanExporter.getDefault()).build()) .setIdGenerator(AwsXrayIdGenerator.getInstance()) .build()) .buildAndRegisterGlobal();
Using the AWS resource detectors
AWS resource detectors for enriching traces with AWS infrastructure information is available in the opentelemetry-sdk-extension-aws
artifact.
For Gradle:
1dependencies {2 implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")3}
For Maven:
1<dependencies>2 <dependency>3 <groupId>io.opentelemetry</groupId>4 <artifactId>opentelemetry-sdk-extension-aws</artifactId>5 </dependency>6</dependencies>
Register the detectors you would like to use when initializing the SDK.
OpenTelemetrySdk.builder() ... .setTracerProvider( SdkTracerProvider.builder() ... .setResource( Resource.getDefault() .merge(BeanstalkResource.get()) .merge(Ec2Resource.get()) .merge(EcsResource.get() .merge(EksResource.get()))) .build()) .buildAndRegisterGlobal();
Adding support for Metrics
The API and SDK for Metrics became stable in v1.15.0 of OpenTelemetry for Java. The following piece of code initialize the OpenTelemetry SDK to use Metrics and Traces.
MetricReader metricReader = PeriodicMetricReader.builder( OtlpGrpcMetricExporter.getDefault()) .build();OpenTelemetry opentelemetry = OpenTelemetrySdk.builder() // Traces configuration .setPropagators( ContextPropagators.create( TextMapPropagator.composite( W3CTraceContextPropagator.getInstance(), AwsXrayPropagator.getInstance())))
.setTracerProvider( SdkTracerProvider.builder() .addSpanProcessor( BatchSpanProcessor.builder(OtlpGrpcSpanExporter.getDefault()).build()) .setIdGenerator(AwsXrayIdGenerator.getInstance()) .build() // Metrics Configuration .setMeterProvider( SdkMeterProvider.builder() .registerMetricReader(metricReader) .build()) .buildAndRegisterGlobal();
Debug Logging
The SDK uses java.util.logging
to log messages at FINE
level - logging frameworks like Logback or Log4J map this to
debug
level. To view debug statements, configure your logging framework to output io.opentelemetry
with debug
level.
Instrumenting an application
OpenTelemetry provides a wide range of instrumentations for popular Java libraries such as Spring, gRPC, OkHttp, and JDBC. Instrumenting a library means that every time the library is used to make or handle a request is automatically wrapped with a populated span.
View the full list of instrumented libraries.
Note that library instrumentation is currently alpha
and some APIs may change before a stable release. You must use
the io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
BOM to manage versions when adding
library instrumentation. When using this, do not include opentelemetry-bom
.
For Gradle:
1dependencies {2 api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.32.1-alpha"))3
4 implementation("io.opentelemetry:opentelemetry-api")5 implementation("io.opentelemetry:opentelemetry-exporter-otlp")6 implementation("io.opentelemetry:opentelemetry-sdk")7
8 implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-<framework>")9
10 ...11}
For Maven:
1<dependencyManagement>2 <dependencies>3 <dependency>4 <groupId>io.opentelemetry.instrumentation</groupId>5 <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>6 <version>1.32.1-alpha</version>7 <type>pom</type>8 <scope>import</scope>9 <dependency>10 </dependencies>11</dependencyManagement>12<dependencies>13 <dependency>14 <groupId>io.opentelemetry</groupId>15 <artifactId>opentelemetry-api</artifactId>16 </dependency>17 <dependency>18 <groupId>io.opentelemetry</groupId>19 <artifactId>opentelemetry-exporter-otlp</artifactId>20 </dependency>21 <dependency>22 <groupId>io.opentelemetry</groupId>23 <artifactId>opentelemetry-sdk</artifactId>24 </dependency>25 <dependency>26 <groupId>io.opentelemetry.instrumentation</groupId>27 <artifactId>opentelemetry-instrumentation-<framework></artifactId>28 </dependency>29 ...30</dependencies>
Instrumenting the AWS SDK
The opentelemetry-instrumentation-aws-sdk-2.2
artifact provides instrumentation for the AWS SDK v2.
For Gradle:
1dependencies {2 api(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.32.1-alpha"))\3
4 implementation("io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2")5
6 ...7}
For Maven:
1<dependencyManagement>2 <dependencies>3 <dependency>4 <groupId>io.opentelemetry.instrumentation</groupId>5 <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>6 <version>1.32.1-alpha</version>7 <type>pom</type>8 <scope>import</scope>9 <dependency>10 </dependencies>11</dependencyManagement>12<dependencies>13 <dependency>14 <groupId>io.opentelemetry.instrumentation</groupId>15 <artifactId>opentelemetry-instrumentation-aws-sdk-2.2</artifactId>16 </dependency>17 ...18</dependencies>
And when initializing an AWS SDK, add the ExecutionInterceptor
which enables tracing.
1DynamoDbClient.builder()2 .overrideConfiguration(ClientOverrideConfiguration.builder()3 .addExecutionInterceptor(AwsSdkTracing.create(openTelemetry).newExecutionInterceptor())4 .build())5 .build();
This will enable tracing for all DynamoDB calls using this client.
Using X-Ray Remote Sampling
The opentelemetry-aws-xray
artifact provides a Sampler
implementation for use with X-Ray remote sampling.
When initializing the OpenTelemetry SDK, register the AwsXrayRemoteSampler
. Moreover, You can configure the following attributes.
Attribute | Type | Description | Default |
---|---|---|---|
pollingInterval | Duration | Duration between polling the GetSamplingRules API | 5 minutes |
endpoint | string | Endpoint used to communicate with the awsproxy collector extension | http://localhost:2000 |
1Resource resource = Resource.builder()2 ...3 .build();4
5OpenTelemetrySdk.builder()6 .setTracerProvider(SdkTracerProvider.builder()7 .setResource(resource)8 .setSampler(AwsXrayRemoteSampler.newBuilder(resource).setEndpoint("http://localhost:2000")9 .setPollingInterval(Duration.ofSeconds(300))10 .build())11 ...12 .build())13 .build();
You will also need to configure the OpenTelemetry collector to allow the application to fetch sampling configuration.
Custom Instrumentation
Creating Custom Spans
You can use custom spans to monitor the performance of internal activities that are not captured by instrumentation
libraries. Note that only spans of kind Server
are converted into X-Ray segments, all other spans are converted into
X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.
First, create a Tracer
to associate with generated spans. It is common to have one Tracer
for the entire application,
often available via dependency injection.
Tracer tracer = openTelemetry.getTracer("my-app");
Then to create spans:
// SERVER span will become an X-Ray segmentSpan span = tracer.spanBuilder("get-token") .setKind(SpanKind.SERVER) .setAttribute(USER_ID, "user") .startSpan();try (Scope ignored = span.makeCurrent()) { doGetToken();}
// Default span of type INTERNAL will become an X-Ray subsegmentSpan span = tracer.spanBuilder("process-header") .startSpan();try (Scope ignored = span.makeCurrent()) { doProcessHeader();}
Adding custom attributes
You can also add custom key-value pairs as attributes onto your spans. Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray Developer Guide.
class RequestHandler { // Not storing AttributeKey as a constant will result in significantly degraded performance. private static final AttributeKey<String> USER_ID_KEY = AttributeKey.stringKey("user.id");
Response handle(Request request) { // Library instrumentation, for example for Spring, has already created a span for this request. We access it with // Span.current() and can add any attributes we define ourselves. Span.current().setAttribute(USER_ID_KEY, request.getUserId()); }}
Creating Metrics
Similarly to Traces, you can create custom metrics in your application using the OpenTelemetry API and SDK.
In the following example application we demonstrate how to use the three types of metric instruments that are available to record metrics: Counters, Gauges and Histograms.
The theoretic application being depicted is a worker that process messages from 2 different queues.
Meter meter = opentelemetry.getMeter("consumer-application");
LongCounter counter = meter.counterBuilder("messages_consumed") .setDescription("Number of messages consumed") .setUnit("n") .build();
Attributes attributes1 = Attributes.of(AttributeKey.stringKey("processing_place"), "Place1");Attributes attributes2 = Attributes.of(AttributeKey.stringKey("processing_place"), "Place2");
// Counters can be synchronouscounter.record(getProcessedMessagesQueue1(), attributes1);
// Different attributes can be associated with the valuecounter.record(getProcessedMessagesQueue2(), attributes2);
// Counters also have the asynchronous formLongCounter messagesDroppedCounter = meter.counterBuilder("messages_dropped") .setDescription("Number of messages dropped") .buildWithCallback( (consumer) -> consumer.record(getTotalMessagesDropped()));
Meter meter = opentelemetry.getMeter("consumer-application");
Attributes attributes1 = Attributes.of(AttributeKey.stringKey("queue_name"), "Queue1");Attributes attributes2 = Attributes.of(AttributeKey.stringKey("queue_name"), "Queue2");
Gauge gauge = meter .gaugeBuilder("consumer_queue_size") .setDescription("The size of the queue that is being consumed") .setUnit("1") .ofLongs() // Gauges are asynchronous .buildWithCallback( measurement -> { measurement.record(getQueueSize1(), attributes1); measurement.record(getQueueSize2(), attributes2); });
Meter meter = opentelemetry.getMeter("consumer-application");
// Histograms metric data points convey a population of recorded measurements in a compressed format.// A histogram bundles a set of events into divided populations with an overall event count and aggregate sum for all events.// Histograms are useful to record measurements such as latency. With histograms we can extract the min, max and percentiles.LongHistogram histogram = meter.histogramBuilder("processing_time") .setUnit("ms") .setDescription("Amount of time it takes to process a message") .ofLongs() .build();
histogram.record(messageProcessingTime)
There are more examples in the OpenTelemetry Java Manual.