AWS Open Distro for OpenTelemetry

Tracing with the AWS Distro for OpenTelemetry Python Auto-Instrumentation and X-Ray

Tracing with the AWS Distro for OpenTelemetry Python Auto-Instrumentation and X-Ray

Introduction

OpenTelemetry Python supports automatic instrumentation. It automatically produces spans with telemetry data describing the values used by the python frameworks in your application without adding a single line of code. This telemetry data can then be exported to a backend like AWS X-Ray using the ADOT Python opentelemetry-sdk-extension-aws package. We also strongly recommend using the opentelemetry-propagator-aws-xray package to support propagating the trace context across AWS services. This propagator handles the extraction and injecting of the AWS X-Ray Tracing header for requests from or to remote services.

In this guide, we walk through the steps needed to trace an application with auto-instrumentation.




Requirements

Python 3.6 or later is required to run an application using OpenTelemetry.

Note: You’ll also need to have the ADOT Collector running to export traces to X-Ray.




Installation

The easiest way to download the packages needed for auto-instrumentation is using pip:

# Install required packages for instrumentation and to support tracing with AWS X-Ray
$ pip install opentelemetry-distro[otlp]>=0.24b0 \
opentelemetry-sdk-extension-aws~=2.0 \
opentelemetry-propagator-aws-xray~=1.0

The opentelemetry-distro package provides methods which configure the OpenTelemetry SDK with some basic defaults. These methods are used by Auto Instrumentation. Because you added the [otlp] "extra" command, the opentelemetry-exporter-otlp package (used to send traces to the ADOT Collector) is also automatically installed.

Installing the opentelemetry-sdk-extension-aws package automatically installs the opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation packages as dependencies. You also need the opentelemetry-propagator-aws-xray package to obtain the AwsXRayPropagator class used to propagate the trace context across AWS services.

opentelemetry-instrumentation provides commands to detect, install, and initialize all instrumentation packages supported for your application’s dependencies. Notably, it installs the opentelemetry-bootstrap and opentelemetry-instrument executables on your system.

Go to the directory of the python application which you want to instrument. Here, use the opentelemetry-bootstrap command to automatically detect and install OpenTelemetry python packages. These packages contain Instrumentors that will instrument the packages your system has downloaded and that your application is already using.

# Automatically install supported Instrumentors for the application's dependencies
$ opentelemetry-bootstrap --action=install

For example, if you have boto3 installed, this command will automatically install the opentelemetry-instrumentation-botocore package which auto-instrumentation can subsequently configure automatically. Check out the OpenTelemetry registry for a full list of instrumentation packages provided by OpenTelemetry Python.

Running an Application with Auto-Instrumentation

Auto-instrumentation uses the opentelemetry-instrument executable functions as a wrapper to automatically initialize the Instrumentors installed by the opentelemetry-bootstrap command and start the provided application.

The AWS X-Ray Id Generator can be configured using an environment variable as OTEL_PYTHON_ID_GENERATOR=xray, and the AWS X-Ray Propagator can be configured using OTEL_PROPAGATORS=xray.

Currently, it is not possible to configure the Resource Detectors using auto-instrumentation.

Putting this all together, starting your application using auto-instrumentation can be as simple as the following:

$ OTEL_PROPAGATORS=xray \
OTEL_PYTHON_ID_GENERATOR=xray \
opentelemetry-instrument python3 ./path/to/your/app.py

Configuring Auto-Instrumentation

Environment variables are the primary way in which the OpenTelemetry SDK for Python is configured to enable compatibility with the AWS X-Ray backend. Some key environment variables are:

  • OTEL_PYTHON_ID_GENERATOR
  • OTEL_PROPAGATORS
  • OTEL_TRACES_EXPORTER
  • OTEL_EXPORTER_OTLP_ENDPOINT
  • OTEL_EXPORTER_OTLP_CERTIFICATE

The IdGenerator can be configured to use the AWS X-Ray Id Generator with an environment variable as OTEL_PYTHON_ID_GENERATOR=xray to ensure spans use an Id format compatible with the AWS X-Ray backend.

The global propagator can be configured to use the AWS X-Ray Propagator with an environment variable as OTEL_PROPAGATORS=xray to allow the span context to propagate downstream when the application makes calls to external services.

The SpanExporter can be configured with an environment variables OTEL_TRACES_EXPORTER=otlp to export spans in the format required by the ADOT Collector. However, if opentelemetry-distro[otlp] is used, it already uses the otlp exporter by default without the need for any more configuration.

The configuration of your SDK exporter depends on how you have configured your ADOT Collector. To learn more about how the ADOT Collector can be configured, refer to the ADOT Collector Documentation.

We can use the OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 environment variable to set the address that the exporter will use to connect to the collector. If unset, the SDK will try to connect to http://localhost:4317 by default. Note that because the scheme is http by default, you have to explicitly set it to be https if necessary.

If the Collector the application will connect to is running with TLS configured, the OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/my-cert.crt environment variable is used to give a path to credentials to be used to establish a secure connection for the app’s exporter. The credentials at this path should be the public certificate of the collector, or one of its root certificates. If no certificate is found, the gRPC method ssl_channel_credentials() will attempt to “retrieve the PEM-encoded root certificates from a default location chosen by gRPC runtime” as explained in the gRPC Python Documentation.

Thus, an advanced configuration of auto-instrumentation may look like this:

$ OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/my-cert.crt \
OTEL_EXPORTER_OTLP_ENDPOINT=collector.service.local \
OTEL_PROPAGATORS=xray \
OTEL_PYTHON_ID_GENERATOR=xray \
opentelemetry-instrument python3 ./path/to/your/app.py



Using Manual Instrumentation

Because there can only be one global TracerProvider, manual instrumentation should not instantiate its own TracerProvider if used together alongside auto-instrumentation. Given that the same TracerProvider is used, custom tracing works the same way when using automatic instrumentation or manual instrumentation. For information about custom trace instrumentation, see our docs on manual instrumentation.




Sample Application

See a sample Flask App using OpenTelemetry Python SDK Automatic Instrumentation.

NOTE: Python Web Frameworks like Flask and Django normally include a "reloader" when running in debug mode so that you can apply changes during development. This reloader will break auto-instrumentation because the app is restarted without giving OpenTelemetry a chance to wrap the instrumented libraries. When using debug mode, set the use_reloader=False as is done in the referenced sample app:

# See more: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/546
if __name__ == "__main__":
app.run(port=8082, debug=True, use_reloader=False)