AWS Distro for OpenTelemetry
Tracing and Metrics with the AWS Distro for OpenTelemetry Python Auto-Instrumentation
Tracing and Metrics with the AWS Distro for OpenTelemetry Python Auto-Instrumentation
Introduction
OpenTelemetry Python supports automatic instrumentation. It automatically produces spans with telemetry data describing the values used by the python frameworks in your application without adding a single line of code. This telemetry data can then be exported to a backend like AWS X-Ray using the ADOT Python opentelemetry-sdk-extension-aws package. We also strongly recommend using the opentelemetry-propagator-aws-xray package to support propagating the trace context across AWS services. This propagator handles the extraction and injecting of the AWS X-Ray Tracing header for requests from or to remote services.
In this guide, we walk through the steps needed to trace an application and produce metrics with auto-instrumentation.
Requirements
Python 3.6 or later is required to run an application using OpenTelemetry.
Note: You’ll also need to have the ADOT Collector running to export traces and metrics.
Installation
The easiest way to download the packages needed for auto-instrumentation is using pip:
# Install required packages for instrumentation and to support tracing with AWS X-Ray$ pip install opentelemetry-distro[otlp]>=0.24b0 \ opentelemetry-sdk-extension-aws~=2.0 \ opentelemetry-propagator-aws-xray~=1.0The opentelemetry-distro package provides methods which configure the OpenTelemetry SDK with some basic defaults. These methods are used by Auto Instrumentation. Because you added the [otlp] "extra" command, the opentelemetry-exporter-otlp package (used to send traces to the ADOT Collector) is also automatically installed.
Installing the opentelemetry-sdk-extension-aws package automatically installs the opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation packages as dependencies. You also need the opentelemetry-propagator-aws-xray package to obtain the AwsXRayPropagator class used to propagate the trace context across AWS services.
opentelemetry-instrumentation provides commands to detect, install, and initialize all instrumentation packages supported for your application’s dependencies. Notably, it installs the opentelemetry-bootstrap and opentelemetry-instrument executables on your system.
Go to the directory of the python application which you want to instrument. Here, use the opentelemetry-bootstrap command to automatically detect and install OpenTelemetry python packages. These packages contain Instrumentors that will instrument the packages your system has downloaded and that your application is already using.
# Automatically install supported Instrumentors for the application's dependencies$ opentelemetry-bootstrap --action=installFor example, if you have boto3 installed, this command will automatically install the opentelemetry-instrumentation-botocore package which auto-instrumentation can subsequently configure automatically. Check out the OpenTelemetry registry for a full list of instrumentation packages provided by OpenTelemetry Python.
Running an Application with Auto-Instrumentation
Auto-instrumentation uses the opentelemetry-instrument executable functions as a wrapper to automatically initialize the Instrumentors installed by the opentelemetry-bootstrap command and start the provided application.
The AWS X-Ray Id Generator can be configured using an environment variable as OTEL_PYTHON_ID_GENERATOR=xray, and the AWS X-Ray Propagator can be configured using OTEL_PROPAGATORS=xray.
Currently, it is not possible to configure the Resource Detectors using auto-instrumentation.
Putting this all together, starting your application using auto-instrumentation can be as simple as the following:
$ OTEL_PROPAGATORS=xray \OTEL_PYTHON_ID_GENERATOR=xray \opentelemetry-instrument python3 ./path/to/your/app.pyConfiguring Auto-Instrumentation
Environment variables are the primary way in which the OpenTelemetry SDK for Python is configured to enable compatibility with the AWS X-Ray backend. Some key environment variables are:
OTEL_PYTHON_ID_GENERATOROTEL_PROPAGATORSOTEL_TRACES_EXPORTEROTEL_EXPORTER_OTLP_ENDPOINTOTEL_EXPORTER_OTLP_CERTIFICATE
The IdGenerator can be configured to use the AWS X-Ray Id Generator with an environment variable as OTEL_PYTHON_ID_GENERATOR=xray to ensure spans use an Id format compatible with the AWS X-Ray backend.
The global propagator can be configured to use the AWS X-Ray Propagator with an environment variable as OTEL_PROPAGATORS=xray to allow the span context to propagate downstream when the application makes calls to external services.
The SpanExporter can be configured with an environment variables OTEL_TRACES_EXPORTER=otlp to export spans in the format required by the ADOT Collector. However, if opentelemetry-distro[otlp] is used, it already uses the otlp exporter by default without the need for any more configuration.
The configuration of your SDK exporter depends on how you have configured your ADOT Collector. To learn more about how the ADOT Collector can be configured, refer to the ADOT Collector Documentation.
We can use the OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 environment variable to set the address that the exporter will use to connect to the collector. If unset, the SDK will try to connect to http://localhost:4317 by default. Note that because the scheme is http by default, you have to explicitly set it to be https if necessary.
If the Collector the application will connect to is running with TLS configured, the OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/my-cert.crt environment variable is used to give a path to credentials to be used to establish a secure connection for the app’s exporter. The credentials at this path should be the public certificate of the collector, or one of its root certificates. If no certificate is found, the gRPC method ssl_channel_credentials() will attempt to “retrieve the PEM-encoded root certificates from a default location chosen by gRPC runtime” as explained in the gRPC Python Documentation.
Thus, an advanced configuration of auto-instrumentation may look like this:
$ OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/my-cert.crt \OTEL_EXPORTER_OTLP_ENDPOINT=collector.service.local \OTEL_PROPAGATORS=xray \OTEL_PYTHON_ID_GENERATOR=xray \opentelemetry-instrument python3 ./path/to/your/app.pyCreating Metrics
Similarly to Traces, you can create custom metrics in your application using the OpenTelemetry API and SDK.
In the following example application we demonstrate how to use metric instruments to record metrics with a Counter.
1meter = metrics.get_meter(__name__)2time_alive_counter = meter.create_counter(3 name="time_alive",4 description="Total amount of time that the application has been alive",5 unit='ms'6 )7while True:8 time_alive_counter.add(1, attributes={'a': '1'})9 time.Sleep(1)Using Manual Instrumentation
Because there can only be one global TracerProvider and MeterProvider, manual instrumentation should not instantiate its own TracerProvider or MeterProvider if used together alongside auto-instrumentation. Given that the same TracerProvider and MeterProvider is used, custom tracing and metrics works the same way when using automatic instrumentation or manual instrumentation. For information about custom trace instrumentation, see our docs on manual instrumentation.
Sample Application
See a Sample App using OpenTelemetry Python SDK Automatic Instrumentation.
NOTE: Python Web Frameworks like Flask and Django normally include a "reloader" when running in debug mode so that you can apply changes during development. This reloader will break auto-instrumentation because the app is restarted without giving OpenTelemetry a chance to wrap the instrumented libraries. When using debug mode, set the use_reloader=False as is done in the referenced sample app:
# See more: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/546if __name__ == "__main__": app.run(port=8082, debug=True, use_reloader=False)