Tracing and Metrics with the AWS Distro for OpenTelemetry Python Manual-Instrumentation

Introduction

With OpenTelemetry Python manual instrumentation, you configure the OpenTelemetry SDK within your application's code. It automatically produces spans with telemetry data describing the values used by the Python frameworks in your application with only a few lines of code. This telemetry data can then be exported to a backend like AWS X-Ray using the ADOT Python opentelemetry-sdk-extension-aws package. We also strongly recommend using the opentelemetry-propagator-aws-xray package to support propagating the trace context across AWS services. This propagator handles the extraction and injecting of the AWS X-Ray Tracing header for requests from or to remote services.

In this guide, we walk through the steps needed to trace an application and produce metrics with manual instrumentation and produce metrics.

Requirements

Python 3.7 or later is required to run an application using OpenTelemetry.

Note: You’ll also need to have the ADOT Collector running to export traces and metrics.

Installation

Install the following packages and their dependencies from OpenTelemetry Python using pip.

Copy
$ pip install opentelemetry-sdk==1.11.1 \
              opentelemetry-sdk-extension-aws~=2.0 \
              opentelemetry-propagator-aws-xray~=1.0 \
              opentelemetry-exporter-otlp==1.11.1 \

OpenTelemetry Python distributes many packages, which provide instrumentation for well-known Python dependencies. You need to install the relevant instrumentation package for every dependency you want to generate traces for. To see supported frameworks and libraries, check out the OpenTelemetry Registry.

For example, use pip to install the follow instrumentation libraries:

Copy
# Supported instrumentation packages for the dependencies of the example above
$ pip install opentelemetry-instrumentation-flask==0.30b1 \
              opentelemetry-instrumentation-requests==0.30b1

Setting up the Global Tracer and Meter

Sending Traces and Metrics

As soon as possible in your application code, add imports for the OpenTelemetry packages installed above.

Copy
1# Basic packages for your application
2import boto3
3from flask import Flask
4import json
5
6# Add imports for OTel components into the application
7from opentelemetry import trace, metrics
8from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
9from opentelemetry.metrics import CallbackOptions, Observation
10from opentelemetry.sdk.trace import TracerProvider
11from opentelemetry.sdk.trace.export import BatchSpanProcessor
12from opentelemetry.sdk.metrics import MeterProvider
13from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
14from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
15
16# Import the AWS X-Ray for OTel Python IDs Generator into the application.
17from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

Next, configure the Global Tracer Provider and Meter Provider to export to the ADOT Collector. The configuration of your SDK exporter depends on how you wish to connect with your configured ADOT Collector.

Connecting to an ADOT Collector running as a sidecar, we can set up the TracerProvider as follows:

Copy
1# Sends generated traces in the OTLP format to an ADOT Collector running on port 4317
2otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
3# Processes traces in batches as opposed to immediately one after the other
4span_processor = BatchSpanProcessor(otlp_exporter)
5# Configures the Global Tracer Provider
6trace.set_tracer_provider(TracerProvider(active_span_processor=span_processor, id_generator=AwsXRayIdGenerator()))
7# Setting up Metrics
8metric_reader = PeriodicExportingMetricReader(exporter=OTLPMetricExporter())
9metric_provider = MeterProvider(metric_readers=[metric_reader])
10metrics.set_meter_provider(metric_provider)

The endpoint= argument allows you to set the address that the exporter will use to connect to the collector. If unset, the SDK will try to connect to http://localhost:4317 by default. Note that because the scheme is http by default, you have to explicitly set it to be https if necessary.

If the Collector the application will connect to is running without TLS configured, the http scheme is used to disable client transport security for our OTLP exporter’s connection. This will use the gRPC insecure_channel() method as explained in the gRPC Python Documentation. This option should never be used in production, non-sidecar deployments.

If the Collector the application will connect to is running with TLS configured, the https scheme and the credentials=/path/to/cert.pem argument should be used to give a path to credentials that allow the application to establish a secure connection for the app’s exporter. The credentials at this path should be the public certificate of the collector, or one of its root certificates. If no certificate is found, the gRPC method ssl_channel_credentials() will attempt to “retrieve the PEM-encoded root certificates from a default location chosen by gRPC runtime” as explained in the gRPC Python Documentation.

Instead of setting the IdGenerator of the TracerProvider in code, you can also set the IdGenerator using the OTEL_PYTHON_ID_GENERATOR environment variable:

Copy
OTEL_PYTHON_ID_GENERATOR=xray

To allow the span context to propagate downstream when the application makes calls to external services, configure the global propagator to use the AWS X-Ray Propagator, which is found in the opentelemetry-propagator-aws-xray package. You can set the global propagator in code, and should configure the propagator as soon as possible in your application's code.

Copy
1from opentelemetry import propagate
2from opentelemetry.propagators.aws import AwsXRayPropagator
3propagate.set_global_textmap(AwsXRayPropagator())

Alternatively, set the OTEL_PROPAGATORS environment variable to achieve the same result.

Copy
OTEL_PROPAGATORS=xray

Using the AWS resource Detectors

When you install opentelemetry-sdk-extension-aws, you automatically get AWS Resource Detectors in the same package. Use the provided Resource Detectors to automatically populate attributes under the resource namespace of each generated span.

The ADOT Python SDK supports automatically recording metadata in EC2, Elastic Beanstalk, ECS, and EKS environments.

For example, if tracing with OpenTelemetry on an Amazon EC2 instance, you can automatically populate resource attributes by creating a TraceProvider using the AwsEc2ResourceDetector:

Copy
1import opentelemetry.trace as trace
2from opentelemetry.sdk.trace import TracerProvider
3from opentelemetry.sdk.extension.aws.resource.ec2 import (
4    AwsEc2ResourceDetector,
5)
6from opentelemetry.sdk.resources import get_aggregated_resources
7
8trace.set_tracer_provider(
9    TracerProvider(
10        resource=get_aggregated_resources(
11            [
12                AwsEc2ResourceDetector(),
13            ]
14        ),
15    )
16)

To see what attributes are captured and how to add other resource detectors, refer to each detectors' docstring in the OpenTelemetry SDK Extension for AWS to determine any requirements for that detector.

Debug Logging

You can expose better debug logging by modifying the log level for the OpenTelemetry packages your application is using.

Copy
1import logging
2
3logging.basicConfig(
4    format="%(asctime)s %(levelname)-8s %(message)s",
5    level=logging.DEBUG,
6    datefmt="%Y-%m-%d %H:%M:%S",
7)

Additionally, you can provide your own logger that uses the log level you set above.

Copy
1logger = logging.getLogger(__file__)
2
3logger.debug("My debug level log.")

Instrumenting an Application

Warning: Some instrumentations are not yet stable and the attributes they collect are subject to change until the instrumentation reaches 1.0 stability. It is recommended to pin a specific version of an instrumentation

OpenTelemetry provides a wide range of instrumentations for popular python libraries such as Flask, Django, Redis, MySQL, PyMongo and many more. Instrumenting a library means that every time the library is used to make or handle a request, that library call is automatically wrapped with a populated span contain the relevant values that were used. Web framework, downstream HTTP, SQL, gRPC, and other requests can all be recorded using OpenTelemetry.

A full list of supported instrumentation packages and configuration instructions can be found on the OpenTelemetry Python Contrib repo.

To enable tracing of the calls made by your package dependencies, you need to import and initialize the relevant Instrumentor classes. Instrumentors have individual initialization requirements, so refer to the Instrumentor’s package documentation for configuration details.

Copy
1from opentelemetry.instrumentation.requests import RequestsInstrumentor
2from opentelemetry.instrumentation.flask import FlaskInstrumentor
3
4# Initialize `Instrumentor` for the `requests` library
5RequestsInstrumentor().instrument()
6# Initialize `Instrumentor` for the `flask` web framework
7FlaskInstrumentor().instrument_app(app)

Instrumenting the AWS SDK

To install the instrumentation library for the AWS SDK and its dependencies, run the pip install command from below which applies to your application. NOTE: Since these instrumentations are not yet stable, we recommend installing it at a pinned version.

For instrumenting the boto (AWS SDK V2) package:

Copy
$ pip install opentelemetry-instrumentation-boto==0.30b1

For instrumenting the boto3 (AWS SDK V3) package (which depends on the botocore package):

Copy
$ pip install opentelemetry-instrumentation-botocore==0.30b1

Instrumenting the AWS SDK is as easy as configuring the BotoInstrumentor or BotocoreInstrumentor class. This should be done as soon as possible in your application so that subsequent calls using the SDK are wrapped by OpenTelemetry. This give OpenTelemetry the chance to record relevant information used by the SDK at the time of your application's call and export the information as spans.

For instrumenting the boto package:

Copy
1from opentelemetry.instrumentation.boto import BotoInstrumentor
2
3# Initialize `Instrumentor` for the `boto` library
4BotoInstrumentor().instrument()

For instrumenting the boto3 package:

Copy
1from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
2
3# Initialize `Instrumentor` for the `botocore` library
4BotocoreInstrumentor().instrument()

For more information refer to the upstream documentation for OpenTelemetry Python boto Instrumentation or OpenTelemetry Python botocore Instrumentation.

Custom Instrumentation

Creating Custom Spans

You can use custom spans to monitor the performance of internal activities that are not captured by instrumentation libraries. Note that only spans of kind Server are converted into X-Ray segments, all other spans are converted into X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.

Copy
1import boto3
2import json
3from opentelemetry import trace
4
5# Get a tracer from the Global Tracer Provider
6tracer = trace.get_tracer(__name__)
7
8with tracer.start_as_current_span("Root Span", kind=trace.SpanKind.SERVER):
9    print('Started a root span')
10
11    # This 'Child Span' will become an X-Ray subsegment.
12    with tracer.start_span("Child Span"):
13
14        print('Started a child span')
15
16        ec2_client = boto3.client('ec2')
17        result = ec2_client.describe_instances()
18
19        print('EC2 Describe Instances: ', json.dumps(result, default=str, indent=4))
20
21        return '<h1>Good job! Traces recorded!</h1>'

Adding custom attributes

You can also add custom key-value pairs as attributes onto your spans. Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray Developer Guide.

One way to add custom attributes is as follows:

Copy
1from opentelemetry import trace
2
3# Get a tracer from the Global Tracer Provider
4tracer = trace.get_tracer(__name__)
5
6with tracer.start_as_current_span(
7    "Root Span",
8    kind=trace.SpanKind.SERVER) as span:
9
10    print('Started a root span')
11
12    span.set_attribute("my_attribute", "foo")

Alternatively, you can do the following:

Copy
1from opentelemetry import trace
2
3# Get a tracer from the Global Tracer Provider
4tracer = trace.get_tracer(__name__)
5
6with tracer.start_as_current_span(
7    "Root Span",
8    kind=trace.SpanKind.SERVER) as span:
9
10    print('Started a root span')
11
12    current_span = trace.get_current_span()
13    current_span.set_attribute("my_attribute", "foo")

Creating Metrics

Similarly to Traces, you can create custom metrics in your application using the OpenTelemetry API and SDK.

In the following example application we demonstrate how to use metric instruments to record metrics with a Counter.

Copy
1meter = metrics.get_meter(__name__)
2time_alive_counter = meter.create_counter(
3            name="time_alive",
4            description="Total amount of time that the application has been alive",
5            unit='ms'
6        )
7while True:
8    time_alive_counter.add(1, attributes={'a': '1'})
9    time.Sleep(1)

Sample Application

See a Sample App using OpenTelemetry Python SDK Manual Instrumentation.