AWS Open Distro for OpenTelemetry

Tracing with the AWS Distro for OpenTelemetry Go SDK

Tracing with the AWS Distro for OpenTelemetry Go SDK

Welcome to the AWS Distro for OpenTelemetry Go (ADOT Go) getting started guide. This walkthrough covers the ADOT Go components, how to configure the ADOT components for traces with OpenTelemetry Go and AWS X-Ray, as well as how to use the AWS Elastic Container Service (AWS ECS) and AWS Elastic Kubernetes Service (AWS EKS) resource detectors. Before reading this guide, you should familiarize with distributed tracing and the basics of OpenTelemetry. To learn more about getting started with OpenTelemetry Go, see the OpenTelemetry developer documentation.

Diagram



Installation and Configuration

To get started with ADOT Go, create a new directory for your project and add a new file inside called main.go. Open up your command line interface (CLI) and run the command go mod init main in the same directory, which will create a go.mod file. This file is used by Go to manage imports.

To install the necessary prerequisites, run the following command in the same directory that the go.mod file is in:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp
go get go.opentelemetry.io/contrib/propagators/aws/xray

In your main.go file, add the following imports:

1package main
2
3import (
4 "context"
5
6 "go.opentelemetry.io/otel"
7 "go.opentelemetry.io/otel/api/global"
8 "go.opentelemetry.io/otel/api/trace"
9 "go.opentelemetry.io/otel/exporters/otlp"
10 "go.opentelemetry.io/contrib/propagators/aws/xray"
11 sdktrace "go.opentelemetry.io/otel/sdk/trace"
12)

The packages above contain only the basic requirements for using OpenTelemetry Go with AWS X-Ray. The specific libraries and packages that are used will vary greatly depending on the type of application and what features you need. Check out the full list of libraries that contains instrumentation for 3rd-party Go packages and some packages from the standard library.




Creating an OpenTelemetry Protocol (OTLP) Exporter

Diagram

OpenTelemetry Go requires an exporter to send traces to a backend. Exporters allow telemetry data to be transferred either to the AWS Distro for OpenTelemetry Collector (ADOT Collector), or to a remote system or console for further analysis.

The ADOT Collector is a separate process that is designed to be a ‘sink’ for telemetry data emitted by many processes, which can then export that data to back-end systems. The base components vendor-agnostic implementation on how to receive, process and export telemetry data.

To initialize the OTLP exporter, add the following code to the file the main.go file -

1creds, err := credentials.NewClientTLSFromFile("my-cert.pem", "")
2if err != nil {
3 // Handle error here...
4}
5
6// Create new OTLP Exporter
7exporter, err := otlp.NewExporter(
8 otlp.WithTLSCredentials(creds),
9 otlp.WithAddress(<INSERT LISTEN ADDRESS HERE>),
10)
11
12if err != nil {
13 // Handle error here...
14}

This creates a new OTLP exporter with a few options - WithTLSCredentials() is a way to intialize gRPC TransportCredentials and you can learn more about it here, and WithAddress() allows you to set the address that the exporter will connect to the Collector on. If the address is unset, it will instead try to use connect to DefaultCollectorHost:DefaultCollectorPort.

If the Collector you are connecting to does not use TLS, for example in a sidecar deployment, then pass otlp.WithInsecure() instead to otlp.NewExporter.




Creating a Tracer Provider

Tracing refers to the practice used by engineers and developers to analyze application code. Traditionally, this was done through using various different debugging tools and programming techniques. As applications evolved and adopted the microservices architecture, tracing now refers to distributed tracing. This term refers to tracing a path that a request takes as it goes through different services and processes in a multi-service architecture design. To define a trace more precisely in OpenTelemetry terms, it is essentially a group of linked spans, which represent a unit of work in a request. The spans can have information attached such as a name and other data describing the timed operation. Every trace consists of at least a root span and can have zero or more child spans. The root span describes the end-to-end latency of the entire trace, while child spans representing sub-operations.

Diagram

In order to generate traces, OpenTelemetry Go requires a trace provider to be created. A Trace provider can have multiple different span processors, which are components that give the ability to modify and export span data after it has been created.

To create a new trace provider, add the following lines to the main.go file:

1cfg := sdktrace.Config{
2 DefaultSampler: sdktrace.AlwaysSample(),
3}
4
5idg := xray.NewIDGenerator()
6
7tp := sdktrace.NewTracerProvider(
8 sdktrace.WithConfig(cfg),
9 sdktrace.WithSyncer(exporter),
10 sdktrace.WithIDGenerator(idg),
11)

The block of code above will create a new TracerProvider with a Sampler that samples every trace, and an ID Generator that will generate trace IDs that conform to AWS X-Ray’s format, as well as register the OLTP exporter we created in the previous section.




Setting Global Options

In this section we will be setting a global tracer provider since its good practice when using OpenTelemetry Go. By doing this, it will make it easier for other libraries/dependencies that use the OpenTelemetry API to easily discover the SDK and emit telemetry data.

In addition to setting a global tracer provider, we will also configure the context propagation option. Context propagation refers to sharing data across multiple processes or services. This ability to correlate information across service boundaries is one of the principal concepts behind distributed tracing. To find these correlations, components in a distributed system need to be able to collect, store, and transfer metadata referred to as context. Propagator structs are configured inside Tracer structs in order to support transferring of context across process boundaries. A context will often have information identifying the current span and trace, and can contain arbitrary correlations as key-value pairs. Propagation is when context is bundled and transferred across services, often via HTTP headers.

To set up global options, we will use the global package and add the following lines to the main.go file:

1global.SetTracerProvider(tp)
2global.SetTextMapPropagator(xray.Propagator{})



Demo Walkthrough

Let’s take the concepts we’ve just covered in the previous section and create a demo application. In this application we will be creating a HTTP application with one endpoint called /hello-world. When a client hits this endpoint, the server will return a simple response containing a string saying “Hello World”. The goal of this example application is to show you how to instrument send traces to AWS X-Ray using OpenTelemetry Go.

Setting up a Router Using Mux

The first thing we need to do is create the /hello-world endpoint. To do this, we will be using a routing package in Go called Mux. To install this package, run the following command:

go get -u github.com/gorilla/mux

Now that we have Mux installed, let’s go ahead and make a new main.go file. In this file, we will first create a new router using Mux. Then, we will define the /hello-world endpoint and pass in a function called handler(), which will return “Hello World”. Finally, we will call the http.ListenAndServe() function to start our server and listen on port :8080.

The file should look something like this:

1package main
2
3import (
4 "encoding/json"
5 "net/http"
6 "github.com/gorilla/mux"
7)
8
9func main() {
10
11 // Create a new HTTP router to handle incoming client requests
12 r := mux.NewRouter()
13
14 // When client makes GET request to /hello-world
15 // handler() will execute
16 r.HandleFunc("/hello-world", handler).Methods(http.MethodGet)
17
18 // Start the server and listen on localhost:8080
19 http.ListenAndServe(":8080", r)
20}
21
22// Function for handling the /hello-world endpoint
23func handler(w http.ResponseWriter, r *http.Request) {
24
25 // Set the header content-type and return hello world
26 w.Header().Set("Content-Type", "text/plain")
27 json.NewEncoder(w).Encode("hello world")
28
29}

To start the server we will use the go run command to build and execute our main.go file. To do this, open up your command line interface and navigate to the directory that contains your main.go file. Once you’re in the directory execute the following command to run the main.go file and start the server on port 8080.

$ go run main.go

After starting the server, we can test that it works by opening up the browser and typing in localhost:8080/hello-world , you should now see “Hello World” printed to the screen!

Note: Make sure the aws-otel-collector is properly set up and running

Instrument our Application

Now that we have a simple HTTP server set up with one endpoint, it’s time to instrument our application with OpenTelemetry Go so that we can start tracing requests. To do this, we will first create a helper function called initTracer(), where we include all the instrumentation code. In this helper function, we will instantiate a new OTLP exporter struct. Then, we will create a new TracerProvider struct and pass in configuration options such as a custom ID Generator, which will be used for sending the traces to AWS X-Ray . The last part of this function is specifying the TracerProvider struct that we want to use, as well as the propagator.

The function should look something like this:

1func initTracer() {
2
3 // Create new OTLP Exporter struct
4 driver := otlpgrpc.NewDriver(
5 otlpgrpc.WithInsecure(),
6 otlpgrpc.WithEndpoint("localhost:30080"),
7 otlpgrpc.WithDialOption(grpc.WithBlock()), // useful for testing
8 )
9
10 exporter, err := otlp.NewExporter(ctx, driver)
11 if err != nil {
12 // Handle error here...
13 }
14
15 // AlwaysSample() returns a Sampler that samples every trace.
16 // Be careful about using this sampler in a production application with
17 // significant traffic: a new trace will be started and exported for every request.
18 cfg := sdktrace.Config{
19 DefaultSampler: sdktrace.AlwaysSample(),
20 }
21
22 // A custom ID Generator to generate traceIDs that conform to
23 // AWS X-Ray traceID format
24 idg := xray.NewIDGenerator()
25
26 // Create a new TraceProvider object passing in the config, the exporter
27 // and the ID Generator we want to use for our tracing
28 tp := sdktrace.NewTracerProvider(
29 sdktrace.WithConfig(cfg),
30 sdktrace.WithSyncer(exporter),
31 sdktrace.WithIDGenerator(idg),
32 )
33
34 // Set the traceprovider and the propagator we want to use
35 otel.SetTracerProvider(tp)
36 otel.SetTextMapPropagator(xray.Propagator{})
37}

Please note, if your Collector is using TLS, usually because it is deployed as a service, you will need to use otlp.WithTLSCredentials instead of otlp.WithInsecure.

After creating the initTracer() function, we can now call it in our main() function, along with creating a new tracer and giving it a name of “demo-app”. In addition to initializing a tracer, we also need to add a line to allow the ADOT Collector to act as a middleware in order to intercept requests and send those traces to AWS X-Ray.

The updated main() function should now look something like this:

1func main() {
2
3 initTracer()
4 tracer := otel.Tracer("demo-app")
5
6 // Create a new HTTP router to handle incoming client requests
7 r := mux.NewRouter()
8
9 r.Use(otelmux.Middleware("my-server"))
10
11 // When client makes GET request to /hello-world, handler() will execute
12 r.HandleFunc("/hello-world", handler).Methods(http.MethodGet)
13
14 // Start the server and listen on localhost:8080
15 http.ListenAndServe(":8080", r)
16}

Once again, we will start our server by using the following command and then visiting localhost:8080/hello-world.

$ go run main.go

Now, along with seeing “Hello World” printed to the browser, you can also log into your AWS X-Ray Console and see that there is a new trace request for the /hello-world endpoint.

Diagram

The completed main.go file should now look something like this:

1package main
2
3import (
4 "encoding/json"
5 "net/http"
6
7 "github.com/gorilla/mux"
8 "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"
9 "go.opentelemetry.io/contrib/propagators/aws/xray"
10 "go.opentelemetry.io/otel"
11 "go.opentelemetry.io/otel/exporters/otlp"
12 "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc"
13
14 sdktrace "go.opentelemetry.io/otel/sdk/trace"
15)
16
17var tracer = otel.Tracer("demo-app")
18
19func main() {
20
21 initTracer()
22
23 // Create a new HTTP router to handle incoming client requests
24 r := mux.NewRouter()
25
26 r.Use(otelmux.Middleware("my-server"))
27
28 // When client makes GET request to /hello-world
29 // handler() will execute
30 r.HandleFunc("/hello-world", handler).Methods(http.MethodGet)
31
32 // Start the server and listen on localhost:8080
33 http.ListenAndServe(":8080", r)
34
35}
36
37// Function for handling the /hello-world endpoint
38func handler(w http.ResponseWriter, r *http.Request) {
39
40 // Set the header content-type and return hello world
41 w.Header().Set("Content-Type", "application/json")
42 json.NewEncoder(w).Encode("hello world")
43
44}
45
46func initTracer() {
47
48 // Create new OTLP Exporter struct
49 driver := otlpgrpc.NewDriver(
50 otlpgrpc.WithInsecure(),
51 otlpgrpc.WithEndpoint("localhost:30080"),
52 otlpgrpc.WithDialOption(grpc.WithBlock()), // useful for testing
53 )
54
55 exporter, err := otlp.NewExporter(ctx, driver)
56 if err != nil {
57 // Handle error here...
58 }
59
60 // AlwaysSample() returns a Sampler that samples every trace.
61 // Be careful about using this sampler in a production application with
62 // significant traffic: a new trace will be started and exported for every request.
63 cfg := sdktrace.Config{
64 DefaultSampler: sdktrace.AlwaysSample(),
65 }
66
67 // A custom ID Generator to generate traceIDs that conform to
68 // AWS X-Ray traceID format
69 idg := xray.NewIDGenerator()
70
71 // Create a new TraceProvider struct passing in the config, the exporter
72 // and the ID Generator we want to use for our tracing
73 tp := sdktrace.NewTracerProvider(
74 sdktrace.WithConfig(cfg),
75 sdktrace.WithSyncer(exporter),
76 sdktrace.WithIDGenerator(idg),
77 )
78
79 // Set the traceprovider and the propagator we want to use
80 otel.SetTracerProvider(tp)
81 otel.SetTextMapPropagator(xray.Propagator{})
82}

Please note, if your Collector is using TLS, usually because it is deployed as a service, you will need to use otlp.WithTLSCredentials instead of otlp.WithInsecure.




Using the AWS ECS Resource Detector

The AWS ECS Resource Detector is responsible for detecting whether or not a Go application instrumented with OpenTelemetry is running on ECS. If the resource detector does in fact detect that it is running on ECS, it will populate the resource with ECS metadata. The metadata will include the containerId and hostName which will be formatted as key value pairs inside the resource struct. If the ECS resource detector detects that the application is not running on ECS, then it will return an empty resource struct.

Diagram

The following code snippet demonstrates how to use the ECS resource detector.

1import (
2 "context"
3 "go.opentelemetry.io/contrib/detectors/aws/ecs"
4 sdktrace "go.opentelemetry.io/otel/sdk/trace"
5)
6
7func main() {
8
9 // Instantiate a new ECS Resource detector
10 ecsResourceDetector := ecs.NewResourceDetector()
11 resource, err := ecsResourceDetector.Detect(context.Background())
12
13 //Associate resource with TracerProvider
14 tracerProvider := sdktrace.NewTracerProvider(
15 sdktrace.WithResource(resource),
16 )
17
18}



Using AWS EKS Resource Detector

The AWS EKS Resource Detector is responsible for detecting whether or not a Go application instrumented with OpenTelemetry is running on EKS. If the EKS resource detector detects that it is running on EKS, it will then populate the resource struct with metadata. The metadata will include the containerId and clusterName which will be formatted as key value pairs. If the EKS resource detector detects that the application is not running on EKS, then it will return an empty resource.

Diagram

The following code snippet demonstrates how to use the EKS Resource detector.

1import (
2 "context"
3 "go.opentelemetry.io/contrib/detectors/aws/eks"
4 sdktrace "go.opentelemetry.io/otel/sdk/trace"
5)
6
7func main() {
8
9 // Instantiate a new EKS Resource detector
10 eksResourceDetector := eks.NewResourceDetector()
11 resource, err := eksResourceDetector.Detect(context.Background())
12
13 //Associate resource with TracerProvider
14 tracerProvider := sdktrace.NewTracerProvider(
15 sdktrace.WithResource(resource),
16 )
17
18}



Conclusion

After reading this guide you should now have a basic understand of how to instrument Go applications with OpenTelemetry and send traces to AWS X-Ray. We also went through how to use the resource detectors to extract information specific to ECS and EKS environments into resource objects.