This blog post will help new and existing customers get started with Dynatrace support for OpenTelemetry. Learn how to send OpenTelemetry data to Dynatrace from an OTel veteran and Dynatrace newbie.

One of the things I love most about OpenTelemetry (OTel) is that it’s vendor-neutral, which means you can send the same OpenTelemetry data to different vendors. In fact, most of the major Observability vendors out there not only support ingesting OpenTelemetry data but also actively contribute to the project, including Dynatrace. Check out the 2023 OpenTelemetry Journey Report for more info.

Why does this matter? I used to work at another Observability vendor, and many of the OpenTelemetry examples that I played with and blogged about in the last 2 years or so featured sending OTel data to that backend.

Now that I work at Dynatrace, which, by the way, ingests OTLP natively, I wanted to educate myself on how to send OpenTelemetry data to Dynatrace. A great way to learn is to try to run my go-to examples using Dynatrace as the Observability backend. Luckily for me, since OTel is vendor-neutral, all I had to do was reconfigure my OTel Collector to point to Dynatrace to get my examples to work.

Want to learn how? Let’s do it together!

Note: If you’re evaluating multiple vendors, you can send the same data to different vendors at the same time (à la “vendor bake-off”) to help you determine which vendor best suits your organization’s needs.

Prerequisites for sending data to Dynatrace

To send OpenTelemetry data to Dynatrace, you need two pieces of information:

Dynatrace tenant: Each user (or, more likely, organization) is assigned a tenant. When sending OpenTelemetry data from your application to Dynatrace, you need to specify the Dynatrace OTLP endpoint (used by Dynatrace to receive data), which includes your tenant name.

Access token: The access token allows you to send OTel data to your Dynatrace instance. It also specifies what kind of data you’re allowed to send to Dynatrace. You can find more on Dynatrace access tokens here.

Before we get to any of that, you first need a Dynatrace account. If you already have a Dynatrace account, feel free to skip the following section.

Create a Dynatrace account

If you don’t have a Dynatrace account, you can create a free trial account, which is valid for 15 days.

1. Go here, and select the Free trial button.
   

   Free trial signup button at dynatrace.com

2. Enter your email, select the Terms of Use checkbox, and then select Continue.

   Enter your email and accept the Terms of Use

3. Enter the rest of the info and select Start free trial.
   

   Fill in the rest of the form fields

   You will receive an email once your account has been created. You will also see a page that looks like the one below. Select Launch Dynatrace to get started.

   Your Dynatrace tenant is ready!

   This takes you to the Dynatrace login page.

   Dynatrace login page

Your Dynatrace tenant

To find your Dynatrace tenant, log into Dynatrace here, and select the Login button at the top right of the page.

This takes you to the sign-in page. Once you sign in, take note of the URL. It should look something like this:

https://<your-dynatrace-tenant>.apps.dynatrace.com

Take note of the value of <your-dynatrace-tenant>, because we’ll need that later.

Create a Dynatrace access token

After confirming that you’re logged into Dynatrace, press ctrl+k, and then type access token. Next, select Access Tokens from the top of the search results.

Access token search

On the Access tokens page, select Generate new token.

Dynatrace Access tokens page

On the Generate new token page, enter:

• Token name: be sure to give it a descriptive name
• Expiration date: this is optional
• Template: Kubernetes Data Ingest

Even if we’re not necessarily using Kubernetes, the Kubernetes Data Ingest template has the token scopes (permissions) that we need to send OpenTelemetry data to Dynatrace, namely:

• Ingest logs (ingest)
• Ingest metrics (ingest)
• Ingest OpenTelemetry traces (ingest)

Find more information on these and other Dynatrace token scopes here.

Once you’re done, select Generate token at the bottom of the page.

Access token generation page

The next page shows your access token. Be sure to copy and store it somewhere for safekeeping (for example, a secrets manager such as HashiCorp Vault or your cloud provider’s secrets manager) before selecting Done, because after that, it’s gone forever. If you lose that token information, you should delete the old one (not necessary, but highly recommended), and create a new one.

Access token page showing generated token

Configure the OTel Collector for Dynatrace

Now that you have your tenant info and access token, you can plug this information into your OpenTelemetry Collector configuration.

Note: There are two ways to send OTel data to an Observability backend: (1) direct from the application, or (2) via the OTel Collector. There’s a time and place for each, and you can check out my blog post on OTel Collector Anti-patterns on the OTel Blog to learn more.

Your OTel Collector config YAML file should look something like this:

receivers: otlp: protocols: grpc: http: processors: cumulativetodelta: batch: exporters: otlphttp: endpoint: "https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp" headers: Authorization: "Api-Token ${DT_TOKEN}" debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp,debug] metrics: receivers: [otlp] processors: [cumulativetodelta] exporters: [otlphttp,debug] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp,debug]

Dynatrace accepts data in the native OpenTelemetry Protocol (OTLP) format via HTTP (gRPC is not yet supported). You need to specify the Dynatrace OTLP endpoint (used by Dynatrace to receive data), which includes your tenant name, ${DT_TENANT}:

https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp

${DT_TENANT} is the value of your Dynatrace access token, which you hopefully jotted down and stored in a secrets manager for safekeeping.

Finally, “Dynatrace requires metrics data to be sent with delta temporality, not cumulative temporality”. This means that you’ll need to include the cumulativetodelta processor in:

• Your Collector configuration (cumulativetodelta)
• Your metrics pipeline (pipelines.metrics)

Never store your Dynatrace token and tenant name in plain text. Instead, store them in a secrets manager and pull them from the secrets manager at runtime.

Dynatrace and the OTel Operator

If you’re using the OpenTelemetry Operator to send OpenTelemetry data to Dynatrace, you’ll need to configure your OpenTelemetryCollector resource as follows:

apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otelcol namespace: opentelemetry spec: mode: statefulset image: ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:0.7.0 env: - name: DT_TOKEN valueFrom: secretKeyRef: key: DT_TOKEN name: otel-collector-secret - name: DT_TENANT valueFrom: secretKeyRef: key: DT_TENANT name: otel-collector-secret config: receivers: otlp: protocols: grpc: {} http: {} processors: cumulativetodelta: {} batch: {} exporters: otlphttp: endpoint: "https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp" headers: Authorization: "Api-Token ${DT_TOKEN}" debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp,debug] metrics: receivers: [otlp] processors: [cumulativetodelta] exporters: [otlphttp,debug] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp,debug]

Notice that the spec.config looks the same as what we defined in the otelcol-config.yaml file we saw earlier. The only added thing here is that DT_TOKEN and DT_TENANT are environment variables pulled from a Kubernetes Secret. The secret YAML definition looks like this:

apiVersion: v1 kind: Secret metadata: name: otel-collector-secret namespace: data: DT_TOKEN: DT_TENANT: type: "Opaque"

Both DT_TOKEN and DT_TENANT values must be base64 encoded before being added to the secrets YAML. To base64 encode a value and copy the encoded value to your buffer, use this action for easy copy/paste:

echo <value_to_encode> | base64

Remember that storing secrets in Kubernetes (or storing a secrets YAML in version control, for that matter), is not recommended because base64 does not encrypt your data. You should instead consider using sealed secrets or the Kubernetes Secrets Store CSI Driver + your favorite secrets provider. For more info on these better alternatives check out this article.

The Dynatrace OTel Collector Distribution

Many vendors have their own OTel Collector Distributions. These distributions are curated with Collector components that are specific to that vendor. They can be a combination of vendor-developed custom components and components from Collector Core and Contrib. Using vendor-specific distributions ensures you’re using just the Collector components you need, reducing overall bloat. You can learn more here.

Dynatrace also has its own Collector distribution. It features a set of Collector components for sending Observability data to Dynatrace from various sources. It stays up-to-date with upstream components of the opentelemetry-collector and opentelemetry-collector-contrib repositories.

In addition, the Dynatrace Collector distribution offers the following advantages:

• It is covered by Dynatrace support
• Collector components are verified by Dynatrace
• Security patches are independent of OpenTelemetry Collector releases

Try it out!

Want to try sending data to Dynatrace yourself? Then check out my example repo. I created this repo for a talk on the Target Allocator that I gave at KubeCon in March 2024. It has been updated to include instructions on configuring the OpenTelemetryCollector resource to send OTel data to Dynatrace.

OTel Data in Dynatrace

And if you’re curious to see what OTel data looks like in Dynatrace, here are some screenshots of the web UI.

I’m not going in-depth on how to navigate the Dynatrace UI because there are already some great videos on the Dynatrace YouTube channel. I encourage you to check them out for a more in-depth look.

Dynatrace Distributed Tracing UI

Dynatrace Logs UI

Dynatrace Notebooks UI showing a metric called ”some_counter_total”

Final thoughts

As someone with experience sending OpenTelemetry data to various backends, I found that getting OpenTelemetry data into Dynatrace was fairly straightforward. My only personal hiccup was in generating the application token, but I got that sorted out, and now I’ve passed on my knowledge and highly detailed screenshots along to you.

I have to say that it’s always fun to use a product with fresh eyes, a fresh perspective, and a newbie point of view. There’s nothing quite like it. And, having worked at another observability vendor before, it’s always fun to see the similarities and differences. It’s like learning a new programming language and comparing it to another one that you already know. What a blast!

One final point that I want to make. I don’t want to trivialize things and give you the impression that moving from one observability vendor to another is simply a matter of repointing your OTel Collector from one vendor backend to another. That is only one aspect of a vendor migration, no matter what vendor you’re moving to/from. You also must consider the fact that you’ll likely have a bunch of dashboards, alerts, and whatnot that you created with your original vendor. When you migrate to another vendor, there won’t be a 1:1 translation; so keep that in mind.

But that may be a sacrifice that you’re willing to make because OTel’s vendor neutrality means that all vendors supporting OpenTelemetry are ingesting the same data. What sets them apart is what they do with your data. And if one vendor does something with your data better than another, well, don’t you owe it to yourself to check that out?

What’s Next?

If you’re interested in learning first-hand what Dynatrace and OpenTelemetry can do together, then dive into Dynatrace! You can do so in one of two ways:

• Check out the Dynatrace Playground to explore Dynatrace using pre-populated OpenTelemetry data
• Get started with a free trial and ingest your own OpenTelemetry data today!