Analyze logs with Dynatrace Davis AI Engine utilizing Amazon Kinesis Knowledge Firehose HTTP endpoint supply

This weblog publish is co-authored with Erick Leon, Sr. Technical Alliance Supervisor from Dynatrace.

Amazon Kinesis Knowledge Firehose is the best technique to reliably load streaming knowledge into knowledge lakes, knowledge shops, and analytics providers. With just some clicks, you may create fully-managed supply streams that auto scale on demand to match the throughput of your knowledge. Prospects already use Kinesis Knowledge Firehose to ingest uncooked knowledge from varied knowledge sources, together with logs from AWS providers. Kinesis Knowledge Firehose now helps delivering streaming knowledge to Dynatrace. Dynatrace begins analyzing incoming knowledge inside minutes of Amazon CloudWatch knowledge technology.

Beginning at the moment, you should utilize Kinesis Knowledge Firehose to ship CloudWatch Metrics and Logs on to the Dynatrace observability platform to carry out your explorations and evaluation. Dynatrace, an AWS Associate Community (APN) has offered full observability into AWS Providers by ingesting CloudWatch metrics which can be printed by AWS providers. Dynatrace ingests this knowledge to carry out root-cause evaluation utilizing the Dynatrace Davis AI engine.

On this publish, we describe the Kinesis Knowledge Firehose and associated Dynatrace integration.

Stipulations

For this walkthrough, you need to have the next conditions:

• AWS account.
• Entry to the CloudWatch and Kinesis Knowledge Firehose with permissions to handle HTTP endpoints.
• Dynatrace Clever Observability Platform account, or get a free 15 day trial right here.
• Dynatrace model 1.182+.
• An up to date AWS monitoring coverage to incorporate the extra AWS providers.
  To replace the AWS Identification and Entry Administration (IAM) coverage, use the JSON within the hyperlink above, containing the monitoring coverage (permissions) for all supporting providers.
• Dynatrace API token: create token with the next permission and preserve available in a notepad.

Determine 1 – Dynatrace API Token

The way it works

Determine 2 – Amazon Kinesis Knowledge Firehose HTTP endpoint supply

Merely create a log stream on your Amazon providers to ship your context wealthy logs to the Amazon CloudWatch Logs service. Subsequent, choose your Dynatrace HTTP endpoint to reinforce your logs streams with the ability of the Dynatrace Intelligence Platform. Lastly, you may as well again up your logs to an Amazon Easy Storage Service (Amazon S3) bucket.

Setup directions

So as to add a service to monitoring, comply with these steps:

1. Within the Dynatrace menu, go to Settings > Cloud and virtualization, and choose AWS.
2. On the AWS overview web page, scroll down and choose the specified AWS occasion. Choose the Edit button.
3. Scroll down and choose Add service. Select the service identify from the drop-down, and choose Add service.
4. Choose Save modifications.

To course of and ship AWS CloudWatch Metrics to Dynatrace, comply with these steps.

1. Log in to the AWS console and kind “Kinesis” within the textual content search bar. Choose Kinesis

Determine 3 – AWS Console

2. On the Amazon Kinesis providers web page, choose the radio button for Kinesis Knowledge Firehose and choose the Create supply stream button.

Determine 4 – Amazon Kinesis

3. Select the “Direct PUT” from the drop down, and from Vacation spot drop down, select “Dynatrace”.

Determine 5 – Amazon Kinesis Knowledge Firehose

4. Supply stream identify – Give your stream a brand new identify, for instance: – “KFH-StreamToDynatrace”

Determine 6 – Supply stream identify

5. Within the part “Vacation spot settings”:

Determine 7 – Vacation spot settings

6. HTTP endpoint identify – “Dynatrace”.
7. HTTP endpoint URL – From the drop down, choose “Dynatrace – US”.
8. API token – Enter Dynatrace API TOKEN created within the prerequisite part.
9. API URL – enter the Dynatrace URL on your tenant, for instance: https://xxxxx.stay.dynatrace.com
10. Again Up Settings – Both choose an current S3 bucket or create a brand new one and add particulars and choose the Create supply stream button.

Determine 8 – Backup settings

As soon as profitable, your AWS Console will appear like the next:

Determine 9 – Amazon Kinesis Knowledge Firehose

The Dynatrace Expertise

As soon as the preliminary setups are accomplished in each Dynatrace and the AWS Console, comply with these steps to visualise your new KHF stream knowledge within the Dynatrace console.

1. Log in to the Dynatrace Console, and on the left facet menu develop the “infrastructure” part, and choose “AWS”
2. From the display, choose the AWS account that you just need to add the KFH stream to.
3. Subsequent, you’ll see a virtualization of your AWS property for the account chosen. Choose the field marked “Supporting Providers”.
4. Subsequent, press the “Configure providers” button.
5. Subsequent, choose “Add service”.
6. From the drop down, choose “Kinesis Knowledge Firehose”.
7. Subsequent, choose the “Add metric” button, and choose the metrics that you just need to see for this stream. Dynatrace has a complete listing of metrics that may be chosen from the UI. The listing may be discovered on this hyperlink.

Troubleshooting

1. After configuration, load to the brand new KFH stream no knowledge within the Dynatrace Console.
   1. Test the Error Logs tab examine to be sure that the Vacation spot URL is appropriate for the Dynatrace Tenant.

Determine 10 – Vacation spot error logs

2. Invalid or misconfigured Dynatrace API token or scope isn’t correctly set.

Determine 11 – Vacation spot error logs

Conclusion

On this publish, we reveal the Kinesis Knowledge Firehose and associated Dynatrace integration. As well as, engineers can use CloudWatch Metrics to discover their manufacturing programs alongside occasions in Dynatrace. This gives a seamless, present view of your system (from logs to occasions and traces) in a single knowledge retailer.

To be taught extra about CloudWatch Service, see the Amazon CloudWatch residence web page. You probably have any questions, publish them on the AWS CloudWatch service discussion board.

If you happen to haven’t but signed up for Dynatrace, then you may check out Kinesis Knowledge Firehose with Dynatrace with a free Dynatrace trial.

In regards to the Authors

Erick Leon is a Technical Alliances Sr. Supervisor at Dynatrace, Observability Observe Architect, and Buyer Advocate. He promotes sturdy technical integrations with a concentrate on AWS. With over 15 years as a Dynatrace buyer, his real-world experiences and classes discovered deliver precious insights into the Dynatrace Clever Observability Platform.

Shashiraj Jeripotula (Raj) is a San Francisco-based Sr. Associate Options Architect at AWS. He works with varied unbiased software program distributors (ISVs), and companions who focus on cloud administration instruments and DevOps to develop joint options and speed up cloud adoption on AWS.