Saturday, December 3, 2022
HomeBig DataMicroservice observability with Amazon OpenSearch Service half 1: Hint and log correlation

Microservice observability with Amazon OpenSearch Service half 1: Hint and log correlation

Fashionable enterprises are more and more adopting microservice architectures and transferring away from monolithic constructions. Though microservices present agility in growth and scalability, and encourage use of polyglot techniques, in addition they add complexity. Troubleshooting distributed providers is difficult as a result of the applying behavioral knowledge is distributed throughout a number of machines. Subsequently, with a purpose to have deep insights to troubleshoot distributed functions, operational groups want to gather utility behavioral knowledge in a single place to scan via them.

Though establishing monitoring techniques focuses on analyzing solely log knowledge may help you perceive what went incorrect and notify about any anomalies, it fails to supply perception into why one thing went incorrect and precisely the place within the utility code it went incorrect. Fixing points in a fancy community of techniques is like discovering a needle in a haystack. Observability primarily based on Open Requirements outlined by OpenTelemetry addresses the issue by offering help to deal with logs, traces, and metrics inside a single implementation.

On this collection, we cowl the setup and troubleshooting of a distributed microservice utility utilizing logs and traces. Logs are immutable, timestamped, discreet occasions taking place over a time period, whereas traces are a collection of associated occasions that seize the end-to-end request circulate in a distributed system. We glance into find out how to accumulate a big quantity of logs and traces in Amazon OpenSearch Service and correlate these logs and traces to seek out the precise subject and the place the problem was generated.

Any investigation of points in enterprise functions must be logged in an incident report, in order that operational and growth groups can collaborate to roll out a repair. When any investigation is carried out, it’s necessary to jot down a story in regards to the subject in order that it may be utilized in dialogue later. We glance into find out how to use the most recent pocket book function in OpenSearch Service to create the incident report.

On this submit, we focus on the structure and utility troubleshooting steps.

Resolution overview

The next diagram illustrates the observability resolution structure to seize logs and traces.

The answer parts are as follows:

  • Amazon OpenSearch Service is a managed AWS service that makes it simple to deploy, function, and scale OpenSearch clusters within the AWS Cloud. OpenSearch Service helps OpenSearch and legacy Elasticsearch open-source software program (as much as 7.10, the ultimate open-source model of the software program).
  • FluentBit is an open-source processor and forwarder that collects, enriches, and sends metrics and logs to numerous locations.
  • AWS Distro for OpenTelemetry is a safe, production-ready, AWS-supported distribution of the OpenTelemetry undertaking. With AWS Distro for OpenTelemetry, you may instrument your functions simply as soon as to ship correlated metrics and traces to a number of AWS and Associate monitoring options, together with OpenSearch Service.
  • Information Prepper is an open-source utility service with the flexibility to filter, enrich, rework, normalize, and combination knowledge to allow an end-to-end evaluation lifecycle, from gathering uncooked logs to facilitating subtle and actionable interactive advert hoc analyses on the information.
  • We use a pattern observability store net utility constructed as a microservice to display the capabilities of the answer parts.
  • Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you should use to run Kubernetes on AWS with no need to put in, function, and preserve your personal Kubernetes management aircraft or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and administration of the container.

On this resolution, we have now a pattern o11y (Observability) Store net utility written in Python and Java, and deployed in an EKS cluster. The online utility consists of varied providers. When some operations are executed from the entrance finish, the request travels via a number of providers on the backend. The applying providers are operating as separate containers, whereas AWS Distro for OpenTelemetry, FluentBit, and Information Prepper are operating as sidecar containers.

FluentBit is used for accumulating log knowledge from utility containers, after which sends logs to Information Prepper. For accumulating traces, first the applying providers are instrumented utilizing the OpenTelemetry SDK. Then, with AWS Distro for OpenTelemetry collector, hint info is collected and despatched to Information Prepper. Information Prepper forwards the logs and traces knowledge to OpenSearch Service.

We advocate deploying the OpenSearch Service area inside a VPC, so a reverse proxy is required to have the ability to log in to OpenSearch Dashboards.


You want an AWS account with essential permissions to deploy the answer.

Arrange the surroundings

We use AWS CloudFormation to provision the parts of our structure. Full the next steps:

  1. Launch the CloudFormation stack within the us-east-1 Area:
  2. Chances are you’ll hold the stack title default to AOS-Observability.
  3. Chances are you’ll change the OpenSearchMasterUserName parameter used for OpenSearch Service login whereas conserving different parameter values to default. The stack provisions a VPC, subnets, safety teams, route tables, an AWS Cloud9 occasion, and an OpenSearch Service area, together with a Nginx reverse proxy. It additionally configures AWS Identification and Entry Administration (IAM) roles. The stack can even generate a brand new random password for OpenSearch Service area which could be seen within the CloudFormation Outputs tab beneath AOSDomainPassword.
  4. On the stack’s Outputs tab, select the hyperlink for the AWS Cloud9 IDE.
  5. Run the next code to put in the required packages, configure the surroundings variables and provision the EKS cluster:
    curl -sSL | bash -s <<CloudFormation Stack Identify>>

    After the assets are deployed, it prints the hostname for the o11y Store net utility.

  6. Copy the hostname and enter it within the browser.

This opens the o11y Store microservice utility, as proven within the following screenshot.

Entry the OpenSearch Dashboards

To entry the OpenSearch Dashboards, full the next steps:

  1. Select the hyperlink for AOSDashboardsPublicIP from the CloudFormation stack outputs. As a result of the OpenSearch Service area is deployed contained in the VPC, we use an Nginx reverse proxy to ahead the visitors to the OpenSearch Service area. As a result of the OpenSearch Dashboards URL is signed utilizing a self-signed certificates, you want to bypass the safety exception. In manufacturing, a sound certificates is really useful for safe entry.
  2. Assuming you’re utilizing Google Chrome, when you are on this web page, enter thisisunsafe.Google Chrome redirects you to the OpenSearch Service login web page.
  3. Log in with the OpenSearch Service login particulars (discovered within the CloudFormation stack output: AOSDomainUserName and AOSDomainPassword).You’re introduced with a dialog requesting you so as to add knowledge for exploration.
  4. Choose Discover alone.
  5. When requested to pick a tenant, depart the default choices and select Verify.
  6. Open the Hamburger menu to discover the plugins inside OpenSearch Dashboards.

That is the OpenSearch Dashboards person interface. We use it within the subsequent steps to research, discover, repair, and discover the foundation reason for the problem.

Logs and traces technology

Click on across the o11y Store utility to simulate person actions. This may generate logs and a few traces for the related microservices saved in OpenSearch Service. You are able to do the method a number of occasions to generate extra pattern logs and traces knowledge.

Create an index sample

An index sample selects the information to make use of and permits you to outline properties of the fields. An index sample can level to a number of indexes, knowledge streams, or index aliases.

You must create an index sample to question the information via OpenSearch Dashboards.

  1. On OpenSearch Dashboards, select Stack Administration.
  2. Select Index Patterns
  3. Select Create index sample.
  4. For Index sample title, enter sample_app_logs. OpenSearch Dashboards additionally helps wildcards.
  5. Select Subsequent step.
  6. For Time subject, select time.
  7. Select Create index sample.
  8. Repeat these steps to create the index sample otel-v1-apm-span* with occasion.time because the time subject for locating traces.

Search logs

Select the menu icon and search for the Uncover part in OpenSearch Dashboards. The Uncover panel permits you to view and question logs. Test the log exercise taking place within the microservice utility.

If you happen to can’t see any knowledge, enhance the time vary to one thing massive (just like the final hour). Alternatively, you may play across the o11y Store utility to generate current logs and traces knowledge.

Instrument functions to generate traces

Purposes should be instrumented to generate and ship hint knowledge downstream. There are two varieties of instrumentation:

  • Automated – In automated instrumentation, no utility code change is required. It makes use of an agent that may seize hint knowledge from the operating utility. It requires utilization of the language-specific API and SDK, which takes the configuration offered via the code or surroundings and supplies good protection of endpoints and operations. It routinely determines the span begin and finish.
  • Guide – In guide instrumentation, builders want so as to add hint seize code to the applying. This supplies customization by way of capturing traces for a customized code block, naming numerous parts in OpenTelemetry like traces and spans, including attributes and occasions, and dealing with particular exceptions inside the code.

In our utility code, we use guide instrumentation. Check with Guide Instrumentation to gather traces within the GitHub repository to know the steps.

Discover hint analytics

OpenSearch Service model 1.3 has a brand new module to help observability.

  1. Select the menu icon and search for the Observability part beneath OpenSearch Plugins.
  2. Select Hint analytics to look at a number of the traces generated by the backend service. If you happen to overlook ample knowledge, enhance the time vary. Alternatively, select all of the buttons on the pattern app webpage for every utility service to generate ample hint knowledge to debug. You may select every possibility a number of occasions. The next screenshot reveals a summarized view of the traces captured.

    The dashboard view teams traces collectively by hint group title and supplies details about common latency, error charge, and developments related to a specific operation. Latency variance signifies if the latency of a request falls under the 95 percentile or above. If there are a number of hint teams, you may scale back the view by including filters on numerous parameters.
  3. Add a filter on the hint group client_checkout.

    The next screenshot reveals our filtered outcomes.

    The dashboard additionally contains a map of all of the linked providers. The Service map helps present a high-level view on what’s happening within the providers primarily based on the color-coding grouped by Latency, Error charge, and Throughput. This helps you establish issues by service.
  4. Select Error charge to discover the error charge of the linked providers.Based mostly on the color-coding within the following diagram, it’s evident that the fee service is throwing errors, whereas different providers are working high quality with none errors.
  5. Swap to the Latency view, which reveals the relative latency in milliseconds with completely different colours.
    That is helpful for troubleshooting bottlenecks in microservices.

    The Hint analytics dashboard additionally reveals distribution of traces over time and hint error charge over time.
  6. To find the listing of traces, beneath Hint analytics within the navigation pane, select Traces.
  7. To seek out the listing of providers, depend of traces per service, and different service-level statistics, select Companies within the navigation pane.

Search traces

Now we wish to drill down and study extra about find out how to troubleshoot errors.

  1. Return to the Hint analytics dashboard.
  2. Select Error Charge Service Map and select the fee service on the graph.The fee service is in darkish purple. This additionally units the fee service filter on the dashboard, and you may see the hint group within the higher pane.
  3. Select the Traces hyperlink of the client_checkout hint group.

    You’re redirected to the Traces web page. The listing of traces for the client_checkout hint group could be discovered right here.
  4. To view particulars of the traces, select Hint IDs.You may see a pie chart exhibiting how a lot time the hint has spent in every service. The hint consists of a number of spans, which is outlined as a timed operation that represents a chunk of workflow within the distributed system. On the fitting, you too can see time spent in every span, and which have an error.
  5. Copy the hint ID within the client-checkout group.

Log and hint correlation

Though the log and hint knowledge supplies beneficial info individually, the precise benefit is after we can relate hint knowledge to log knowledge to seize extra particulars about what went incorrect. There are 3 ways we will correlate traces to logs:

  • Runtime – Logs, traces, and metrics can report the second of time or the vary of time the run came about.
  • Run context – That is also referred to as the request context. It’s commonplace follow to report the run context (hint and span IDs in addition to user-defined context) within the spans. OpenTelemetry extends this follow to logs the place doable by together with the TraceID and SpanID within the log information. This enables us to instantly correlate logs and traces that correspond to the identical run context. It additionally permits us to correlate logs from completely different parts of a distributed system that participated within the explicit request.
  • Origin of the telemetry – That is also referred to as the useful resource context. OpenTelemetry traces and metrics comprise details about the useful resource they arrive from. We prolong this follow to logs by together with the useful resource within the log information.

These three correlation strategies could be the muse of highly effective navigational, filtering, querying, and analytical capabilities. OpenTelemetry goals to report and accumulate logs in a fashion that permits such correlations.

  1. Use the copied traceId from the earlier part and seek for corresponding logs on the Occasion analytics web page.
    We use the next PPL question:
    supply = sample_app_logs | the place traceId = “<<trace_id>>

    Be certain that to extend the time vary to a minimum of the final hour.

  2. Select Replace to seek out the corresponding log knowledge for the hint ID.
  3. Select the develop icon to seek out extra particulars.This reveals you the small print of the log together with the traceId. This log reveals that the fee checkout operation failed. This correlation allowed us to seek out key info within the log that permits us to go to the applying and debug the code.
  4. Select the Traces tab to see the corresponding hint knowledge linked with the log knowledge.
  5. Select View surrounding occasions to find different occasions taking place on the identical time.

This info could be beneficial while you wish to perceive what’s happening in the entire utility, significantly how different providers are impacted throughout that point.


This part supplies the mandatory info for deleting numerous assets created as a part of this submit.

It is suggested to carry out the under steps after going via the subsequent submit of the collection.

  1. Execute the next command on the Cloud9 terminal to take away Elastic Kubernetes Service Cluster and its assets.
    eksctl delete cluster --name=observability-cluster

  2. Execute the script to delete the Amazon Elastic Container Registry repositories.
    cd observability-with-amazon-opensearch-blog/scripts

  3. Delete the CloudFormation stacks in sequence - eksDeploy, AOS-Observability.


On this submit, we deployed an Observability (o11y) Store microservice utility with numerous providers and captured logs and traces from the applying. We used FluentBit to seize logs, AWS Distro for Open Telemetry to seize traces, and Information Prepper to gather these logs and traces and ship it to OpenSearch Service. We confirmed find out how to use the Hint analytics web page to look into the captured traces, particulars about these traces, and repair maps to seek out potential points. To correlate log and hint knowledge, we demonstrated find out how to use the Occasion analytics web page to jot down a easy PPL question to seek out corresponding log knowledge. The implementation code could be discovered within the GitHub repository for reference.

The following submit in our collection covers using PPL to create an operational panel to observe our microservices together with an incident report utilizing notebooks.

In regards to the Writer

Subham Rakshit is a Streaming Specialist Options Architect for Analytics at AWS primarily based within the UK. He works with clients to design and construct search and streaming knowledge platforms that assist them obtain their enterprise goal. Outdoors of labor, he enjoys spending time fixing jigsaw puzzles along with his daughter.

Marvin Gersho is a Senior Options Architect at AWS primarily based in New York Metropolis. He works with a variety of startup clients. He beforehand labored for a few years in engineering management and hands-on utility growth, and now focuses on serving to clients architect safe and scalable workloads on AWS with a minimal of operational overhead. In his free time, Marvin enjoys biking and technique board video games.

Rafael Gumiero is a Senior Analytics Specialist Options Architect at AWS. An open-source and distributed techniques fanatic, he supplies steering to clients who develop their options with AWS Analytics providers, serving to them optimize the worth of their options.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments