ACD2022 Session Report: Observability Masterclass #ACD2022

In this blog, I would like to share with you some of the information I gained while attending the AWS Community Day 2022.

Session Name Observability Masterclass Presented By: Chris Cooney, Developer Advocate@Corologix

Agenda:

• The fundamentals
  1. What is observability?
  2. Why is observability such an important conversation in 2022

• Best practices
  1. How to get the most out of your logs, metrics and traces
  2. Techniques for scaling your platform
• The big question that you will face
  1. Build or buy? the eternal Debate.
  2. How are we going to pay for all the data?

Report:

 What is observability?

Monitoring: it allows you to take a simple, static view of your data for example car dashboard which shows speed, and fuel. 

But observability allows you to Discover new things about your system.

For eg: using distance and fuel consumed we can figure out the mileage

3 pillars of observability:

1. Metrics that can be queried and rendered in different ways. Metrics are measured values derived from system performance. Examples:
   1. Application metrics
   2. Infrastructure Metrics
   3. Custom Metrics -SLOs, Sales rate, profit etc.
2. Logs: that can capture transformed and visualized in a new format to yield novel insights Example: 
   1. Infrastructure Logs
   2. Application logs
   3. API Logs
   4. Audit and Security logs
3. Traces that can tell you the journey of even a single request through your system. The trace consists of:
   1. A collection of “Spans”
   2. Any errors that occurred
   3. The duration of the trace and the duration of each span 
   4. Custom Values can be added.

Difference between  old monolithic architecture and microservices monitoring

monolithic architecture: less number of Resources to monitor easy to monitor but less scalable 

1. microservices architecture: more number of services to monitor but more scalable

General Best Practices:

1. Focus on open-source integrations, even if you use a SaaS solution. to be safe from lock-in

2. Think about the scale of your infrastructure upfront. How much data does each application output?  How much do you need to store? This is going to become very important.

Logging Best Practice

1. Decorate your logs with useful metadata 
2. Log in the machine-readable format “JSON”
3. Hot worm and cold storage to manage cost
4. USE MDC(Mapped diagnostic context: it will make sure we don’t pollute our log generation

Metric Best practice:

1. Create granular metrics. You can always aggregate them later.

Traces Best practices:

1. Identify a discrete, high-value transaction.

1. 1. Identify the point of ingress and egress in your system.
   2. Try to instrument everything. An incomplete trace can mislead you about the problem.

Challenges:

Deciding whether to buy or build an observability solution.

What to store, where to store it, and how to log it to prevent data explosion.

Summary:

This session will give you some really good practices to keep in mind before thinking about observability.

Reference video: