[REPORT] How to analyze Amazon CloudFront logs using generative #CDN304 #AWSreInvent

Overview

Gain critical insights for optimizing the performance of your web application with Amazon CloudFront logs using Amazon Bedrock. In this builders’ session, learn why analyzing and monitoring logs is essential for detecting anomalies, enhancing reliability, and improving site performance. Whether you’re an SRE focused on fine-tuning performance or a CTO ensuring a robust infrastructure, discover how AI-powered log analysis empowers proactive decision-making. Find out how to harness the power of Amazon CloudFront data for actionable intelligence and to drive your application’s success.

Agenda

1. Amazon CloudFront observability options
2. Challenges with existing log analysis tools and techniques
3. The potential of Generative AI in log analysis
4. Demo: Using Amazon Bedrock to analyze Amazon CloudFront logs
5. Workshop walkthrough

The session started with presentations on CloudFront observability options, challenges with existing log analysis tools and techniques, and the potential of Generative AI in log analysis.

Amazon CloudFront Observability Options

Options include CloudWatch Metrics ,Access Logs - Real Time, Access Logs - Standard.

Use Cases:

• CloudWatch Metrics: Simple dashboards, monitoring, 4xx/5xx error alerts.
• Access Logs - Real Time: Real-time monitoring, custom dashboards, troubleshooting, clickstream, tracking.
• Access Logs - Standard: Ad hoc analysis, troubleshooting.

Challenges with existing log analysis tools

The potential of Generative AI in log analysis

Workshop walkthrough

The workshop included modules for:

• Scenario 1 - Summarize logs
• Scenario 2 - Detect anomalies
• Scenario 3 - Generate predictive insights
• Scenario 4 - Perform root cause analysis
• Scenario 5 - Identify trends
• Scenario 6 - Monitor proactively

Participants learned how to leverage Generative AI for detailed analysis of log data using natural language queries.

The architecture is as follows Analyze Amazon CloudFront logs using generative AI through natural language queries.

• Amazon S3 with Amazon CloudFront

  • Amazon　Cloudfront is the application, logs about access requests are stored in Amazon S3.

• Amazon Athena

  • Analyze data stored in S3 with standard SQL.
  • Execute SQL queries received from Lambda and retrieve data from CloudFront logs.

• AWS Lambda

  • Sends SQL queries received from Bedrock LLM to Amazon Athena.
  • Receive Athena processing results and return analysis results.

• Amazon Bedrock Action Group

  • Send SQL query and data schema information to AWS Lambda.
  • Retrieve data schema from S3 (define CloudFront log structure).
  • Forward SQL query and schema information to Lambda.

• Amazon Bedrock Agent

  • Receive natural language queries sent from Streamlit.
  • Sends queries to the Amazon Bedrock Large Language Model (LLM).

• Amazon Bedrock LLM

  • The key component that processes natural language queries.
  • It processes natural language queries and generates SQL queries.
  • These SQL queries retrieve specific details from CloudFront logs based on user requests.

• Amazon EC2

  • Hosts the Streamlit application.
  • Provides an intuitive and user-friendly web interface for entering natural language queries.

Environment Setup

Enable access to the model from the Amazon Bedrock console. For this session, selected Claude 3 Sonnet.

The other environments are prepared by CloudFormation, and the necessary information is found in Outputs.

Configure your IP address in the EC2 Security Group to allow access to the Streamlit application.

The following site was used to check the IP.

https://www.ipchicken.com/ or https://whatismyipaddress.com/

Access the EC2 instance and start the Streamlit Application.

Access the Streamlit application.

After accessing the web interface, submitted a query.

This example query is a prompt from Scenario 1 - Log Summary.

Example Query:

Provide a comprehensive monthly summary of CloudFront performance metrics from July to November 2024. Include average response times, cache hit rates, error rates, and total bandwidth usage for each month. Highlight any significant month-to-month changes.	

Result: After submitting the query, AI processed the data and returned a detailed summary of CloudFront's performance metrics, showcasing the benefits of AI-driven log analysis for efficiently handling large data volumes.

Conclusion

Although I didn’t have time to complete all scenarios, the session provided valuable experience in using Amazon Bedrock to analyze CloudFront logs, demonstrating the power of Generative AI in simplifying and enhancing log analysis.