Amazon Managed Streaming for Apache Kafka

Introduction to Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Amazon MSK makes it easy to ingest and process streaming data in real-time with fully managed Apache Kafka.

APACHE KAFKA AT A HIGH LEVEL

 What can fully manage Apache Kafka on AWS do:

• Allow you to create, update, and delete clusters • MSK creates & manages Kafka brokers nodes & Zookeeper nodes for you • Deploy the MSK cluster in your VPC, multi-AZ (up to 3 for HA) • Automatic recovery from common Apache Kafka failures Data is stored on EBS volumes 

• You can build producers and consumers of data 
• Can create custom configurations for your clusters
• Default message size of 1MB
• Possibilities of sending large messages (ex: 10MB) into Kafka after custom configuration 

MSK – Configurations 

• Choose the number of AZs (3 – recommended, or 2)
• Choose the VPC & Subnets
• The broker instance type (ex: kafka.m5.large)
• The number of brokers per AZ (can add brokers later)
• Size of your EBS volumes (1GB – 16TB) 

MSK – Security 

Encryption:

• • Optional in-flight using TLS between the brokers
  • Optional in-flight with TLS between the clients and brokers
  • At rest for your EBS volumes using KMS

Network Security:

• • Authorize specific security groups for your Apache Kafka clients

Authentication & Authorization

• • Define who can read/write to which topics
  • Mutual TLS (AuthN) + ​​Kafka ACLs (AuthZ)
  • SASL/SCRAM (AuthN) + ​​Kafka ACLs (AuthZ)
  • IAMAccessControl(AuthN+AuthZ)

MSK – Monitoring 

CloudWatch Metrics • Basic monitoring (cluster and broker metrics) • Enhanced monitoring (++enhanced broker metrics) • Topic-level monitoring (++enhanced topic-level metrics) 

• Prometheus (Open-Source Monitoring) • Opens a port on the broker to export cluster, broker, and topic-level metrics • Setup the JMX Exporter (metrics) or Node Exporter (CPU and disk metrics) 

• Broker Log Delivery • Delivery to CloudWatch Logs • Delivery to Amazon S3 • Delivery to Kinesis Data Streams 

MSK Serverless 

• Run Apache Kafka on MSK without managing the capacity
• MSK automatically provisions resources and scales compute & storage
• You just define your topics and your partitions and you're good to go!
• Security: IAM Access Control for all clusters

Kinesis Data Streams vs Amazon MSK