#ACD2022 [Report]How to choose a right messaging service for your workload

This Blog is a session report of the AWS Community Day 2022 India virtual event organised by AWS User Group India

The session is presented by Yan Cui The date of the Event is 11 November 2022

Report

Requirements of messaging Service

Functional requirement

1. FIFO (is the request should be processed in order)
2. replay event (Do we need to save the event or replay the event)

Non -functional requirement

1. error
2. cost
3. scalability

AWS offer few services which can perform the similar task

1. Event bridge

2. SNS

3. SQL

4. Kinesis

5. MSK(Managed kafka Service)

6. IoT Core

7. DynamoDB Stream

etc. etc.

How to choose the best one between SNS, SQS, Kinesis and Event Bridge

1. Constraints of Each Service:

every service has some limitations which we should consider before choosing that service

for eg:

Even Bridge: Service quota for put events in a region

2. Scaling concurrency

How Does Lambda Concurrency Scale with throughput?

SNS and Event Bridge can invoke the same number of lambda events as the number of requests for eg: 100 requests will invoke the 100 Lambda function concurrently

SQS: in this Service, the concurrency is controlled by several pollers therefore the concurrency will increase more gradually as compared to SNS and Event Bridge We will get Message patching

Kinesis: 1 concurrent for 1 shard(can be configure 10 lambda for 1 shard) Concurrency of Kinesis is Discrete

More Concurrency is not always Better

Because we need to control our downstream system which can be a legacy system that has a limitation of throughput and is not able to scale as quickly as lambda so it can return an error. in this case, there is a configuration where lambda can be configured for retry but we need to keep in mind the max number of retries we get. if the request crosses the retry limit we need to think of a solution like DLQ and then again we need to create some mechanism to process the events inside DLQ.

Or We Can use Kinesis we can control how much throughput we are processing from the stream and how much throughput we are sending downstream

Or we can use reserved concurrency

only if you have 2 or 3 function it can cause under-provisioning

Error Handling:

• Lambda: async invokes 2 retries and also supports DLQ
• DLQ only capture the failed event it will not give us the reason why the event failed or other necessary details. for this reason, Lambda destination should be preferred over DLQs,

Lambda Destination Do not support SQS.

Cost:

The Prices can differ based on usage so we need to select based on our needs.

Kinesis can be expensive for less number of messages but can be very cost-effective if we have a large number of messages.

Observability :

XRAY Can only trace Event Bridge

Decision Tree for Choosing Messaging service:

 

Thoughts:

This session shares problems and solutions which we could face in real-world projects. and also shared tips and tricks to make choices among the top-used messaging service.

Youtube Link for the event Track: