This Blog is a session report of the AWS Community Day 2022 India virtual event organised by AWS User Group India
The session is presented by Yan Cui The date of the Event is 11 November 2022
Requirements of messaging Service
- FIFO (is the request should be processed in order)
- replay event (Do we need to save the event or replay the event)
Non -functional requirement
AWS offer few services which can perform the similar task
- Event bridge
MSK(Managed kafka Service)
How to choose the best one between SNS, SQS, Kinesis and Event Bridge
1. Constraints of Each Service:
every service has some limitations which we should consider before choosing that service
Even Bridge: Service quota for put events in a region
2. Scaling concurrency
How Does Lambda Concurrency Scale with throughput?
SNS and Event Bridge can invoke the same number of lambda events as the number of requests for eg: 100 requests will invoke the 100 Lambda function concurrently
SQS: in this Service, the concurrency is controlled by several pollers therefore the concurrency will increase more gradually as compared to SNS and Event Bridge We will get Message patching
Kinesis: 1 concurrent for 1 shard(can be configure 10 lambda for 1 shard) Concurrency of Kinesis is Discrete
More Concurrency is not always Better
Because we need to control our downstream system which can be a legacy system that has a limitation of throughput and is not able to scale as quickly as lambda so it can return an error. in this case, there is a configuration where lambda can be configured for retry but we need to keep in mind the max number of retries we get. if the request crosses the retry limit we need to think of a solution like DLQ and then again we need to create some mechanism to process the events inside DLQ.
Or We Can use Kinesis we can control how much throughput we are processing from the stream and how much throughput we are sending downstream
Or we can use reserved concurrency
only if you have 2 or 3 function it can cause under-provisioning
- Lambda: async invokes 2 retries and also supports DLQ
- DLQ only capture the failed event it will not give us the reason why the event failed or other necessary details. for this reason, Lambda destination should be preferred over DLQs,
Lambda Destination Do not support SQS.
The Prices can differ based on usage so we need to select based on our needs.
Kinesis can be expensive for less number of messages but can be very cost-effective if we have a large number of messages.
XRAY Can only trace Event Bridge
Decision Tree for Choosing Messaging service:
This session shares problems and solutions which we could face in real-world projects. and also shared tips and tricks to make choices among the top-used messaging service.