I presented a talk titled "AWS Step Functions: Breaking Through the Wall of Large-Scale Parallelism" at JAWS SONIC 2026 & MIDNIGHT JAWS 2026 #jawsug #jawssonic2026
This page has been translated by machine translation. View original
This is Kasahara from the Data Business Division.
I gave a presentation titled "Overcoming the Walls of Large-Scale Parallelism with AWS Step Functions" at "JAWS SONIC 2026 & MIDNIGHT JAWS 2026," a 24-hour online streaming event held from Saturday, September 5, 2026, 12:00 to Sunday, September 6, 2026, 12:00.
I presented as the Niigata branch at 13:00 on Saturday, September 5, 2026, making it one of the very first presentations of the event.
Presentation Slides
Video
Summary
This time, I put together a guide on the mindset for large-scale parallel processing with Step Functions, focusing on how to overcome the common walls you tend to run into.
I frequently use Step Functions myself, and I struggle with configuring large-scale state machines. I wrote this while drawing on my own experiences.
- Standard event history: 25,000 events
- Split state machines using a nested structure
- Input/output payload size: 256 KiB
- Use JSONPath / JSONata to narrow down and pass only the necessary values
- Making good use of Map states
- Use Distributed Map for simple processing / leverage regular Map states for complex processing
- Increasing parallelism shifts the bottleneck to downstream service quota limits
- Pay attention to the quotas of services called from Step Functions
- Control effectively using MaxConcurrency
I think keeping these points in mind is the way to go when building large-scale parallel workflows with Step Functions.