
Trying Out "Flex execution", a New Feature of Amazon EMR Serverless
This page has been translated by machine translation. View original
Introduction
On November 17, 2026, Flex execution for Amazon EMR Serverless was announced.
Flex execution is an execution option that runs Spark executors on spare compute capacity and the driver on Standard (the default execution option) workers.
In this article, I submit the same script to a single application with both Standard and Flex, and check how to enable Flex and the differences visible through get-job-run and other commands.
Test Environment
- Region: Tokyo (ap-northeast-1)
- EMR release: emr-7.14.0
- AWS CLI: 2.41.2
- Worker size: not specified
Preparation
I submit a Standard job and a Flex job to a single application. The script is a lightweight process written in PySpark that generates 20 million rows without any external input and aggregates them.
Commands and files for reproduction
This is the command to create the application. The applicationId included in the response is written as <アプリケーションID> in the subsequent commands.
aws emr-serverless create-application \
--name flex-verify-1120 \
--type SPARK \
--release-label emr-7.14.0 \
--maximum-capacity '{"cpu":"64 vCPU","memory":"256 GB","disk":"1000 GB"}'
This is the command to place the script in S3.
aws s3 cp agg_job.py s3://<バケット名>/flex-verify-1120/scripts/agg_job.py
This is agg_job.py. It writes the number of keys and the sum of value to standard output.
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("flex-verify-agg").getOrCreate()
# A lightweight job that generates 20 million rows and aggregates them by 10,000 keys (no external input)
df = (
spark.range(0, 20_000_000)
.withColumn("key", F.col("id") % 10_000)
.withColumn("value", (F.col("id") * 7) % 1_000)
)
agg = df.groupBy("key").agg(F.count("*").alias("cnt"), F.sum("value").alias("total"))
print("RESULT keys=", agg.count())
print("RESULT total_sum=", agg.agg(F.sum("total")).collect()[0][0])
spark.stop()
This is configuration-overrides.json. It outputs the job logs to S3.
{
"monitoringConfiguration": {
"s3MonitoringConfiguration": {
"logUri": "s3://<バケット名>/flex-verify-1120/logs/"
}
}
}
Submitting Jobs
I used the same job-driver.json for the Standard job and the Flex job.
{
"sparkSubmit": {
"entryPoint": "s3://<バケット名>/flex-verify-1120/scripts/agg_job.py"
}
}
I submitted the Standard job with the following command.
aws emr-serverless start-job-run \
--application-id <アプリケーションID> \
--execution-role-arn arn:aws:iam::<アカウントID>:role/<実行ロール名> \
--name standard-run \
--job-driver file://job-driver.json \
--configuration-overrides file://configuration-overrides.json
I submitted the Flex job with --flex-execution added.
aws emr-serverless start-job-run \
--application-id <アプリケーションID> \
--execution-role-arn arn:aws:iam::<アカウントID>:role/<実行ロール名> \
--name flex-run \
--flex-execution '{"enabled": true}' \
--job-driver file://job-driver.json \
--configuration-overrides file://configuration-overrides.json
The only differences between the two commands are the value of --name and whether --flex-execution is added. The jobRunId included in the response is used as <ジョブ実行ID> in get-job-run in the next section.
Execution Results
I submitted the two jobs to the same application about one second apart and ran them in parallel. The execution times (Japan time) were 21:36:19 to 21:37:13 for the Standard job and 21:36:19 to 21:37:27 for the Flex job.
Differences Seen in get-job-run and list-job-runs
I retrieved the state of each job with get-job-run.
aws emr-serverless get-job-run \
--application-id <アプリケーションID> \
--job-run-id <ジョブ実行ID>
| Item | Standard | Flex |
|---|---|---|
| state | SUCCESS | SUCCESS |
| flexExecution | No field | {"enabled": true} |
| billedResourceUtilization vCPUHour | 0.267 | 0.276 |
| billedResourceUtilization memoryGBHour | 1.067 | 1.102 |
The values in the table are from running the two jobs simultaneously, once each. According to the User Guide, billedResourceUtilization is the total usage of Standard and Flex workers used in a single job run. It is a value representing usage rather than cost, and it is not possible to extract only the portion for Flex workers.
I retrieved the list with list-job-runs.
aws emr-serverless list-job-runs \
--application-id <アプリケーションID>
Here too, flexExecution returned {"enabled": true} only for the Flex job.
Aggregation Results
The driver's standard output is compressed with gzip and saved under the log output destination. The storage location is applications/<アプリケーションID>/jobs/<ジョブ実行ID>/SPARK_DRIVER/stdout.gz. The output was the same for the Standard job and the Flex job.
RESULT keys= 10000
RESULT total_sum= 9990000000
Usage Conditions
The following conditions are as described in the User Guide.
- Flex can be used with Spark batch jobs. Hive jobs, Spark Structured Streaming jobs, and interactive sessions using Livy or Spark Connect are not supported.
- Flex is a per-job-run setting and cannot be set as an application default. Specify it with
start-job-run. - Flex executors cannot use 1 vCPU or 2 vCPU workers. If you specify them, StartJobRun is rejected. This restriction does not apply to the driver.
- If no executor joins the driver within 20 minutes of the job starting, the job fails. It does not automatically switch to Standard, so you resubmit it yourself, either as Flex or switched to Standard.
- Flex workers are counted against a quota separate from Standard,
Max concurrent flex vCPUs per account. A job that exceeds this quota fails with a quota-exceeded error.
In the account used for this test, the value of Max concurrent flex vCPUs per account was 4000.
Summary
I was able to run jobs with Flex execution for Amazon EMR Serverless.
Flex is an execution option that runs executors on spare compute capacity, and the announcement says it can reduce costs by up to 60%. It is a mechanism comparable to EC2 Spot Fleet or Fargate Spot.
If you have an opportunity to run non-urgent Spark batch jobs at low cost, please give it a try.
