I tried running a Spark job with the new 32vCPU worker size in Amazon EMR Serverless
This page has been translated by machine translation. View original
Introduction
On July 7, 2026, Amazon EMR Serverless added support for larger worker sizes.
The previous maximum configuration was 16 vCPU / 120 GB memory, but this update makes workers with 32 vCPU / up to 244 GB memory available.
| Item | Previous Maximum Configuration | This Update |
|---|---|---|
| Maximum vCPU | 16 | 32 |
| Maximum Memory | 120 GB | 244 GB |
| Memory Configuration | Arbitrary (in 8 GB increments) | 3 options: 60 / 120 / 244 GB |
The official documentation listing worker configurations is available below.
The official announcement lists the following as valid use cases for 32 vCPU workers:
- Shuffle-heavy workloads — can reduce data transfer between executors
- Jobs with data skew — can reduce the risk of out-of-memory failures
- Jobs requiring data caching — can retain more data in memory
In this article, we actually run a Spark job with 32 vCPU workers and verify that 32 cores are recognized and functioning.
Verification Details
Verification Environment
- Region: ap-northeast-1
- EMR release label: emr-7.8.0
- Runtime: Spark (PySpark)
Benchmark Design
We prepared the following benchmark script.
- Creates an RDD with 32 partitions and runs CPU-bound computation for 60 seconds on each partition
- Each partition computes integer exponentiation (
base ** exp), obtains the number of digits usingmath.log10, and accumulates them - No I/O or shuffling at all. CPU-intensive workload focused on large integer exponentiation
- Time-based loop designed so that all partitions process uniformly for 60 seconds
The core part of the script is as follows.
def cpu_burn_digits(partition_index, iterator, duration_seconds):
import time as t
import math
start = t.time()
total_digits = 0
iterations = 0
base = 2
exp = 1000
while True:
elapsed = t.time() - start
if elapsed >= duration_seconds:
break
result = base ** exp
digits = int(math.log10(result)) + 1 if result > 0 else 1
total_digits += digits
iterations += 1
exp += 100
if exp > 50000:
exp = 1000
base += 1
actual_elapsed = t.time() - start
yield (partition_index, total_digits, iterations, actual_elapsed)
Full script (cpu_bound_bench.py)
"""
CPU-bound evenly distributed benchmark for EMR Serverless
Design:
- Each partition repeatedly performs digit count calculations (integer exponentiation) for 60 seconds
- After 60 seconds, returns the total digits processed, and sums across all partitions
- No I/O or shuffling → CPU usage is expected to be evenly distributed
Arguments:
sys.argv[1] = duration_seconds (default: 60)
sys.argv[2] = num_partitions (default: 4)
"""
import subprocess
import sys
import time
from datetime import datetime
from pyspark.sql import SparkSession
def log(msg: str) -> None:
print(f"[{datetime.now().isoformat()}] {msg}")
def print_cpu_info() -> None:
"""Logs CPU information on the driver side."""
try:
out = subprocess.run(["lscpu"], capture_output=True, text=True, timeout=10)
log("=== lscpu (driver) ===")
print(out.stdout)
except Exception as e:
log(f"cpu info failed: {e}")
def cpu_burn_digits(partition_index, iterator, duration_seconds):
import time as t
import subprocess as sp
import math
if partition_index == 0:
try:
out = sp.run(["lscpu"], capture_output=True, text=True, timeout=10)
print(f"[partition-{partition_index}] === lscpu (executor) ===")
print(out.stdout)
except Exception:
pass
start = t.time()
total_digits = 0
iterations = 0
base = 2
exp = 1000
while True:
elapsed = t.time() - start
if elapsed >= duration_seconds:
break
result = base ** exp
digits = int(math.log10(result)) + 1 if result > 0 else 1
total_digits += digits
iterations += 1
exp += 100
if exp > 50000:
exp = 1000
base += 1
actual_elapsed = t.time() - start
yield (partition_index, total_digits, iterations, actual_elapsed)
def run(duration_seconds: int, num_partitions: int) -> None:
spark = SparkSession.builder.appName("CPUBoundBench").getOrCreate()
sc = spark.sparkContext
log("=== Job start ===")
log(f"duration_seconds={duration_seconds} num_partitions={num_partitions}")
log(f"defaultParallelism={sc.defaultParallelism}")
print_cpu_info()
start = time.time()
rdd = sc.parallelize(range(num_partitions), num_partitions)
results = rdd.mapPartitionsWithIndex(
lambda idx, it: cpu_burn_digits(idx, it, duration_seconds)
).collect()
total_wall_time = time.time() - start
log("=== Per-partition results ===")
grand_total_digits = 0
for partition_idx, digits, iters, elapsed in results:
log(f" partition[{partition_idx}]: digits={digits:,}, iterations={iters}, time={elapsed:.2f}s")
grand_total_digits += digits
log(f"=== TOTAL DIGITS (all partitions): {grand_total_digits:,} ===")
log(f"total_wall_time={total_wall_time:.2f}s")
log("=== Job end ===")
spark.stop()
if __name__ == "__main__":
duration = int(sys.argv[1]) if len(sys.argv) > 1 else 60
parts = int(sys.argv[2]) if len(sys.argv) > 2 else 4
run(duration, parts)
Creating the Application and Submitting the Job
We created an EMR Serverless application and submitted a job with 32 vCPU executors.
# Create application
aws emr-serverless create-application \
--name cpu-bench-32vcpu \
--release-label emr-7.8.0 \
--type SPARK \
--maximum-capacity '{
"cpu": "36vCPU",
"memory": "248GB",
"disk": "200GB"
}' \
--region ap-northeast-1
Details about the maximumCapacity settings will be discussed in the notes section below.
# Submit job
aws emr-serverless start-job-run \
--application-id <application-id> \
--execution-role-arn arn:aws:iam::<account-id>:role/emr-serverless-job-role \
--name cpu-bench-digits-32vcpu \
--job-driver '{
"sparkSubmit": {
"entryPoint": "s3://<bucket>/cpu_bound_bench.py",
"entryPointArguments": ["60", "32"],
"sparkSubmitParameters": "--conf spark.executor.cores=32 --conf spark.executor.memory=221g --conf spark.executor.instances=1 --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.dynamicAllocation.enabled=false"
}
}' \
--configuration-overrides '{
"monitoringConfiguration": {
"managedPersistenceMonitoringConfiguration": {"enabled": true}
}
}' \
--region ap-northeast-1
We specify spark.executor.memory=221g. This is the value configured to fit within the 244 GB worker configuration, and the calculation details will be discussed in the notes section below.
Job Execution Results
The job completed successfully. The following is an excerpt from the get-job-run API response.
{
"jobRun": {
"name": "cpu-bench-digits-32vcpu",
"state": "SUCCESS",
"releaseLabel": "emr-7.8.0",
"jobDriver": {
"sparkSubmit": {
"sparkSubmitParameters": "--conf spark.executor.cores=32 --conf spark.executor.memory=221g --conf spark.executor.instances=1 --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.dynamicAllocation.enabled=false"
}
},
"totalResourceUtilization": {
"vCPUHour": 0.753,
"memoryGBHour": 5.655,
"storageGBHour": 1.111
},
"totalExecutionDurationSeconds": 118,
"startedAt": "2026-07-09T00:34:46+09:00",
"endedAt": "2026-07-09T00:36:44+09:00"
}
}
The job completed in 118 seconds.
Verifying 32-Core Recognition
From the executor GC logs, we confirmed that 32 cores and 244 GB memory were correctly recognized.
[0.268s][info][gc,init] CPUs: 32 total, 32 available
[0.268s][info][gc,init] Memory: 244G
[0.268s][info][gc,init] Heap Min Capacity: 221G
[0.268s][info][gc,init] Heap Initial Capacity: 221G
[0.268s][info][gc,init] Heap Max Capacity: 221G
[0.268s][info][gc,init] Parallel Workers: 23
The JVM recognized 32 CPUs, and a heap of 221 GB was allocated. The GC Parallel Workers was set to 23 (a value determined by the JVM based on the number of cores).
Results of 32-Partition Parallel Execution
All 32 partitions processed for approximately 60 seconds each, computing a total of approximately 41.3 billion digits. The variance in throughput between partitions was less than 1% (minimum 1,284M digits, maximum 1,297M digits). These results confirm that the 32 partitions were processed in parallel with the 32 executor cores setting.
Per-partition results for all 32 partitions (full text)
partition[0]: digits=1,295,403,119, iterations=34815, time=60.00s
partition[1]: digits=1,294,568,243, iterations=34805, time=60.00s
partition[2]: digits=1,296,170,388, iterations=34824, time=60.00s
partition[3]: digits=1,294,816,756, iterations=34808, time=60.00s
partition[4]: digits=1,292,021,453, iterations=34773, time=60.00s
partition[5]: digits=1,293,033,148, iterations=34786, time=60.00s
partition[6]: digits=1,284,219,297, iterations=34654, time=60.00s
partition[7]: digits=1,294,733,733, iterations=34807, time=60.00s
partition[8]: digits=1,294,816,756, iterations=34808, time=60.00s
partition[9]: digits=1,291,944,930, iterations=34772, time=60.00s
partition[10]: digits=1,294,321,402, iterations=34802, time=60.00s
partition[11]: digits=1,296,256,569, iterations=34825, time=60.00s
partition[12]: digits=1,295,066,940, iterations=34811, time=60.00s
partition[13]: digits=1,293,751,941, iterations=34795, time=60.00s
partition[14]: digits=1,294,485,777, iterations=34804, time=60.00s
partition[15]: digits=1,295,912,960, iterations=34821, time=60.00s
partition[16]: digits=1,295,912,960, iterations=34821, time=60.00s
partition[17]: digits=1,295,066,940, iterations=34811, time=60.00s
partition[18]: digits=1,294,076,233, iterations=34799, time=60.00s
partition[19]: digits=1,292,796,894, iterations=34783, time=60.00s
partition[20]: digits=1,295,657,204, iterations=34818, time=60.00s
partition[21]: digits=1,293,430,620, iterations=34791, time=60.00s
partition[22]: digits=1,293,350,754, iterations=34790, time=60.00s
partition[23]: digits=1,289,806,569, iterations=34743, time=60.00s
partition[24]: digits=1,294,983,360, iterations=34810, time=60.00s
partition[25]: digits=1,289,806,569, iterations=34743, time=60.00s
partition[26]: digits=1,291,189,920, iterations=34762, time=60.00s
partition[27]: digits=1,289,244,165, iterations=34735, time=60.00s
partition[28]: digits=1,296,777,553, iterations=34831, time=60.00s
partition[29]: digits=1,284,494,185, iterations=34659, time=60.00s
partition[30]: digits=1,290,380,860, iterations=34751, time=60.00s
partition[31]: digits=1,293,590,909, iterations=34793, time=60.00s
TOTAL DIGITS: 41,382,089,107
Reference: 1 vCPU Job Results
Results from running the same script with a 1 vCPU / 1 partition configuration.
| Item | 1 vCPU Job | 32 vCPU Job |
|---|---|---|
| executor cores | 1 | 32 |
| executor memory | 4g | 221g |
| Number of partitions | 1 | 32 |
| Total digits | 1,473,360,014 | 41,382,089,107 |
| totalExecutionDurationSeconds | 133s | 118s |
| vCPUHour | 0.064 | 0.753 |
| memoryGBHour | 0.321 | 5.655 |
※ totalExecutionDurationSeconds includes job startup and shutdown processing. The benchmark computation itself is fixed at 60 seconds for both jobs.
Notes
Memory Configuration for 32 vCPU Workers Is Limited to 3 Options
The memory size allocated for 32 vCPU workers cannot be set to an arbitrary value; it must be one of the following three discrete values.
- 60 GB
- 120 GB
- 244 GB
For other vCPU sizes (1–16 vCPU), memory can be configured within a range and increment defined for each vCPU size, but only 32 vCPU has this restriction.
Calculating Memory Overhead
In EMR Serverless, executor memory overhead is added on top of spark.executor.memory. When not specified, it is based on spark.executor.memoryOverheadFactor, and approximately 10% of the executor memory (minimum 384 MB) is treated as overhead. The total must fit within one of the three configuration values listed above. For details, please refer to the official documentation on worker configurations.
For example, when using the 244 GB configuration, set spark.executor.memory to approximately 221g (221 GB + 10% ≈ 243 GB → fits within the 244 GB configuration). If the total is too far from the configuration value, the job may be rejected.
Setting maximumCapacity
The application's maximumCapacity must be set to accommodate the total resources of both the driver and executors. When using a 32 vCPU / 244 GB executor, add the driver resources (minimum 1 vCPU / 4 GB) to determine the value to set.
Summary
32 vCPU workers are now available in EMR Serverless. We actually launched a 32 vCPU executor, confirmed that the JVM recognized 32 CPUs and 244 GB memory, and verified that a CPU-intensive job with 32 partitions completed with roughly equal throughput across all partitions.
Compared to the previous 16 vCPU configuration, a new option of 32 vCPU / up to 244 GB is now available for workloads that require greater parallelism or memory capacity per executor. The fact that memory configurations are limited to three choices — 60 / 120 / 244 GB — is something to keep in mind when configuring jobs.
