Trying Out the New S3 Tables Feature "Variant Data Type" with EMR Serverless
This page has been translated by machine translation. View original
Introduction
On July 28, 2026, S3 Tables added support for the Variant data type of Apache Iceberg V3. The Variant type can store nested JSON as-is in a single column, so you can handle semi-structured data without defining a schema in advance. When reading, you can extract only the fields you need with types, using JSONPath-style paths. What's New also mentions the file pruning benefits of shredding, but this article does not verify them.
What I Tested
Test Environment
The DDL and DML for the Variant type require Spark 4.0.x. The VARIANT data type was added in Apache Spark 4.0.0. This time I used EMR Serverless emr-spark-8.0.0, which includes Spark 4.0.2 and Iceberg 1.10.1, in the Tokyo region.
Connection to S3 Tables goes through the Glue Iceberg REST endpoint. The catalog settings on the Spark side are as follows.
spark.sql.catalog.s3t = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.s3t.type = rest
spark.sql.catalog.s3t.uri = https://glue.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3t.warehouse = 123456789012:s3tablescatalog/variant-test-bucket
spark.sql.catalog.s3t.rest.sigv4-enabled = true
spark.sql.catalog.s3t.rest.signing-name = glue
spark.sql.catalog.s3t.rest.signing-region = ap-northeast-1
Specify the warehouse not as an ARN but in the format of the account ID followed by s3tablescatalog/<bucket name>.
Creating the Table
I created a table with payload declared as the VARIANT type. Since the Variant type was added in V3, you need to specify 3 for format-version. How to create a V3 table is summarized in Using Apache Iceberg V3.
CREATE TABLE IF NOT EXISTS s3t.myns.variant_demo (
event_id STRING NOT NULL,
device_id STRING NOT NULL,
event_ts TIMESTAMP NOT NULL,
payload VARIANT NOT NULL
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
Writing Data
Passing a JSON string to parse_json() converts it to VARIANT. I INSERTed three records imitating sensor data. The third record is data without a tags key.
INSERT INTO s3t.myns.variant_demo VALUES
('evt_001', 'sensor_A', current_timestamp(),
parse_json('{"temperature": 23.5, "humidity": 62.1, "location": {"city": "Tokyo", "building": "HQ"}, "tags": ["indoor", "floor3"]}')),
('evt_002', 'sensor_B', current_timestamp(),
parse_json('{"temperature": 31.2, "humidity": 78.5, "location": {"city": "Osaka", "building": "Lab"}, "tags": ["outdoor"]}')),
('evt_003', 'sensor_C', current_timestamp(),
parse_json('{"temperature": 18.9, "humidity": 45.0, "location": {"city": "Nagoya", "building": "DC"}}'))
After writing, I loaded the table and checked the DataFrame schema.
root
|-- event_id: string (nullable = false)
|-- device_id: string (nullable = false)
|-- event_ts: timestamp (nullable = false)
|-- payload: variant (nullable = false)
The payload is recognized as variant.
Reading and Filtering Data
You pass three things to variant_get(): the target column, a JSONPath-style path, and the type to extract.
SELECT
event_id,
device_id,
variant_get(payload, '$.temperature', 'DOUBLE') AS temperature,
variant_get(payload, '$.humidity', 'DOUBLE') AS humidity,
variant_get(payload, '$.location.city', 'STRING') AS city,
variant_get(payload, '$.tags[0]', 'STRING') AS first_tag
FROM s3t.myns.variant_demo
+--------+---------+-----------+--------+------+---------+
|event_id|device_id|temperature|humidity|city |first_tag|
+--------+---------+-----------+--------+------+---------+
|evt_001 |sensor_A |23.5 |62.1 |Tokyo |indoor |
|evt_002 |sensor_B |31.2 |78.5 |Osaka |outdoor |
|evt_003 |sensor_C |18.9 |45.0 |Nagoya|NULL |
|evt_001 |sensor_A |23.5 |62.1 |Tokyo |indoor |
|evt_002 |sensor_B |31.2 |78.5 |Osaka |outdoor |
|evt_003 |sensor_C |18.9 |45.0 |Nagoya|NULL |
+--------+---------+-----------+--------+------+---------+
For evt_003, which has no tags, first_tag is NULL, and the query as a whole does not fail even for rows where the key does not exist. Nested fields and array elements are also retrieved exactly as the specified paths indicate. The reason the output has two sets of rows with the same values is that I re-ran the same INSERT INTO without recreating the table, since CREATE TABLE IF NOT EXISTS does not recreate it.
The extracted values can also be used as-is in the WHERE clause. I narrowed down to the rows where the temperature exceeds 25.0.
SELECT
event_id,
device_id,
variant_get(payload, '$.temperature', 'DOUBLE') AS temperature,
variant_get(payload, '$.location.city', 'STRING') AS city
FROM s3t.myns.variant_demo
WHERE variant_get(payload, '$.temperature', 'DOUBLE') > 25.0
+--------+---------+-----------+-----+
|event_id|device_id|temperature|city |
+--------+---------+-----------+-----+
|evt_002 |sensor_B |31.2 |Osaka|
|evt_002 |sensor_B |31.2 |Osaka|
+--------+---------+-----------+-----+
You can complete everything up to numeric filtering within SQL, without a preprocessing step to parse JSON strings.
Summary
Even when the JSON key structure is not yet settled, you can start ingesting data into S3 Tables without finalizing the table definition. Because you can decide which fields to use on the SQL side after writing, it can serve as a landing place for data whose keys differ by sender, such as IoT sensor data or application event logs.
However, even the latest AWS Glue version, 5.1, uses Spark 3.5.6, so using the Variant type presupposes an execution environment like EMR where you can choose Spark 4.0.x. Also, extraction results for rows with missing keys are NULL, so some agreement on which keys are always present is needed on either the writing side or the reading side. I recommend deciding on adoption after evaluating the constraints of the execution environment and where to guarantee key management.
