I tried the new "Variant data type" feature of S3 Tables with EMR Serverless
This page has been translated by machine translation. View original
Introduction
On July 28, 2026, S3 Tables added support for the Variant data type in Apache Iceberg V3. The Variant type can store nested JSON as-is in a single column, allowing you to handle semi-structured data without defining a schema in advance. At read time, you can extract only the fields you need with type information using JSONPath-style paths. The What's New announcement also mentions file pruning effects through shredding, but this article does not verify that.
Verification Details
Verification Environment
Spark 4.0.x is required for DDL and DML with the Variant type. The VARIANT data type was added in Apache Spark 4.0.0. This time, I used EMR Serverless emr-spark-8.0.0, which includes Spark 4.0.2 and Iceberg 1.10.1, in the Tokyo region.
The connection to S3 Tables goes through the Glue Iceberg REST endpoint. The catalog configuration on the Spark side is as follows.
spark.sql.catalog.s3t = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.s3t.type = rest
spark.sql.catalog.s3t.uri = https://glue.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3t.warehouse = 123456789012:s3tablescatalog/variant-test-bucket
spark.sql.catalog.s3t.rest.sigv4-enabled = true
spark.sql.catalog.s3t.rest.signing-name = glue
spark.sql.catalog.s3t.rest.signing-region = ap-northeast-1
The warehouse is specified not as an ARN, but in the format of the account ID followed by s3tablescatalog/<bucket-name>.
Table Creation
I created a table with payload declared as the VARIANT type. Since the Variant type was added in V3, you need to specify 3 for format-version. How to create V3 tables is summarized in Working with Apache Iceberg V3.
CREATE TABLE IF NOT EXISTS s3t.myns.variant_demo (
event_id STRING NOT NULL,
device_id STRING NOT NULL,
event_ts TIMESTAMP NOT NULL,
payload VARIANT NOT NULL
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
Writing Data
Passing a JSON string to parse_json() converts it to VARIANT. I inserted three records simulating sensor data. The third record has no tags key.
INSERT INTO s3t.myns.variant_demo VALUES
('evt_001', 'sensor_A', current_timestamp(),
parse_json('{"temperature": 23.5, "humidity": 62.1, "location": {"city": "Tokyo", "building": "HQ"}, "tags": ["indoor", "floor3"]}')),
('evt_002', 'sensor_B', current_timestamp(),
parse_json('{"temperature": 31.2, "humidity": 78.5, "location": {"city": "Osaka", "building": "Lab"}, "tags": ["outdoor"]}')),
('evt_003', 'sensor_C', current_timestamp(),
parse_json('{"temperature": 18.9, "humidity": 45.0, "location": {"city": "Nagoya", "building": "DC"}}'))
After writing, I read the table and checked the DataFrame schema.
root
|-- event_id: string (nullable = false)
|-- device_id: string (nullable = false)
|-- event_ts: timestamp (nullable = false)
|-- payload: variant (nullable = false)
The payload is recognized as variant.
Reading Data and Filtering
variant_get() takes three arguments: the target column, a JSONPath-style path, and the type to extract.
SELECT
event_id,
device_id,
variant_get(payload, '$.temperature', 'DOUBLE') AS temperature,
variant_get(payload, '$.humidity', 'DOUBLE') AS humidity,
variant_get(payload, '$.location.city', 'STRING') AS city,
variant_get(payload, '$.tags[0]', 'STRING') AS first_tag
FROM s3t.myns.variant_demo
+--------+---------+-----------+--------+------+---------+
|event_id|device_id|temperature|humidity|city |first_tag|
+--------+---------+-----------+--------+------+---------+
|evt_001 |sensor_A |23.5 |62.1 |Tokyo |indoor |
|evt_002 |sensor_B |31.2 |78.5 |Osaka |outdoor |
|evt_003 |sensor_C |18.9 |45.0 |Nagoya|NULL |
|evt_001 |sensor_A |23.5 |62.1 |Tokyo |indoor |
|evt_002 |sensor_B |31.2 |78.5 |Osaka |outdoor |
|evt_003 |sensor_C |18.9 |45.0 |Nagoya|NULL |
+--------+---------+-----------+--------+------+---------+
The first_tag for evt_003, which has no tags, becomes NULL, and the entire query does not fail even for rows where the key is absent. Nested fields and array elements are also retrieved as specified by the given path. The reason the same values appear in two sets of rows is that the same INSERT INTO was re-executed without recreating the table using CREATE TABLE IF NOT EXISTS.
The extracted values can also be used directly in a WHERE clause. I filtered for rows where the temperature exceeds 25.0.
SELECT
event_id,
device_id,
variant_get(payload, '$.temperature', 'DOUBLE') AS temperature,
variant_get(payload, '$.location.city', 'STRING') AS city
FROM s3t.myns.variant_demo
WHERE variant_get(payload, '$.temperature', 'DOUBLE') > 25.0
+--------+---------+-----------+-----+
|event_id|device_id|temperature|city |
+--------+---------+-----------+-----+
|evt_002 |sensor_B |31.2 |Osaka|
|evt_002 |sensor_B |31.2 |Osaka|
+--------+---------+-----------+-----+
Numeric filtering can be completed entirely within SQL without any preprocessing to parse the JSON string.
Summary
Even at a stage where the key structure of the JSON is not yet settled, you can start ingesting data into S3 Tables without finalizing the table definition. Since you can decide which fields to use on the SQL side after writing, it can serve as a landing zone for data where keys differ per source, such as IoT sensor data or application event logs.
However, the AWS Glue version is Spark 3.5.6 even in the latest 5.1, so using the Variant type requires an execution environment where you can choose Spark 4.0.x, such as EMR. Also, since extraction results for rows with missing keys become NULL, some agreement about which keys are guaranteed to exist is needed either on the write side or the read side. It is recommended to evaluate the constraints of your execution environment and where key management will be enforced before deciding to adopt this approach.
