I Tried Ingesting 10 MiB Records from Kinesis Data Streams with Amazon Redshift Streaming Ingestion
This page has been translated by machine translation. View original
Introduction
With the update on August 27, 2026, Amazon Redshift streaming ingestion now supports 10 MiB records from Amazon Kinesis Data Streams.
The increase in the maximum record size on the Kinesis Data Streams side is covered in the following article.
In this article, I put dummy JSON records ranging from 100 KiB to 10 MiB + 1,000 B into a stream whose maximum record size was set to 10 MiB. I check which sizes could be ingested and what constraints apply when querying them with Redshift SQL.
What I Tested
I proceeded in two stages: putting records into the stream, and ingesting and querying them on the Redshift side.
Test Environment
The regions where Kinesis large records can be used are limited. The us-east-1 region used here is included in Regions where large records are supported in Handle large records. Tokyo (ap-northeast-1) and Osaka (ap-northeast-3) are also supported regions.
For Redshift, I used a provisioned cluster. It was a 2-node configuration with node type rg.xlarge, and the version was 1.0.416217.
Putting Large Records
I created a stream with the maximum record size set to 10 MiB (the CLI option is specified in KiB).
aws kinesis create-stream \
--stream-name devio-large-record-stream \
--stream-mode-details StreamMode=ON_DEMAND \
--max-record-size-in-ki-b 10240 \
--region us-east-1
As described in Handle large records, if --max-record-size-in-ki-b is not specified, the maximum record size is the default of 1 MiB.
DescribeStreamSummary output
{
"StreamDescriptionSummary": {
"StreamName": "devio-large-record-stream",
"StreamARN": "arn:aws:kinesis:us-east-1:123456789012:stream/devio-large-record-stream",
"StreamStatus": "ACTIVE",
"StreamModeDetails": {
"StreamMode": "ON_DEMAND"
},
"RetentionPeriodHours": 24,
"OpenShardCount": 4,
"ConsumerCount": 0,
"MaxRecordSizeInKiB": 10240
}
}
I put the records one at a time using PutRecord.
aws kinesis put-record \
--stream-name devio-large-record-stream \
--partition-key 13 \
--data fileb://records/r13.json \
--region us-east-1
I passed a file prepared for each size with --data fileb://. For records of 2 MiB or more, I left a 30-second interval between puts, and set it to retry with exponential backoff if throttled.
I varied the size step by step from 100 KiB to 10 MiB + 1,000 B, for a total of 20 puts.
| Size | Bytes | Count | Result |
|---|---|---|---|
| 100 KiB | 102,400 | 2 | Success |
| 1 MiB | 1,048,576 | 2 | Success |
| 1 MiB + 100 B | 1,048,676 | 2 | Success |
| 2 MiB | 2,097,152 | 1 | Success |
| 5 MiB | 5,242,880 | 1 | Success |
| 9 MiB | 9,437,184 | 1 | Success |
| 10 MiB − 1,000 B | 10,484,760 | 1 | Success |
| 10 MiB − 100 B | 10,485,660 | 2 | Success |
| 10 MiB | 10,485,760 | 4 | Success |
| 10 MiB + 100 B | 10,485,860 | 3 | ValidationException |
| 10 MiB + 1,000 B | 10,486,760 | 1 | ValidationException |
There were 16 successes and 4 failures. The dummy JSON has the format {"id":..,"label":..,"bytes":..,"payload":"<padding>"}, and the padding length was adjusted so that the record matches the target byte count exactly.
Records exceeding the limit returned the following error.
An error occurred (ValidationException) when calling the PutRecord operation: 1 validation error detected: Value at 'data' failed to satisfy constraint: Member must have length less than or equal to 10485760
In this test, a data portion of exactly 10 MiB succeeded, and exceeding it by 100 bytes was rejected. The AWS PutRecord API reference explains that the total of the data before base64 encoding and the partition key must be less than or equal to the maximum record size. However, although the partition keys used here were 1 to 2 bytes, all four records with exactly 10 MiB of data succeeded. Because this result differs from the published specification, I report it in this article as a measured result. No retries due to ProvisionedThroughputExceededException occurred in any of the 20 puts.
All sizes so far are raw byte lengths. As stated under Data payload size in Quotas and limits, the Kinesis limit is defined on the data payload before base64 encoding. Data passed to the AWS CLI as a file is base64-encoded when sent, but since the 10,485,760-byte file succeeded, the check is evidently based on the raw byte length. Note that the dummy JSON consists only of hex strings and ASCII symbols, so the character count and byte count match.
Ingestion and Querying
I created a materialized view for streaming ingestion on the stream I had put records into, and checked the column definitions.
The creation steps follow Getting started with streaming ingestion from Amazon Kinesis Data Streams. Applying the object names used here to the documented steps gives the following SQL.
CREATE EXTERNAL SCHEMA kds
FROM KINESIS
IAM_ROLE 'arn:aws:iam::123456789012:role/devio-redshift-streaming-role';
CREATE MATERIALIZED VIEW mv_large_record AS
SELECT approximate_arrival_timestamp,
partition_key,
sequence_number,
kinesis_data
FROM kds."devio-large-record-stream";
REFRESH MATERIALIZED VIEW mv_large_record;
The IAM role was granted permissions to read the stream (such as kinesis:GetRecords). If AUTO REFRESH YES is not specified, refresh is manual. Of the columns available in streaming ingestion, I selected the following four.
| Column | Type |
|---|---|
| approximate_arrival_timestamp | timestamp without time zone |
| partition_key | character varying(256) |
| sequence_number | character varying(128) |
| kinesis_data | binary varying(16777216) |
The width of kinesis_data, which receives the payload, is 16,777,216 bytes, that is, 16 MiB. Since this exceeds the Kinesis limit of 10 MiB, any record that can be put into Kinesis fits within the column width.
I counted the ingested records by byte count.
| Bytes | Count |
|---|---|
| 102,400 | 2 |
| 1,048,576 | 2 |
| 1,048,676 | 2 |
| 2,097,152 | 1 |
| 5,242,880 | 1 |
| 9,437,184 | 1 |
| 10,484,760 | 1 |
| 10,485,660 | 2 |
| 10,485,760 | 4 |
The total was 16 records and 94,575,640 bytes. The count and bytes of the successfully put records and the ingested records matched exactly, with no truncation. sys_stream_scan_errors had 0 rows.
I queried them with SQL using the Redshift Data API. I extracted the first 64 bytes to pull out the leading fields of the JSON, and also extracted the last 14 bytes.
SELECT octet_length(kinesis_data) AS bytes,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"id":"[^"]+"') AS id_field,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"label":"[^"]+"') AS label_field,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"bytes":[0-9]+') AS bytes_field,
from_varbyte(substring(kinesis_data, octet_length(kinesis_data)-13, 14),'utf8') AS tail
FROM mv_large_record
ORDER BY bytes DESC, id_field
From the 16-row result, I excerpted 2 of the 4 rows of 10,485,760 bytes and 1 row of 10,485,660 bytes.
| bytes | id_field | label_field | bytes_field | tail |
|---|---|---|---|---|
| 10485760 | "id":"r13" | "label":"10MiB" | "bytes":10485760 | 231e71767334"} |
| 10485760 | "id":"r14" | "label":"10MiB" | "bytes":10485760 | 6523963b106e"} |
| 10485660 | "id":"r11" | "label":"10MiB-100B" | "bytes":10485660 | ec344bcccf94"} |
In addition to the byte counts matching, the closing JSON characters "} remained at the end. The 10 MiB payloads were stored with their original byte counts.
On the other hand, for records larger than 65,535 bytes, trying to convert the entire column value to a string fails. The query I tried was json_extract_path_text(from_varbyte(kinesis_data,'utf8'),'label'). Running it against a 10 MiB record produced the following error. The lines for the query ID and process ID are omitted.
ERROR: Invalid input
Detail:
-----------------------------------------------
error: Invalid input
code: 8001
context: result size 10485760 is too long for type VARCHAR
location: varbyte.cpp:87
-----------------------------------------------
Running the same query against a 100 KiB record also failed.
context: result size 102400 is too long for type VARCHAR
As a simple test to check the contents, I converted part of the VARBYTE to VARCHAR.
| Expression run | Result |
|---|---|
from_varbyte(substring(kinesis_data,1,65535),'utf8') |
Success |
from_varbyte(substring(kinesis_data,1,65536),'utf8') |
Error |
The maximum size that can be handled as a Redshift VARCHAR value is 65,535 bytes. Since the entire kinesis_data exceeding this cannot be converted to VARCHAR, here I kept it as VARBYTE, extracted the first 64 bytes, and retrieved the fields placed at the beginning of the JSON.
Although I did not test it here, the "Data parsing best practices" section of Streaming ingestion describes another method: converting the JSON payload to the SUPER type with JSON_PARSE(kinesis_data) and referencing it with PartiQL. The SUPER type can hold up to 16 MB per object, so a 10 MiB record is within the size limit.
Summary
In Amazon Kinesis Data Streams, the maximum record size of a stream can now be raised from the default of 1 MiB up to 10 MiB. Amazon Redshift streaming ingestion also supports 10 MiB records. In this test, I put records of exactly 10 MiB into Kinesis and confirmed that Redshift could ingest them into a 16 MiB-wide VARBYTE column without truncation.
There are workloads that until now chose Amazon MSK or relayed data through Amazon S3 because of the 1 MiB record size limit. For such setups, ingesting directly from Kinesis into Redshift also becomes an option. Depending on the requirements, you can reduce intermediate storage and preprocessing, and build a simpler ELT pipeline that shifts transformation to SQL on Redshift.
In addition, in Redshift with patch P203 or later, refreshing streaming materialized views connected to Kinesis Data Streams is now eligible for concurrency scaling.
In addition to the larger 10 MiB capacity, options for scaling the ingestion processing are now in place as well. It looks like near-real-time analytics platforms centered on Kinesis Data Streams and Redshift can be considered for a wider range of workloads than before.
References
