I tried ingesting 10 MiB records from Kinesis Data Streams using Amazon Redshift streaming ingestion
This page has been translated by machine translation. View original
Introduction
Amazon Redshift streaming ingestion now supports 10 MiB records for Amazon Kinesis Data Streams, as of the August 27, 2026 update.
The increase in the maximum record size on the Kinesis Data Streams side is covered in the following article.
In this article, we inserted dummy JSON ranging from 100 KiB to 10 MiB + 1,000 B into a stream configured with a maximum record size of 10 MiB. We verify the sizes that could be ingested and the constraints when referencing them via SQL in Redshift.
Verification Details
We proceeded in two stages: inserting into the stream, and ingesting and referencing on the Redshift side.
Verification Environment
The regions where Kinesis large records can be used are limited. The us-east-1 region used this time is included in the Regions where large records are supported section of Handle large records. Tokyo (ap-northeast-1) and Osaka (ap-northeast-3) are also supported regions.
A provisioned cluster was used for Redshift. It has a 2-node configuration with node type rg.xlarge, running version 1.0.416217.
Inserting Large Records
We created a stream with the maximum record size set to 10 MiB (specified in KiB units via the CLI).
aws kinesis create-stream \
--stream-name devio-large-record-stream \
--stream-mode-details StreamMode=ON_DEMAND \
--max-record-size-in-ki-b 10240 \
--region us-east-1
As stated in Handle large records, the default maximum record size is 1 MiB when --max-record-size-in-ki-b is not specified.
DescribeStreamSummary output
{
"StreamDescriptionSummary": {
"StreamName": "devio-large-record-stream",
"StreamARN": "arn:aws:kinesis:us-east-1:123456789012:stream/devio-large-record-stream",
"StreamStatus": "ACTIVE",
"StreamModeDetails": {
"StreamMode": "ON_DEMAND"
},
"RetentionPeriodHours": 24,
"OpenShardCount": 4,
"ConsumerCount": 0,
"MaxRecordSizeInKiB": 10240
}
}
Records were inserted one at a time using PutRecord.
aws kinesis put-record \
--stream-name devio-large-record-stream \
--partition-key 13 \
--data fileb://records/r13.json \
--region us-east-1
Files prepared for each size are passed via --data fileb://. For records of 2 MiB or larger, we set a 30-second interval between insertions and configured exponential backoff retry in case of throttling.
We inserted a total of 20 records, varying the size incrementally from 100 KiB to 10 MiB + 1,000 B.
| Size | Bytes | Count | Result |
|---|---|---|---|
| 100 KiB | 102,400 | 2 | Success |
| 1 MiB | 1,048,576 | 2 | Success |
| 1 MiB + 100 B | 1,048,676 | 2 | Success |
| 2 MiB | 2,097,152 | 1 | Success |
| 5 MiB | 5,242,880 | 1 | Success |
| 9 MiB | 9,437,184 | 1 | Success |
| 10 MiB − 1,000 B | 10,484,760 | 1 | Success |
| 10 MiB − 100 B | 10,485,660 | 2 | Success |
| 10 MiB | 10,485,760 | 4 | Success |
| 10 MiB + 100 B | 10,485,860 | 3 | ValidationException |
| 10 MiB + 1,000 B | 10,486,760 | 1 | ValidationException |
16 records succeeded and 4 failed. The dummy JSON follows the format {"id":..,"label":..,"bytes":..,"payload":"<padding>"}, with the padding length adjusted to exactly match the target byte count.
For records exceeding the limit, the following error was returned.
An error occurred (ValidationException) when calling the PutRecord operation: 1 validation error detected: Value at 'data' failed to satisfy constraint: Member must have length less than or equal to 10485760
In this verification, records with a data portion of exactly 10 MiB succeeded, while those exceeding it by 100 bytes were rejected. The AWS PutRecord API reference states that the combined total of the data before base64 encoding and the partition key must not exceed the maximum record size. However, the partition keys used this time were 1–2 bytes, yet all 4 records with exactly 10 MiB of data succeeded. Since this differs from the published specification, we record it here as an empirical result. No retries due to ProvisionedThroughputExceededException occurred for any of the 20 records.
All sizes mentioned above are raw byte lengths. As noted under Data payload size in Quotas and limits, the Kinesis limit is defined against the data payload before base64 encoding. Data passed to the AWS CLI via a file is sent base64-encoded, but since 10,485,760-byte files succeeded, we can confirm that the evaluation is based on raw byte length. Note that the dummy JSON consists solely of hex strings and ASCII symbols, so the character count and byte count are identical.
Ingestion and Reference
We created a materialized view for streaming ingestion targeting the stream we inserted into, and confirmed the column definitions.
The creation procedure follows Getting started with streaming ingestion from Amazon Kinesis Data Streams. Substituting the object names used this time into the documented procedure yields the following SQL.
CREATE EXTERNAL SCHEMA kds
FROM KINESIS
IAM_ROLE 'arn:aws:iam::123456789012:role/devio-redshift-streaming-role';
CREATE MATERIALIZED VIEW mv_large_record AS
SELECT approximate_arrival_timestamp,
partition_key,
sequence_number,
kinesis_data
FROM kds."devio-large-record-stream";
REFRESH MATERIALIZED VIEW mv_large_record;
The IAM role is granted permissions to read from the stream (such as kinesis:GetRecords). Without specifying AUTO REFRESH YES, refresh is manual. From the columns available for streaming ingestion reference, we selected the following 4 columns.
| Column | Type |
|---|---|
| approximate_arrival_timestamp | timestamp without time zone |
| partition_key | character varying(256) |
| sequence_number | character varying(128) |
| kinesis_data | binary varying(16777216) |
The width of kinesis_data, which receives the payload, is 16,777,216 bytes, i.e., 16 MiB. Since this exceeds the Kinesis limit of 10 MiB, any record that can be inserted into Kinesis fits within the column width.
We counted the ingested records by byte count.
| Bytes | Count |
|---|---|
| 102,400 | 2 |
| 1,048,576 | 2 |
| 1,048,676 | 2 |
| 2,097,152 | 1 |
| 5,242,880 | 1 |
| 9,437,184 | 1 |
| 10,484,760 | 1 |
| 10,485,660 | 2 |
| 10,485,760 | 4 |
The total is 16 records, 94,575,640 bytes. The number of records and bytes ingested exactly matched those successfully inserted, with no truncation. sys_stream_scan_errors showed 0 rows.
We referenced the data via SQL using the Redshift Data API. The first 64 bytes were extracted to retrieve the leading fields from the JSON, and the last 14 bytes were also extracted.
SELECT octet_length(kinesis_data) AS bytes,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"id":"[^"]+"') AS id_field,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"label":"[^"]+"') AS label_field,
regexp_substr(from_varbyte(substring(kinesis_data,1,64),'utf8'),'"bytes":[0-9]+') AS bytes_field,
from_varbyte(substring(kinesis_data, octet_length(kinesis_data)-13, 14),'utf8') AS tail
FROM mv_large_record
ORDER BY bytes DESC, id_field
From the 16-row result, we excerpt 2 of the 4 rows at 10,485,760 bytes and 1 row at 10,485,660 bytes.
| bytes | id_field | label_field | bytes_field | tail |
|---|---|---|---|---|
| 10485760 | "id":"r13" | "label":"10MiB" | "bytes":10485760 | 231e71767334"} |
| 10485760 | "id":"r14" | "label":"10MiB" | "bytes":10485760 | 6523963b106e"} |
| 10485660 | "id":"r11" | "label":"10MiB-100B" | "bytes":10485660 | ec344bcccf94"} |
In addition to the byte counts matching, the closing JSON symbol "} remained at the tail. Even the 10 MiB payloads were stored with their original byte counts intact.
On the other hand, for records exceeding 65,535 bytes, attempting to convert the entire column value to a string fails. The query tested was json_extract_path_text(from_varbyte(kinesis_data,'utf8'),'label'). When executed against a 10 MiB record, the following error occurred. The query ID and process ID lines are omitted.
ERROR: Invalid input
Detail:
-----------------------------------------------
error: Invalid input
code: 8001
context: result size 10485760 is too long for type VARCHAR
location: varbyte.cpp:87
-----------------------------------------------
Running the same query against a 100 KiB record also failed.
context: result size 102400 is too long for type VARCHAR
In this verification, as a simple test to inspect the content, we converted a portion of the VARBYTE to VARCHAR.
| Expression executed | Result |
|---|---|
from_varbyte(substring(kinesis_data,1,65535),'utf8') |
Success |
from_varbyte(substring(kinesis_data,1,65536),'utf8') |
Error |
The maximum size that can be handled as a VARCHAR value in Redshift is 65,535 bytes. Since the entire kinesis_data exceeding this cannot be converted to VARCHAR, we extracted the first 64 bytes as VARBYTE and retrieved the fields placed at the beginning of the JSON.
Although not verified this time, the "Data parsing best practices" section of Streaming ingestion describes an alternative approach: converting the JSON payload to the SUPER type using JSON_PARSE(kinesis_data) and referencing it with PartiQL. Since the SUPER type can hold up to 16 MB per object, a 10 MiB record falls within the size limit.
Summary
With Amazon Kinesis Data Streams, it is now possible to raise the maximum record size for a stream from the default 1 MiB up to 10 MiB. Amazon Redshift streaming ingestion also now supports 10 MiB records. In this verification, we inserted records of exactly 10 MiB into Kinesis and confirmed that Redshift can ingest them without truncation into a VARBYTE column with a width of 16 MiB.
There have been workloads that selected Amazon MSK or relayed through Amazon S3 due to the 1 MiB record size limit. For such configurations, direct ingestion from Kinesis into Redshift becomes a viable option. Depending on the requirements, it is possible to construct a simpler ELT pipeline that reduces relay storage and pre-processing, consolidating transformations into SQL on Redshift.
Additionally, in Redshift patch P203 and later, refresh of streaming materialized views connected to Kinesis Data Streams is eligible for concurrency scaling.
In addition to the increased capacity up to 10 MiB, options for scaling the ingestion process are now in place as well. It seems increasingly feasible to consider a near-real-time analytics infrastructure centered on Kinesis Data Streams and Redshift for a broader range of workloads than before.
Reference Links
