Kinesis Data Streams Now Supports Direct Delivery to S3 General Purpose Buckets, So I Tried It Out
This page has been translated by machine translation. View original
Introduction
The August 2026 update made it possible to deliver data directly from Kinesis Data Streams to S3 general purpose buckets. Until now, archiving to S3 generally went through Amazon Data Firehose, but now you can deliver data just by preparing a service execution role and creating one channel on the stream.
Here is a side-by-side comparison of the conventional setup and direct delivery.
| Item | KDS → Firehose → S3 | KDS → S3 direct delivery |
|---|---|---|
| Additional resources to create | Firehose stream | Channel |
| Data transformation / format conversion | Supported | Not supported |
| Minimum buffer | 60 seconds (zero buffering can also be selected) | 300 seconds |
| Additional delivery charges (unit price) | data-out $0.040/GB + ingestion $0.029/GB | S3 delivery $0.0275/GB |
The direct delivery charge is added on top of the existing KDS charges (per-stream / data-in / retention). Firehose ingestion is billed rounded up to 5KB units. $0.029/GB is the unit price applied to the first 500 TB per month. The unit prices are the values from the US East calculation examples on Kinesis Data Streams pricing (S3 delivery tab) and Amazon Data Firehose pricing. The minimum buffer values are those listed in delivery quotas and restrictions for direct delivery and in buffering hints for Firehose.
In this article, I build a setup that delivers from an On-Demand Standard stream to an SSE-S3 bucket in GZIP / JSON. I check the delivery creation steps, the S3 key structure, the contents of the delivered objects, the elapsed time until the first record arrives, and where invalid records go.
For an article that tried direct delivery to an Apache Iceberg table on S3 Tables, see below.
What I tested
In ap-northeast-1, I created a channel on an On-Demand Standard stream.
Test environment
All operations were run with the AWS CLI. The version used is as follows.
aws-cli/2.36.36 Python/3.14.6 Linux/7.1.6-400.asahi.fc44.aarch64+16k docker/aarch64.amzn.2023
With 2.36.36, I was able to run kinesis create-channel. If the command is not found, update your CLI.
The execution environment is the Docker image public.ecr.aws/aws-cli/aws-cli:2.36.36. The digest is sha256:7047de9ebb37c39d07f652ff2ff3f6c42308a4e189788aeeae70563b0c82b57d.
The source stream settings are as follows.
{
"StreamDescriptionSummary": {
"StreamName": "kds-s3-gp-ods-20260901-2210",
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamStatus": "ACTIVE",
"StreamModeDetails": {
"StreamMode": "ON_DEMAND"
},
"RetentionPeriodHours": 24,
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"EncryptionType": "NONE",
"OpenShardCount": 4,
"ConsumerCount": 0,
"MaxRecordSizeInKiB": 1024,
"ChannelCount": 0
}
}
Only On-Demand streams are eligible for direct delivery.
Creating the delivery
First, create the service execution role. The trusted principal is kinesis.amazonaws.com, with conditions on the source account and a wildcard of the channel ARN.
{
"Role": {
"Path": "/",
"RoleName": "kds-s3-gp-role-20260901-2210",
"Arn": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"CreateDate": "2026-09-01T13:58:07+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "kinesis.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/*"
}
}
}
]
}
}
}
The permissions policy includes writing to the delivery destination bucket, listing the bucket, and actions for the DLQ. Delivery succeeded with this configuration. Executing multipart uploads itself is permitted by s3:PutObject (CreateMultipartUpload).
Permissions policy of the service execution role
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeliveryBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DeliveryBucketWrite",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:CreateMultipartUpload",
"s3:UploadPart",
"s3:CompleteMultipartUpload",
"s3:ListMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DLQBucketAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "123456789012"
}
}
}
]
}
I passed the delivery settings together in an input JSON.
{
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"RecordConfiguration": { "RecordFormatType": "JSON" }
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
}
}
aws kinesis create-channel \
--cli-input-json file://scripts/params/create-channel.json
The DLQ settings are directly under S3DestinationConfiguration, not inside StorageConfiguration. Also, when specifying GZIP for compression, the key template needs !{extension}. Since no encryption was specified, the delivery destination uses the S3 default, SSE-S3.
create-channel response
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelARN": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/xxxxxxxxxxxxxxxxx",
"ChannelId": "xxxxxxxxxxxxxxxxx",
"ChannelStatus": "CREATING",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"RecordConfiguration": {
"RecordFormatType": "JSON"
}
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
},
"LoggingConfiguration": {
"CloudWatchLogs": {
"Enabled": false
}
}
}
}
The status immediately after creation was CREATING. Below is the status portion extracted from the response I retrieved after waiting a little.
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelStatus": "ACTIVE",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00"
}
}
From creation to observing ACTIVE took 8 seconds, from 13:58:38Z to 13:58:46Z. The create-channel response reflected the specified record format, GZIP compression, and the destination and DLQ settings as they were.
Key structure
{
"Contents": [
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000000000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:15:52+00:00",
"ETag": "\"dd02b756664cd84f8b3d675727905c1d-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 66,
"StorageClass": "STANDARD"
},
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000100000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:16:17+00:00",
"ETag": "\"2b73988bd978d1be87230ceb356aff6e-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 89,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "data/"
}
As specified in the key template, the keys include the year, month, day, and hour as Hive-style partitions. What I specified in the template ends at records-; the sequence number, unique suffix, and .gz that follow are added by the service. The valid JSON records (the 3 sent in the next section) were split into two objects, distinguished by keys containing sequence numbers.
Contents of the delivered objects
I sent 3 valid JSON records and 1 non-JSON string. Only the beta record has spaces after the colons and commas.
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key alpha \
--cli-binary-format raw-in-base64-out \
--data '{"message":"alpha","n":1,"nested":{"z":true}}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key beta \
--cli-binary-format raw-in-base64-out \
--data '{"message": "beta", "n": 2, "array": [1, 2, 3]}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key gamma \
--cli-binary-format raw-in-base64-out \
--data '{"message":"gamma","n":3,"null":null}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key invalid \
--cli-binary-format raw-in-base64-out \
--data 'not-json'
When I decompressed the 2 delivered objects with GZIP and concatenated them, the result was as follows. The first (sequence number 000000000000000000) contains beta, and the second contains alpha and gamma.
{"message": "beta", "n": 2, "array": [1, 2, 3]}
{"message":"alpha","n":1,"nested":{"z":true}}
{"message":"gamma","n":3,"null":null}
To check the delimiter, I displayed the object containing alpha and gamma with od -c.
0000000 { " m e s s a g e " : " a l p h
0000020 a " , " n " : 1 , " n e s t e d
0000040 " : { " z " : t r u e } } \r \n {
0000060 " m e s s a g e " : " g a m m a
0000100 " , " n " : 3 , " n u l l " : n
0000120 u l l } \r \n
0000126
In this object, the record delimiter was CRLF, and each record also had a CRLF at its end. Even with JSON specified as the record format, no re-serialization such as whitespace removal or key reordering occurred with this input, and the notation differences from when the records were sent were preserved as they were.
Elapsed time until delivery
I lined up the time ACTIVE was observed for the channel, the time the first record was sent, and the LastModified of the first object.
channel_active_observed_utc=2026-09-01T13:58:46+00:00
first_record_put_utc=2026-09-01T13:58:46+00:00
s3_first_object_last_modified_utc=2026-09-01T14:15:52+00:00
Send time of each of the 4 records
record_1_put_utc=2026-09-01T13:58:46+00:00
record_2_put_utc=2026-09-01T13:58:47+00:00
record_3_put_utc=2026-09-01T13:58:48+00:00
record_4_put_utc=2026-09-01T13:58:49+00:00
In this result, from a single run on one channel, it took about 1,026 seconds, roughly 17 minutes, from the first PutRecord to the first object. Since the first PutRecord was at 13:58:46Z, the same time ACTIVE was observed, the value does not include the channel startup wait time. I set DataFreshnessInSeconds to the minimum value of 300, but delivery did not happen at the timing of this value. The second object was created 25 seconds after the first.
DLQ
{
"Contents": [
{
"Key": "dlq/DESERIALIZATION_ERROR/2026/09/01/13/kds-s3-gp-channel-20260901-2210-000000000000000000-000000-9d6757de83b.json",
"LastModified": "2026-09-01T14:15:52+00:00",
"Size": 372,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "dlq/"
}
Under the specified ErrorOutputPrefix, partitions by error type and time were added automatically. The contents of the object are a single line of JSON.
{"approximateArrivalTimestamp":1788271129616,"streamArn":"arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210","shardId":"shardId-xxxxxxxxxxxxxxxxx","sequenceNumber":"xxxxxxxxxxxxxxxxx","errorCode":"Record.InvalidJson","errorMessage":"The record contains invalid JSON. Please verify the record is valid JSON."}
In the object I was able to check, the stream ARN, shard ID, sequence number, arrival time, and error details were output as JSON, and the record body was not included. The original data cannot be restored from the DLQ contents alone.
Conditions and restrictions for using S3 direct delivery
Direct delivery from Kinesis Data Streams to S3 has the following conditions and restrictions. Check the developer guide and delivery quotas and restrictions first to see whether your stream is eligible.
- It cannot be used if the source stream is encrypted with an AWS managed key (
aws/kinesis). Migrating to a CMK is a prerequisite - Only the On-Demand capacity mode is supported; Provisioned streams are not eligible
- After creation, only DataFreshnessInSeconds and LoggingConfiguration can be updated. Compression, key template, record format, and destination bucket cannot be changed, so to change them you must delete and recreate. There is also no backfill of existing data
- A single stream can have up to 2 deliveries: 1 for Iceberg and 1 for general purpose S3. A quota increase request is not possible
- The only variables available in the key template are time, channel name, and stream name. Partitioning based on record contents is not possible
- Cross-region delivery is not possible. Only the destination bucket can be in a different account
Division of roles with Firehose
Even when direct delivery is available, some features exist only on the Firehose side. These are data transformation with Lambda, format conversion to Parquet / ORC, dynamic partitioning based on record contents, freshness of under 60 seconds, and delivery from one source to multiple destinations. Conversely, what exists only on the direct delivery side is ZSTD compression and specifying the storage class of the destination (STANDARD / INTELLIGENT_TIERING / GLACIER_IR).
Summary
I was able to deliver directly from Kinesis Data Streams to an S3 general purpose bucket without going through Firehose. In this test, I confirmed output with Hive-style keys, delivery of JSON records, and DLQ output of invalid JSON records.
If you don't need Firehose features or Lambda transformation, try the newly added direct delivery.
If your goal is archiving with reprocessing or testing in mind, there are cases where designing S3 as the starting point for reprocessing is easier to handle than extending the Kinesis Data Streams retention period. If your purpose is long-term data retention, please also consider evaluating direct delivery.
