Kinesis Data Streams added support for direct delivery to S3 general purpose buckets, so I tried it out
This page has been translated by machine translation. View original
Introduction
With the August 2026 update, it became possible to deliver data directly from Kinesis Data Streams to S3 general-purpose buckets. Previously, archiving to S3 generally required going through Amazon Data Firehose, but now you can deliver data simply by preparing a service execution role and creating one channel on the stream.
Comparing the traditional configuration with direct delivery:
| Item | KDS → Firehose → S3 | KDS → S3 Direct Delivery |
|---|---|---|
| Additional resources to create | Firehose stream | Channel |
| Data transformation / format conversion | Supported | Not supported |
| Minimum buffer | 60 seconds (zero buffering also available) | 300 seconds |
| Additional charges for delivery (unit price) | data-out $0.040/GB + ingestion $0.029/GB | S3 delivery $0.0275/GB |
Direct delivery charges are added on top of existing KDS charges (per-stream / data-in / retention). Firehose ingestion is billed rounded up to 5KB units. $0.029/GB applies to the first 500 TB per month. Unit prices are values from the US East calculation examples listed on the Kinesis Data Streams pricing page (S3 delivery tab) and Amazon Data Firehose pricing. Minimum buffer values are from delivery quotas and constraints for direct delivery, and buffering hints for Firehose.
In this article, we will set up a configuration that delivers GZIP / JSON from an On-Demand Standard stream to an SSE-S3 bucket. We will walk through the delivery creation steps, the S3 key structure, the contents of delivered objects, the elapsed time until the first record arrives, and where invalid records end up.
For an article on trying direct delivery to Apache Iceberg tables on S3 Tables, please see the following:
Verification Details
In ap-northeast-1, we created a channel on an On-Demand Standard stream.
Verification Environment
All operations were performed using the AWS CLI. The version used is as follows:
aws-cli/2.36.36 Python/3.14.6 Linux/7.1.6-400.asahi.fc44.aarch64+16k docker/aarch64.amzn.2023
With 2.36.36, kinesis create-channel could be executed. If the command is not found, please update the CLI.
The execution environment is Docker image public.ecr.aws/aws-cli/aws-cli:2.36.36. The digest is sha256:7047de9ebb37c39d07f652ff2ff3f6c42308a4e189788aeeae70563b0c82b57d.
The source stream configuration is as follows:
{
"StreamDescriptionSummary": {
"StreamName": "kds-s3-gp-ods-20260901-2210",
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamStatus": "ACTIVE",
"StreamModeDetails": {
"StreamMode": "ON_DEMAND"
},
"RetentionPeriodHours": 24,
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"EncryptionType": "NONE",
"OpenShardCount": 4,
"ConsumerCount": 0,
"MaxRecordSizeInKiB": 1024,
"ChannelCount": 0
}
}
Only On-Demand streams are eligible for direct delivery.
Creating the Delivery
First, create the service execution role. The trusted principal is kinesis.amazonaws.com, with conditions specifying the source account and a wildcard for the channel ARN.
{
"Role": {
"Path": "/",
"RoleName": "kds-s3-gp-role-20260901-2210",
"Arn": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"CreateDate": "2026-09-01T13:58:07+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "kinesis.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/*"
}
}
}
]
}
}
}
The permission policy includes write access to the delivery destination bucket, bucket listing, and actions for the DLQ. Delivery succeeded with this configuration. Multipart upload execution itself is permitted by s3:PutObject (CreateMultipartUpload).
Service execution role permission policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeliveryBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DeliveryBucketWrite",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:CreateMultipartUpload",
"s3:UploadPart",
"s3:CompleteMultipartUpload",
"s3:ListMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DLQBucketAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "123456789012"
}
}
}
]
}
The delivery configuration was passed together in an input JSON file.
{
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"RecordConfiguration": { "RecordFormatType": "JSON" }
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
}
}
aws kinesis create-channel \
--cli-input-json file://scripts/params/create-channel.json
The DLQ configuration is directly under S3DestinationConfiguration, not inside StorageConfiguration. Also, when specifying GZIP compression, !{extension} is required in the key template. Since no encryption is specified, the delivery destination uses S3's default SSE-S3.
create-channel response
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelARN": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/xxxxxxxxxxxxxxxxx",
"ChannelId": "xxxxxxxxxxxxxxxxx",
"ChannelStatus": "CREATING",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"RecordConfiguration": {
"RecordFormatType": "JSON"
}
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
},
"LoggingConfiguration": {
"CloudWatchLogs": {
"Enabled": false
}
}
}
}
The status immediately after creation was CREATING. The following is the status portion extracted from the response retrieved after waiting a moment.
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelStatus": "ACTIVE",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00"
}
}
From creation to observing ACTIVE was 8 seconds, from 13:58:38Z to 13:58:46Z. The create-channel response reflected the specified record format, GZIP compression, storage destination, and DLQ configuration as-is.
Key Structure
{
"Contents": [
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000000000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:15:52+00:00",
"ETag": "\"dd02b756664cd84f8b3d675727905c1d-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 66,
"StorageClass": "STANDARD"
},
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000100000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:16:17+00:00",
"ETag": "\"2b73988bd978d1be87230ceb356aff6e-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 89,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "data/"
}
The keys include year, month, day, and hour as Hive-style partitions, as specified in the key template. The template was specified up to records-, and the sequential number, unique suffix, and .gz after that are appended by the service. The valid JSON records (3 records submitted in the next section) were split into 2 objects, distinguished by keys containing sequential numbers.
Contents of Delivered Objects
3 valid JSON records and 1 non-JSON string were submitted. Only the beta record has spaces after colons and commas.
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key alpha \
--cli-binary-format raw-in-base64-out \
--data '{"message":"alpha","n":1,"nested":{"z":true}}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key beta \
--cli-binary-format raw-in-base64-out \
--data '{"message": "beta", "n": 2, "array": [1, 2, 3]}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key gamma \
--cli-binary-format raw-in-base64-out \
--data '{"message":"gamma","n":3,"null":null}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key invalid \
--cli-binary-format raw-in-base64-out \
--data 'not-json'
Decompressing and concatenating the two delivered GZIP objects yields the following. The first object (sequential number 000000000000000000) contains beta, and the second contains alpha and gamma.
{"message": "beta", "n": 2, "array": [1, 2, 3]}
{"message":"alpha","n":1,"nested":{"z":true}}
{"message":"gamma","n":3,"null":null}
To verify the delimiter, the object containing alpha and gamma was displayed with od -c.
0000000 { " m e s s a g e " : " a l p h
0000020 a " , " n " : 1 , " n e s t e d
0000040 " : { " z " : t r u e } } \r \n {
0000060 " m e s s a g e " : " g a m m a
0000100 " , " n " : 3 , " n u l l " : n
0000120 u l l } \r \n
0000126
In this object, the record delimiter was CRLF, and each record also ended with CRLF. Even though JSON was specified as the record format, no re-serialization such as whitespace removal or key reordering occurred with this input, and the formatting differences at submission time were preserved as-is.
Elapsed Time Until Delivery
The time at which the channel ACTIVE status was observed, the time the first record was submitted, and the LastModified of the first object are listed below.
channel_active_observed_utc=2026-09-01T13:58:46+00:00
first_record_put_utc=2026-09-01T13:58:46+00:00
s3_first_object_last_modified_utc=2026-09-01T14:15:52+00:00
Submission times for each of the 4 records
record_1_put_utc=2026-09-01T13:58:46+00:00
record_2_put_utc=2026-09-01T13:58:47+00:00
record_3_put_utc=2026-09-01T13:58:48+00:00
record_4_put_utc=2026-09-01T13:58:49+00:00
In this single run with one channel, it took approximately 1,026 seconds, or about 17 minutes, from the first PutRecord to the first object. Since the first PutRecord was at 13:58:46Z, the same time ACTIVE was observed, this value does not include channel startup wait time. DataFreshnessInSeconds was set to the minimum value of 300, but delivery did not occur at that exact timing. The second object was created 25 seconds after the first.
DLQ
{
"Contents": [
{
"Key": "dlq/DESERIALIZATION_ERROR/2026/09/01/13/kds-s3-gp-channel-20260901-2210-000000000000000000-000000-9d6757de83b.json",
"LastModified": "2026-09-01T14:15:52+00:00",
"Size": 372,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "dlq/"
}
Under the specified ErrorOutputPrefix, error type and timestamp partitions were automatically added. The object contents are a single-line JSON.
{"approximateArrivalTimestamp":1788271129616,"streamArn":"arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210","shardId":"shardId-xxxxxxxxxxxxxxxxx","sequenceNumber":"xxxxxxxxxxxxxxxxx","errorCode":"Record.InvalidJson","errorMessage":"The record contains invalid JSON. Please verify the record is valid JSON."}
In the object confirmed this time, the stream ARN, shard ID, sequence number, arrival timestamp, and error details were output as JSON, but the original record data was not included. The original data cannot be restored from the DLQ contents alone.
Conditions and Constraints for Using S3 Direct Delivery
To deliver data directly from Kinesis Data Streams to S3, the following conditions and constraints apply. Check the Developer Guide and delivery quotas and constraints beforehand to verify whether your stream is eligible.
- Not available if the source stream is encrypted with an AWS managed key (
aws/kinesis). Migration to a CMK is a prerequisite. - Only On-Demand capacity mode is supported; Provisioned streams are not eligible.
- After creation, only DataFreshnessInSeconds and LoggingConfiguration can be updated. Compression, key template, record format, and destination bucket cannot be changed; to change them, you must delete and recreate. There is also no backfill of existing data.
- Up to 2 deliveries can be created per stream: 1 for Iceberg and 1 for general-purpose S3. This limit cannot be increased by request.
- Variables available in the key template are limited to timestamp, channel name, and stream name. Partitioning based on record content is not possible.
- Cross-region delivery is not supported. Only the destination bucket can be in a different account.
Positioning Relative to Firehose
Even when direct delivery is available, there are features exclusive to Firehose. These include data transformation via Lambda, format conversion to Parquet / ORC, dynamic partitioning based on record content, freshness under 60 seconds, and delivery from a single source to multiple destinations. Conversely, features exclusive to direct delivery include ZSTD compression and specification of the delivery storage class (STANDARD / INTELLIGENT_TIERING / GLACIER_IR).
Summary
We successfully delivered data directly from Kinesis Data Streams to a general-purpose S3 bucket without going through Firehose. In the verification, we confirmed output with Hive-style keys, delivery of JSON records, and DLQ output of invalid JSON records.
If you do not need Firehose features or Lambda-based transformation, try the newly added direct delivery.
If your goal is archiving with reprocessing or testing in mind, there are cases where designing S3 as the starting point for reprocessing is more manageable than extending the Kinesis Data Streams retention period. For long-term data retention purposes as well, please consider evaluating direct delivery.
