Kinesis Data Streams added support for direct delivery to S3 general purpose buckets, so I tried it out
This page has been translated by machine translation. View original
Introduction
With the August 2026 update, it became possible to deliver data directly from Kinesis Data Streams to S3 general-purpose buckets. Previously, routing through Amazon Data Firehose was the common approach for archiving to S3, but now delivery is possible simply by preparing a service execution role and creating one channel on the stream.
Comparing the traditional configuration with direct delivery side by side:
| Item | KDS → Firehose → S3 | KDS → S3 Direct Delivery |
|---|---|---|
| Additional resources to create | Firehose stream | Channel |
| Data transformation / format conversion | Possible | Not possible |
| Minimum buffer | 60 seconds (zero-buffering also available) | 300 seconds |
| Additional charges for delivery (unit price) | data-out $0.040/GB + ingestion $0.029/GB | S3 delivery $0.0275/GB |
Direct delivery charges are added on top of existing KDS charges (per-stream / data-in / retention). Firehose ingestion is billed in 5KB increments, and $0.029/GB is the unit price applied to the first 500 TB per month. Unit prices are the values from the US East calculation examples listed on the Kinesis Data Streams pricing page (S3 delivery tab) and the Amazon Data Firehose pricing page. Minimum buffer values are listed in delivery quotas and limits for direct delivery and buffering hints for Firehose.
In this article, I'll build a configuration that delivers data in GZIP / JSON format from an On-Demand Standard stream to an SSE-S3 bucket. I'll walk through the delivery creation steps, S3 key structure, contents of delivered objects, elapsed time until the first record arrives, and where invalid records end up.
Validation Details
In ap-northeast-1, I created a channel on an On-Demand Standard stream.
Validation Environment
All operations were performed using the AWS CLI. The version used is as follows:
aws-cli/2.36.36 Python/3.14.6 Linux/7.1.6-400.asahi.fc44.aarch64+16k docker/aarch64.amzn.2023
With 2.36.36, kinesis create-channel could be executed. If the command is not found, update the CLI.
The execution environment is Docker image public.ecr.aws/aws-cli/aws-cli:2.36.36. The digest is sha256:7047de9ebb37c39d07f652ff2ff3f6c42308a4e189788aeeae70563b0c82b57d.
The source stream settings are as follows:
{
"StreamDescriptionSummary": {
"StreamName": "kds-s3-gp-ods-20260901-2210",
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamStatus": "ACTIVE",
"StreamModeDetails": {
"StreamMode": "ON_DEMAND"
},
"RetentionPeriodHours": 24,
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"EncryptionType": "NONE",
"OpenShardCount": 4,
"ConsumerCount": 0,
"MaxRecordSizeInKiB": 1024,
"ChannelCount": 0
}
}
On-Demand streams are the target for direct delivery.
Creating the Delivery
First, create a service execution role. The trusted principal is kinesis.amazonaws.com, with conditions for the source account and a wildcard for the channel ARN.
{
"Role": {
"Path": "/",
"RoleName": "kds-s3-gp-role-20260901-2210",
"Arn": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"CreateDate": "2026-09-01T13:58:07+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "kinesis.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/*"
}
}
}
]
}
}
}
The permission policy includes write access to the destination bucket, bucket listing, and DLQ-related actions. Delivery succeeded with this configuration. Multipart upload execution itself is permitted with s3:PutObject (CreateMultipartUpload).
Service Execution Role Permission Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeliveryBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DeliveryBucketWrite",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:CreateMultipartUpload",
"s3:UploadPart",
"s3:CompleteMultipartUpload",
"s3:ListMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
]
},
{
"Sid": "DLQBucketAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "123456789012"
}
}
}
]
}
The delivery settings were passed together in an input JSON file.
{
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"RecordConfiguration": { "RecordFormatType": "JSON" }
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
}
}
aws kinesis create-channel \
--cli-input-json file://scripts/params/create-channel.json
The DLQ configuration is directly under S3DestinationConfiguration, not inside StorageConfiguration. Also, when specifying GZIP compression, !{extension} is required in the key template. Since no encryption is specified, the destination defaults to S3's default SSE-S3.
create-channel Response
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelARN": "arn:aws:kinesis:ap-northeast-1:123456789012:channel/xxxxxxxxxxxxxxxxx",
"ChannelId": "xxxxxxxxxxxxxxxxx",
"ChannelStatus": "CREATING",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00",
"ServiceExecutionRoleARN": "arn:aws:iam::123456789012:role/kds-s3-gp-role-20260901-2210",
"StreamConfigurationList": [
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210",
"StreamCreationTimestamp": "2026-09-01T13:58:21+00:00",
"RecordConfiguration": {
"RecordFormatType": "JSON"
}
}
],
"S3DestinationConfiguration": {
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"ErrorOutputPrefix": "dlq/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::kds-s3-gp-20260901-2210-123456789012",
"ExpectedBucketOwner": "123456789012",
"OutputKeyTemplate": "data/year=!{yyyy}/month=!{MM}/day=!{dd}/hour=!{HH}/records-!{extension}",
"StorageClass": "STANDARD",
"CompressionType": "GZIP"
}
},
"LoggingConfiguration": {
"CloudWatchLogs": {
"Enabled": false
}
}
}
}
The status immediately after creation was CREATING. I extracted the status portion from the response retrieved after waiting a moment.
{
"ChannelDescription": {
"ChannelName": "kds-s3-gp-channel-20260901-2210",
"ChannelStatus": "ACTIVE",
"ChannelCreationTimestamp": "2026-09-01T13:58:38+00:00"
}
}
From creation to observing ACTIVE was 8 seconds, from 13:58:38Z to 13:58:46Z. The create-channel response reflected the specified record format, GZIP compression, and storage destination and DLQ settings as-is.
Key Structure
{
"Contents": [
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000000000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:15:52+00:00",
"ETag": "\"dd02b756664cd84f8b3d675727905c1d-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 66,
"StorageClass": "STANDARD"
},
{
"Key": "data/year=2026/month=09/day=01/hour=13/records--000000000000100000-000000-9d6757de83b.gz",
"LastModified": "2026-09-01T14:16:17+00:00",
"ETag": "\"2b73988bd978d1be87230ceb356aff6e-1\"",
"ChecksumAlgorithm": [
"SHA256"
],
"ChecksumType": "COMPOSITE",
"Size": 89,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "data/"
}
The keys include year, month, day, and hour as Hive-style partitions, as specified in the key template. The template specifies up to records-, and the service appends the sequential number, unique suffix, and .gz. The valid JSON records (3 records put in the next section) were split into 2 objects, distinguished by keys containing sequential numbers.
Contents of Delivered Objects
I put 3 valid JSON records and 1 non-JSON string. Only the beta record has spaces after colons and commas.
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key alpha \
--cli-binary-format raw-in-base64-out \
--data '{"message":"alpha","n":1,"nested":{"z":true}}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key beta \
--cli-binary-format raw-in-base64-out \
--data '{"message": "beta", "n": 2, "array": [1, 2, 3]}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key gamma \
--cli-binary-format raw-in-base64-out \
--data '{"message":"gamma","n":3,"null":null}'
aws kinesis put-record --stream-name "$STREAM_NAME" --partition-key invalid \
--cli-binary-format raw-in-base64-out \
--data 'not-json'
Decompressing and concatenating the two delivered objects yielded the following. The first object (sequential number 000000000000000000) contains beta, and the second contains alpha and gamma.
{"message": "beta", "n": 2, "array": [1, 2, 3]}
{"message":"alpha","n":1,"nested":{"z":true}}
{"message":"gamma","n":3,"null":null}
To verify the delimiter, I displayed the object containing alpha and gamma using od -c.
0000000 { " m e s s a g e " : " a l p h
0000020 a " , " n " : 1 , " n e s t e d
0000040 " : { " z " : t r u e } } \r \n {
0000060 " m e s s a g e " : " g a m m a
0000100 " , " n " : 3 , " n u l l " : n
0000120 u l l } \r \n
0000126
In this object, the record delimiter was CRLF, and each record also ended with CRLF. Even with JSON specified as the record format, no re-serialization such as whitespace removal or key reordering occurred for the input in this test — the formatting differences at the time of insertion were preserved as-is.
Elapsed Time Until Delivery
I lined up the time ACTIVE was observed for the channel, the time the first record was put, and the LastModified of the first object.
channel_active_observed_utc=2026-09-01T13:58:46+00:00
first_record_put_utc=2026-09-01T13:58:46+00:00
s3_first_object_last_modified_utc=2026-09-01T14:15:52+00:00
Put Times for Each of the 4 Records
record_1_put_utc=2026-09-01T13:58:46+00:00
record_2_put_utc=2026-09-01T13:58:47+00:00
record_3_put_utc=2026-09-01T13:58:48+00:00
record_4_put_utc=2026-09-01T13:58:49+00:00
In this single-run result with one channel, the time from the first PutRecord to the first object was approximately 1,026 seconds, or about 17 minutes. Since the first PutRecord was at 13:58:46Z, the same time ACTIVE was observed, this value does not include channel startup wait time. DataFreshnessInSeconds was set to the minimum value of 300, but delivery did not occur at that exact timing. The second object was created 25 seconds after the first.
DLQ
{
"Contents": [
{
"Key": "dlq/DESERIALIZATION_ERROR/2026/09/01/13/kds-s3-gp-channel-20260901-2210-000000000000000000-000000-9d6757de83b.json",
"LastModified": "2026-09-01T14:15:52+00:00",
"Size": 372,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "dlq/"
}
Under the specified ErrorOutputPrefix, error type and timestamp partitions were automatically appended. The object contents are a single-line JSON.
{"approximateArrivalTimestamp":1788271129616,"streamArn":"arn:aws:kinesis:ap-northeast-1:123456789012:stream/kds-s3-gp-ods-20260901-2210","shardId":"shardId-xxxxxxxxxxxxxxxxx","sequenceNumber":"xxxxxxxxxxxxxxxxx","errorCode":"Record.InvalidJson","errorMessage":"The record contains invalid JSON. Please verify the record is valid JSON."}
The object confirmed in this test output the stream ARN, shard ID, sequence number, arrival timestamp, and error details as JSON, but did not include the original record data. The original data cannot be recovered from the DLQ contents alone.
Conditions and Constraints for Using S3 Direct Delivery
To deliver data directly from Kinesis Data Streams to S3, the following conditions and constraints apply. Check the Developer Guide and delivery quotas and limits and confirm in advance whether your stream qualifies.
- Cannot be used if the source stream is encrypted with an AWS managed key (
aws/kinesis). Migration to a CMK is a prerequisite - Only On-Demand capacity mode is supported; Provisioned streams are not eligible
- After creation, only DataFreshnessInSeconds and LoggingConfiguration can be updated. Compression, key template, record format, and destination bucket cannot be changed — to change them, delete and recreate the channel. There is no backfill of existing data
- A maximum of 2 deliveries can be created per stream: 1 for Iceberg and 1 for general-purpose S3. Limit increase requests are not available
- Variables available in the key template are limited to timestamp, channel name, and stream name. Partitioning by record content is not possible
- Cross-region delivery is not supported. Only the destination bucket can be in a different account
Coexistence with Firehose
Even when direct delivery is available, there are features that exist only on the Firehose side. These include data transformation via Lambda, format conversion to Parquet / ORC, dynamic partitioning based on record content, freshness below 60 seconds, and delivery from a single source to multiple destinations. Conversely, features exclusive to direct delivery include ZSTD compression and specifying the destination storage class (STANDARD / INTELLIGENT_TIERING / GLACIER_IR).
Summary
I was able to deliver data directly from Kinesis Data Streams to a general-purpose S3 bucket without going through Firehose. In the validation, I confirmed output using Hive-style keys, delivery of JSON records, and DLQ output of invalid JSON records.
If you don't need Firehose features or Lambda-based transformation, give the newly added direct delivery a try.
Also, if your goal is archiving with reprocessing or testing in mind, there are cases where designing S3 as the starting point for reprocessing is more manageable than extending the retention period of Kinesis Data Streams. If your purpose is long-term data retention, consider evaluating direct delivery as well.
