Trying Out the New Amazon Kinesis Data Streams Feature "Service-Managed Partition Keys" with the CLI
This page has been translated by machine translation. View original
Introduction
On September 23, 2026, "service-managed partition keys" were added to Amazon Kinesis Data Streams.
When you switch the record distribution strategy of an On-Demand mode stream to AUTO, specifying a partition key becomes optional, and you can leave the distribution of records to shards to Kinesis's algorithm.
In this article, I used the AWS CLI to check the steps for switching the record distribution strategy, whether put-record succeeds without a partition key, and what happens to the PartitionKey of the records written.
Test Environment
I used the following CLI version. The region is ap-northeast-1.
aws-cli/2.37.2 Python/3.14.6 Linux/7.1.13-402.asahi.fc44.aarch64+16k
Creating a Stream and Checking the Initial State
Create a stream in On-Demand mode.
aws kinesis create-stream \
--stream-name <stream-name> \
--stream-mode-details StreamMode=ON_DEMAND \
--region ap-northeast-1
Check the created stream with describe-stream-summary. When StreamStatus is ACTIVE, creation is complete. You can also get the ARN used in the following commands here (the output is excerpted).
{
"StreamARN": "arn:aws:kinesis:ap-northeast-1:<account-id>:stream/<stream-name>",
"RecordDistributionStrategy": "USER_PARTITION_KEY"
}
Even when created in On-Demand mode, the initial value is USER_PARTITION_KEY. To use service-managed partition keys, you need to switch explicitly.
Omitting the Partition Key Before Switching
When I ran put-record without --partition-key while still on USER_PARTITION_KEY, it resulted in a validation error.
An error occurred (ValidationException) when calling the PutRecord operation:
1 validation error detected: Value null at 'partitionKey' failed to satisfy constraint: Member must not be null
Switching to AUTO
The strategy is changed with update-stream-record-distribution-strategy. The target must be specified with --stream-arn.
aws kinesis update-stream-record-distribution-strategy \
--stream-arn <stream-arn> \
--record-distribution-strategy AUTO \
--region ap-northeast-1
Immediately after running the command, the stream's StreamStatus became UPDATING. Before proceeding to the next put-record, confirm with describe-stream-summary that the status has returned to ACTIVE and the strategy is AUTO.
{
"StreamStatus": "ACTIVE",
"RecordDistributionStrategy": "AUTO"
}
According to aws kinesis put-record help, on an AUTO stream the partition key becomes an optional item, and even if you pass a value, it is not used to determine which shard the record is distributed to. ExplicitHashKey is likewise ignored, and records are distributed to shards by the service-managed algorithm. In other words, there is no longer a guarantee that records with the same partition key go to the same shard.
Also, switching the strategy is only for On-Demand mode streams. According to the help for update-stream-record-distribution-strategy, trying to set AUTO on a Provisioned mode stream results in an InvalidArgumentException.
Comparing put-record With and Without a Partition Key
First, send a record with a partition key specified to the stream switched to AUTO.
aws kinesis put-record \
--stream-arn <stream-arn> \
--data "$(echo -n 'record-with-pk' | base64)" \
--partition-key "user-specified-key" \
--region ap-northeast-1
{
"ShardId": "shardId-000000000003",
"SequenceNumber": "49678656XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3154"
}
Next, send one with --partition-key removed.
aws kinesis put-record \
--stream-arn <stream-arn> \
--data "$(echo -n 'record-no-pk' | base64)" \
--region ap-northeast-1
{
"ShardId": "shardId-000000000002",
"SequenceNumber": "49678656XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2626"
}
put-record without a partition key also did not return an error, and the response fields are the same as when one is specified. From the response alone, you cannot tell which way the record was sent.
Checking the Record's PartitionKey With GetRecords
Using the ShardId included in the put-record responses as is, read the records for each shard.
# Get the ShardIterator of the target shard
aws kinesis get-shard-iterator \
--stream-arn <stream-arn> \
--shard-id <shard-id> \
--shard-iterator-type TRIM_HORIZON \
--region ap-northeast-1
# Get records (substitute the ShardIterator value from the previous command's output)
aws kinesis get-records \
--shard-iterator <shard-iterator> \
--limit 10 \
--region ap-northeast-1
For the record sent with a partition key specified, although the key was not used for distribution, the specified value was stored in PartitionKey as is.
{
"Records": [
{
"SequenceNumber": "49678656XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3154",
"ApproximateArrivalTimestamp": "2026-09-25T09:19:21.123000+09:00",
"Data": "cmVjb3JkLXdpdGgtcGs=",
"PartitionKey": "user-specified-key"
}
]
}
For the record sent without it, Data was returned as the value sent, but the PartitionKey field itself was not returned.
{
"Records": [
{
"SequenceNumber": "49678656XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2626",
"ApproximateArrivalTimestamp": "2026-09-25T09:19:21.584000+09:00",
"Data": "cmVjb3JkLW5vLXBr"
}
]
}
If you switch to an operation that sends records without partition keys, you need to make sure that consumers do not reference PartitionKey.
Summary
Until now, when key distribution was skewed and writes concentrated on a particular shard, Puts could be throttled. To avoid this, countermeasures on the sending side, such as processing keys by combining hash values, were necessary, but with AUTO this kind of work becomes unnecessary.
On the other hand, because AUTO does not maintain ordering per key, for a design such as using a store ID as the partition key and wanting to process each store's deposits and withdrawals in order, you would choose USER_PARTITION_KEY as before and handle shard distribution yourself. Since the strategy can be chosen per stream, you can handle both by separating streams that need ordering from those that don't.
For use cases such as log aggregation and IoT telemetry, which do not require record ordering and need countermeasures against throttling caused by shard skew, please try the AUTO record distribution strategy together with the latest AWS SDK and KPL.
