
I Tried Out OpenSearch Ingestion Now That It Can Write to NextGen Time Series Collections
This page has been translated by machine translation. View original
Introduction
On November 17, 2026, it became possible to write from an Amazon OpenSearch Ingestion (OSIS) pipeline to an Amazon OpenSearch Serverless (NextGen) time series collection.
In this article, I verify writing from an OSIS pipeline to a time series collection and searching it, inheritance of index template settings, and how documents with a missing timestamp are handled.
Test Environment
| Item | Details |
|---|---|
| Date performed | November 19, 2026 |
| Region | ap-northeast-1 |
| AWS CLI | 2.41.1 |
| Pipeline source | HTTP |
| Collection type | TIMESERIES (NextGen) |
| Signed requests | awscurl |
Resources to Create
Collection
I created, in this order, a NextGen collection group, an encryption policy, a network policy that allows public access so I can search from my local machine, and a time series collection. I created the collection group with --generation NEXTGEN, set OCU to a minimum of 0 and a maximum of 2, and set standby replica to ENABLED.
aws opensearchserverless create-collection-group --name ngts-group --standby-replicas ENABLED --generation NEXTGEN --capacity-limits minIndexingCapacityInOCU=0\,minSearchCapacityInOCU=0\,maxIndexingCapacityInOCU=2\,maxSearchCapacityInOCU=2
aws opensearchserverless create-security-policy --type encryption --name ngts-enc --policy file://encryption-policy.json
aws opensearchserverless create-security-policy --type network --name ngts-net --policy file://network-policy.json
aws opensearchserverless create-collection --name ngts-logs --type TIMESERIES --collection-group-name ngts-group
aws opensearchserverless batch-get-collection --names ngts-logs
The collection type is TIMESERIES, and when I checked about 20 seconds after creation, it was ACTIVE. The endpoint used for the pipeline's hosts and for search requests can be found in collectionEndpoint in the batch-get-collection response.
encryption-policy.json and network-policy.json
encryption-policy.json
{
"Rules": [{"ResourceType": "collection", "Resource": ["collection/ngts-logs"]}],
"AWSOwnedKey": true
}
network-policy.json
[
{
"Description": "Public access for queries",
"Rules": [
{"ResourceType": "collection", "Resource": ["collection/ngts-logs"]},
{"ResourceType": "dashboard", "Resource": ["collection/ngts-logs"]}
],
"AllowFromPublic": true
}
]
Pipeline Role and Access Permissions
Create the pipeline role that OSIS assumes, and attach a permissions policy. Then create a data access policy that includes this role as a Principal.
aws iam create-role --role-name ngts-pipeline-role --assume-role-policy-document file://pipeline-role-trust.json
aws iam put-role-policy --role-name ngts-pipeline-role --policy-name ngts-pipeline-access --policy-document file://pipeline-role-policy.json
aws opensearchserverless create-access-policy --type data --name ngts-data --policy file://data-access-policy.json
In the data access policy, I granted permissions with the same rules to the pipeline role and to the IAM role used for searching and registering templates. I included ReadDocument for searching, and CreateCollectionItems and UpdateCollectionItems on the collection for registering templates. With this policy, writing and searching both worked. Examples of the pipeline role's permissions policy and trust policy are in the official documentation.
pipeline-role-trust.json, pipeline-role-policy.json, data-access-policy.json
pipeline-role-trust.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "osis-pipelines.amazonaws.com"},
"Action": "sts:AssumeRole"
}
]
}
pipeline-role-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Action": [
"aoss:APIAccessAll",
"aoss:BatchGetCollection",
"aoss:CreateSecurityPolicy",
"aoss:GetSecurityPolicy",
"aoss:UpdateSecurityPolicy"
],
"Resource": "*"
}
]
}
data-access-policy.json (replace <ACCOUNT_ID> and <YOUR_ROLE_NAME> with the values for your environment)
[
{
"Description": "Pipeline role and verification user",
"Rules": [
{
"ResourceType": "index",
"Resource": ["index/ngts-logs/*"],
"Permission": [
"aoss:CreateIndex",
"aoss:UpdateIndex",
"aoss:DescribeIndex",
"aoss:WriteDocument",
"aoss:ReadDocument"
]
},
{
"ResourceType": "collection",
"Resource": ["collection/ngts-logs"],
"Permission": [
"aoss:DescribeCollectionItems",
"aoss:CreateCollectionItems",
"aoss:UpdateCollectionItems"
]
}
],
"Principal": [
"arn:aws:iam::<ACCOUNT_ID>:role/ngts-pipeline-role",
"arn:aws:iam::<ACCOUNT_ID>:role/<YOUR_ROLE_NAME>"
]
}
]
Pipeline
The pipeline configuration is as follows. Specify the collection endpoint in the sink's hosts, and set serverless to true. ${/idx} in index is a format that uses the value of the document's idx field as the destination index name. I used this format so that an index with a template and an index without one could be compared in a single pipeline. For network_policy_name, I specified the name of a network policy that does not yet exist (the behavior is described later).
version: "2"
ngts-pipeline:
source:
http:
path: "/logs"
sink:
- opensearch:
hosts: ["https://<collection-id>.aoss.ap-northeast-1.on.aws"]
index: "${/idx}"
aws:
sts_role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/ngts-pipeline-role"
region: "ap-northeast-1"
serverless: true
serverless_options:
network_policy_name: "ngts-osis-net"
Save this configuration as pipeline.yaml and pass it to create-pipeline. The pipeline was created without any additional arguments for NextGen.
aws osis create-pipeline --pipeline-name ngts-pipeline --min-units 1 --max-units 1 --pipeline-configuration-body file://pipeline.yaml --pipeline-role-arn arn:aws:iam::<ACCOUNT_ID>:role/ngts-pipeline-role
aws osis get-pipeline --pipeline-name ngts-pipeline
When I checked 3 minutes 37 seconds after creation, Status was ACTIVE. The ingestion endpoint appears in IngestEndpointUrls of get-pipeline, so POST once a value is there.
The reason a document is rejected does not appear in the POST response (described later), so enable the pipeline's log output. Create a CloudWatch Logs log group and specify it with update-pipeline.
aws logs create-log-group --log-group-name /aws/vendedlogs/OpenSearchIngestion/ngts-pipeline/audit-logs
aws osis update-pipeline --pipeline-name ngts-pipeline --log-publishing-options IsLoggingEnabled=true\,CloudWatchLogDestination=\{LogGroup=/aws/vendedlogs/OpenSearchIngestion/ngts-pipeline/audit-logs\}
Verification
For requests to the collection endpoint and the pipeline ingestion endpoint, I used awscurl, which can attach SigV4 signatures.
Registering a Template and Writing
First, register the following index template before the first write to ngts-tpl-test. It is applied to indexes that match index_patterns.
{
"index_patterns": ["ngts-tpl*"],
"template": {
"settings": {
"index.warm.after": "2d",
"index.data.retention": "30"
}
}
}
Save this content as index-template.json and PUT it to the collection endpoint.
awscurl --service aoss --region ap-northeast-1 -X PUT -H Content-Type:\ application/json -d @index-template.json https://<collection-id>.aoss.ap-northeast-1.on.aws/_index_template/ngts-tpl
For data.retention, pass the number of days as a number without a unit. When I passed "30d", it returned 400 with Data retention value must be a valid integer representing days. When I passed "30", it returned {"acknowledged":true} and the registration succeeded.
Next, POST 5 documents to the pipeline. Based on the value of idx, 3 are routed to ngts-default (no template) and 2 to ngts-tpl-test (covered by template ngts-tpl).
awscurl --service osis --region ap-northeast-1 -X POST -H Content-Type:\ application/json -d @batch-1-valid.json https://<pipeline-endpoint>/logs
The response was 200 OK.
batch-1-valid.json
[
{"idx": "ngts-default", "@timestamp": "2026-11-19T01:25:00Z", "level": "INFO", "message": "valid-1 default index"},
{"idx": "ngts-default", "@timestamp": "2026-11-19T01:25:01Z", "level": "WARN", "message": "valid-2 default index"},
{"idx": "ngts-default", "@timestamp": "2026-11-19T01:25:02Z", "level": "ERROR", "message": "valid-3 default index"},
{"idx": "ngts-tpl-test", "@timestamp": "2026-11-19T01:25:03Z", "level": "INFO", "message": "valid-4 template index"},
{"idx": "ngts-tpl-test", "@timestamp": "2026-11-19T01:25:04Z", "level": "INFO", "message": "valid-5 template index"}
]
When I searched 44 seconds after the POST, ngts-default contained 3 documents and ngts-tpl-test contained 2. @timestamp was mapped as the date type.
awscurl --service aoss --region ap-northeast-1 -X GET https://<collection-id>.aoss.ap-northeast-1.on.aws/ngts-default/_search\?sort=@timestamp:asc
{"took":109,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":null,"hits":[{"_index":"ngts-default","_id":"1%3A0%3Arlxn8aEBEOGzI3mxkSZa","_score":null,"_source":{"idx":"ngts-default","@timestamp":"2026-11-19T01:25:00Z","level":"INFO","message":"valid-1 default index"},"sort":[1795051500000]},{"_index":"ngts-default","_id":"1%3A0%3Ar1xn8aEBEOGzI3mxkSZa","_score":null,"_source":{"idx":"ngts-default","@timestamp":"2026-11-19T01:25:01Z","level":"WARN","message":"valid-2 default index"},"sort":[1795051501000]},{"_index":"ngts-default","_id":"1%3A0%3AsFxn8aEBEOGzI3mxkSZa","_score":null,"_source":{"idx":"ngts-default","@timestamp":"2026-11-19T01:25:02Z","level":"ERROR","message":"valid-3 default index"},"sort":[1795051502000]}]}}
Inheritance of Index Settings
I retrieved the settings with the following command.
awscurl --service aoss --region ap-northeast-1 -X GET https://<collection-id>.aoss.ap-northeast-1.on.aws/<index名>/_settings
The index that OSIS wrote to for the first time after the template was registered (ngts-tpl-test) inherited warm.after and data.retention. Compared with ngts-default, which has no template, the result is as follows.
| index | template | warm.after | data.retention |
|---|---|---|---|
| ngts-default | None | 1d (default) | Not set |
| ngts-tpl-test | ngts-tpl | 2d | 30d (30 was passed in the template) |
Network Policy Rules
When the pipeline was created, OSIS created a network policy with the name specified in network_policy_name (ngts-osis-net). It is a separate policy from ngts-net, which is for public access. Its content is a rule that allows access to the collection from the VPC endpoint that OSIS creates. The ID of the VPC endpoint can be found in ServiceVpcEndpoints of get-pipeline.
aws opensearchserverless get-security-policy --type network --name ngts-osis-net
The contents of the policy included in the response are as follows.
[
{
"Rules": [
{
"Resource": [
"collection/ngts-logs"
],
"ResourceType": "collection"
}
],
"SourceVPCEs": [
"<VPC_ENDPOINT_ID>"
],
"Description": "Created by Data Prepper"
}
]
Handling of timestamp
In a time series collection, every document requires a timestamp field. Documents whose timestamp is missing, null, or invalid are rejected.
I compared how this condition applies when writing via OSIS with writing directly to the collection using _bulk.
Differences Between Writing via OSIS and Writing Directly
I POSTed 5 documents to OSIS: 2 with a valid @timestamp, and 1 each with it missing, null, and the invalid string "not-a-date". The response was 200 OK, and a search found 4 of the 5 documents. The one with the invalid string was not saved.
awscurl --service osis --region ap-northeast-1 -X POST -H Content-Type:\ application/json -d @batch-2-mixed.json https://<pipeline-endpoint>/logs
batch-2-mixed.json
[
{"idx": "ngts-default", "@timestamp": "2026-11-19T01:30:00Z", "message": "mixed-1 valid"},
{"idx": "ngts-default", "message": "mixed-2 timestamp missing"},
{"idx": "ngts-default", "@timestamp": null, "message": "mixed-3 timestamp null"},
{"idx": "ngts-default", "@timestamp": "not-a-date", "message": "mixed-4 timestamp invalid string"},
{"idx": "ngts-default", "@timestamp": "2026-11-19T01:30:04Z", "message": "mixed-5 valid"}
]
I sent the same pattern of 4 documents (valid, missing, null, invalid string) directly to the collection with _bulk.
awscurl --service aoss --region ap-northeast-1 -X POST -H Content-Type:\ application/x-ndjson -d @direct-bulk-mixed.ndjson https://<collection-id>.aoss.ap-northeast-1.on.aws/_bulk
direct-bulk-mixed.ndjson
{"index":{"_index":"ngts-direct"}}
{"@timestamp":"2026-11-19T01:35:00Z","message":"direct-1 valid"}
{"index":{"_index":"ngts-direct"}}
{"message":"direct-2 timestamp missing"}
{"index":{"_index":"ngts-direct"}}
{"@timestamp":null,"message":"direct-3 timestamp null"}
{"index":{"_index":"ngts-direct"}}
{"@timestamp":"not-a-date","message":"direct-4 timestamp invalid string"}
The response had errors: true, and the status of the one valid document was 201. The missing and null ones were rejected with document is missing required timestamp field [@timestamp], and the invalid string was rejected with failed to parse field [@timestamp] of type [date].
| @timestamp | Via OSIS (search results) | Direct write (status of each _bulk item) |
|---|---|---|
| Missing | Saved as the number 1795051761411 | 400 |
| null | Saved as the number 1795051761411 | 400 |
| Invalid string | Not saved | 400 |
The 1795051761411 filled in via OSIS is epoch milliseconds, which is 2026-11-19T01:29:21.411Z. I sent the POST at 01:29:20Z, so the time right after sending was filled in. The 2 documents with a valid @timestamp were saved with their original strings.
The change log for OSS Data Prepper 2.17.0 describes the sink's behavior for NextGen time series indexes. For a document that lacks the field corresponding to timestamp_field, the sink moves the value of @timestamp into it, and if that is also absent, fills in the event time. Items rejected with "document is missing required timestamp field" have a timestamp filled in and are resent in the same flush.
There is also an example with intelligent ingestion, which ingests S3 logs through an ingestion job, where rows with a missing @timestamp were not rejected and the S3 object's LastModified was filled in. The filled-in value differs from the ingestion time used via OSIS.
Confirming the Rejections
Check the reason for a rejection in the pipeline's CloudWatch Logs.
aws logs filter-log-events --log-group-name /aws/vendedlogs/OpenSearchIngestion/ngts-pipeline/audit-logs --start-time <ミリ秒>
Right after POSTing the mixed batch, the following 4 lines appeared (excluding AWS SDK warnings and the OpenSearchSink startup lines).
2026-11-19T01:29:22.059 [ngts-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - Bulk Operation Failed. Number of retries 0. Retrying...
2026-11-19T01:29:22.059 [ngts-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - index = ngts-default operation = Index, error = document is missing required timestamp field [@timestamp]
2026-11-19T01:29:22.059 [ngts-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - index = ngts-default operation = Index, error = failed to parse field [@timestamp] of type [date] in document with id '15xs8aEBEB__X7foMJKF'. Preview of field's value: 'not-a-date'
2026-11-19T01:29:22.118 [ngts-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.BulkIngester - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: failed to parse field [@timestamp] of type [date] in document with id '15xs8aEBEB__X7foMJKF'. Preview of field's value: 'not-a-date' caused by failed to parse date field [not-a-date] with format [strict_date_optional_time||epoch_millis] caused by date_time_parse_exception: Failed to parse with all enclosed parsers
For the missing and null documents, only one missing line appeared as a retry warning. I confirmed by searching that both were saved. The failure line from BulkIngester corresponds to the one invalid-string document that was not saved.
An Index with a Changed timestamp_field
Before writing, I registered a template ngts-ctf with index.timestamp_field set to event_time.
{
"index_patterns": ["ngts-ctf*"],
"template": {
"settings": {
"index.timestamp_field": "event_time"
}
}
}
I saved this content as index-template-ctf.json and PUT it to /_index_template/ngts-ctf with the same command as above. Then I POSTed the following 4 documents in the same way.
batch-4-custom-timestamp-field.json
[
{"idx": "ngts-ctf-test", "event_time": "2026-11-19T01:45:00Z", "message": "ctf-1 event_time present"},
{"idx": "ngts-ctf-test", "@timestamp": "2026-11-19T01:45:01Z", "message": "ctf-2 only @timestamp"},
{"idx": "ngts-ctf-test", "message": "ctf-3 no timestamp"},
{"idx": "ngts-ctf-test", "event_time": "not-a-date", "message": "ctf-4 event_time invalid"}
]
The response was 200 OK, and 3 documents were saved. The one with event_time was saved as is. For the one with only @timestamp, its value "2026-11-19T01:45:01Z" was moved to event_time, and @timestamp did not remain. For the one with neither, the ingestion time as the number 1795052025250 was put into event_time. The one whose event_time was an invalid string was not saved. In the _settings of ngts-ctf-test, created by this POST, timestamp_field was event_time as set in the template.
Summary
I confirmed that you can write from an OSIS pipeline to a NextGen time series collection without passing any additional NextGen-specific arguments to create-pipeline.
Documents with a missing timestamp were also saved via OSIS with the ingestion time filled in.
If you want to ingest logs stored in S3 without using an OSIS pipeline, please also consider the recently announced update to intelligent ingestion.



