I Tried Out Intelligent Ingestion with an OpenSearch Serverless Time Series Collection

I Tried Out Intelligent Ingestion with an OpenSearch Serverless Time Series Collection

I ingested NDJSON and JSON array logs from S3 with an ingestion job. With NDJSON, documents with an invalid @timestamp were kept in a dedicated index, and documents missing it were not rejected but got the S3 modified time.
2026.11.19

This page has been translated by machine translation. View original

Introduction

On November 11, 2026, time series collections gained support for intelligent ingestion. It applies to Amazon OpenSearch Serverless NextGen.

https://aws.amazon.com/jp/about-aws/whats-new/2026/11/amazon-opensearch-serverless-nextgen-time-series-intelligent-ingestion/

Intelligent ingestion is a feature that loads logs in S3 into a time series collection without building a pipeline. In this article, I use an ingestion job to load logs placed in S3 into a collection and check the results.

Prerequisites

collection

The region is ap-northeast-1. I first created a NextGen collection group and then created a collection within it. Before creating the collection, I also created an encryption policy using an AWS owned key and a network policy that allows public access for testing. For the group, specify the upper and lower OCU limits with --capacity-limits.

aws opensearchserverless create-collection-group --region ap-northeast-1 --name devio-ingest-grp --standby-replicas ENABLED --generation NEXTGEN --capacity-limits '{"maxIndexingCapacityInOCU":4,"maxSearchCapacityInOCU":4,"minIndexingCapacityInOCU":0,"minSearchCapacityInOCU":0}'
aws opensearchserverless create-collection --region ap-northeast-1 --name devio-ts-ingest --type TIMESERIES --collection-group-name devio-ingest-grp --standby-replicas ENABLED

Data access policy

I ran the ingestion job with the role of the caller that creates the job, without specifying a dedicated ingestion role. I save a data access policy with this role as the Principal as data-access-policy.json.

[
  {
    "Rules": [
      {
        "ResourceType": "index",
        "Resource": ["index/devio-ts-ingest/app-logs*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:UpdateIndex",
          "aoss:WriteDocument",
          "aoss:ReadDocument"
        ]
      }
    ],
    "Principal": ["arn:aws:iam::111122223333:role/<your-role>"]
  }
]
aws opensearchserverless create-access-policy --region ap-northeast-1 --name devio-ts-ingest-data --type data --policy file://data-access-policy.json

ReadDocument is a permission I added so that I could search after ingestion and verify the results. The permissions the documentation says a job requires are the following four: CreateIndex, DescribeIndex, UpdateIndex, and WriteDocument.

The calling role this time has AdministratorAccess attached, and I did not add an IAM policy. In a production environment, I recommend minimizing permissions by referring to the policy described in the documentation.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-intelligent-ingestion-s3.html

Input data

I prepared an S3 bucket in the same account as the collection and placed three synthetic log files under app-logs/.

S3 key Format Count
app-logs/logs-ndjson-1.ndjson NDJSON 2,000
app-logs/logs-ndjson-2.ndjson NDJSON 2,000
app-logs/logs-array-1.json JSON array 1,000

As bad data, I included 10 lines in the NDJSON with a missing @timestamp, 9 lines with the value "yesterday", and 1 line in the JSON array with the value "yesterday". Of the 5,000 total, 20 are bad.

Only uncompressed objects can be ingested. If you store logs compressed, decompress them before placing them.

I did not try an ingestion stream. To continuously ingest new S3 objects with a stream, you need a dedicated IAM role and an SQS queue that receives S3 event notifications. See the S3 ingestion documentation for the steps.

Ingesting with an ingestion job

Command

Create a job by passing the collection ID, the S3 bucket ARN, the bucket owner's account ID, the key prefix, and the destination index name. I narrowed the targets with the prefix and made NDJSON and JSON array separate jobs.

aws opensearchserverless create-ingestion-job --region ap-northeast-1 --collection-id <collection-id> --ingestion-job-name job-ndjson --index-name app-logs-ndjson --source '{"s3":{"bucket":"arn:aws:s3:::amzn-s3-demo-bucket","bucketOwnerAccountId":"111122223333","keyPrefix":"app-logs/logs-ndjson-"}}'

The failureIndexName in the creation response contains the name of the index that stores rejected documents. For the NDJSON job it was app-logs-ndjson-ingestion-failures.

Check progress with get-ingestion-job.

aws opensearchserverless get-ingestion-job --region ap-northeast-1 --collection-id <collection-id> --ingestion-job-id <ingestion-job-id>

Here are the status, count, and time fields excerpted from the response when the NDJSON job completed.

{
  "status": "COMPLETED",
  "state": "INGESTION_COMPLETE",
  "statistics": {
    "s3": {
      "objectsDiscovered": 2,
      "objectsIngested": 2,
      "objectsFailed": 0
    }
  },
  "createdAt": "2026-11-17T18:49:40.505000+09:00",
  "updatedAt": "2026-11-17T18:50:27.585000+09:00"
}

The counts in statistics are per S3 object.

Time required

The collection was already ACTIVE at the first check, 12 seconds after running the create command.

I checked the job status at roughly 10-second intervals.

  • The NDJSON job (2 objects, about 0.6 MB, 4,000 records) was already ACTIVE at the first check 5 seconds after creation, and COMPLETED at the check 47 seconds after.
  • The JSON array job (1 object, about 0.16 MB, 1,000 records) was CREATING at the check 1 second after, ACTIVE at 11 seconds, and COMPLETED at the check 22 seconds after.

Each value is a single observation. For both jobs, updatedAt was in the same second as the time I confirmed COMPLETED. Since I cannot tell whether updatedAt is the completion time or the check time, completion may have occurred before this time. Check the counts after the job reaches COMPLETED, as described in the documentation's procedure.

Ingestion results

Counts

After the jobs reached COMPLETED, I checked the count for each index.

Input Main index failures index
NDJSON 4,000 records app-logs-ndjson 3,991 app-logs-ndjson-ingestion-failures 9
JSON array 1,000 records app-logs-array 999 app-logs-array-ingestion-failures 1

The 4,990 in the main indexes plus the 10 in the failures indexes add up to 5,000, matching the number of records submitted. Rejected documents are not lost and remain in the failures index. The 3,991 documents in the NDJSON main index include the 10 with a missing @timestamp.

A total of 10 documents were rejected across the two jobs, but objectsFailed was 0 for both. The number of rejected documents can be checked from the failures index counts in the table above. I look at the contents of the rejected documents in the next section.

Invalid @timestamp

Here is one of the 9 NDJSON documents that went into the failures index.

{
  "@timestamp": 1794909011648,
  "failure_reason": "mapping_conflict",
  "failure_message": "MapperParsingException[failed to parse field [@timestamp] of type [date] in document with id '<document-id>'. Preview of field's value: 'yesterday']; nested: IllegalArgumentException[failed to parse date field [yesterday] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];",
  "original_document": "{\"@timestamp\":\"yesterday\",\"seq\":3945,\"level\":\"ERROR\",\"service\":\"checkout\",\"status\":500,\"latency_ms\":14,\"message\":\"GET /search?q=tea 500\",\"ingestionSource\":{\"s3_uri\":\"s3://amzn-s3-demo-bucket/app-logs/logs-ndjson-2.ndjson\",\"s3_bucket\":\"amzn-s3-demo-bucket\",\"type\":\"s3\"}}",
  "ingestionSource": {
    "s3_uri": "s3://amzn-s3-demo-bucket/app-logs/logs-ndjson-2.ndjson",
    "s3_bucket": "amzn-s3-demo-bucket",
    "type": "s3"
  }
}

The reason for rejection is in failure_reason and failure_message, the original log line is in original_document, and the source S3 object is in ingestionSource.

Documents without @timestamp

In the range where I ingested NDJSON into an index using the default @timestamp, none of the 10 documents with a missing @timestamp were rejected, and all went into the main index. The filled-in @timestamp matched the LastModified of the source S3 object.

seq S3 object @timestamp after ingestion S3 LastModified
199 app-logs/logs-ndjson-1.ndjson 2026-11-17T09:49:24Z 2026-11-17T09:49:24+00:00
2463 app-logs/logs-ndjson-2.ndjson 2026-11-17T09:49:25Z 2026-11-17T09:49:25+00:00

All 10 missing ones had the LastModified value of their respective files. The job was created at 09:49:40Z, and the filled-in values are the update times of the S3 objects, which are earlier than that.

When I ingested directly into the same collection with _doc, documents without @timestamp were rejected. Here is an excerpt of the response.

"reason":"document is missing required timestamp field [@timestamp]"
"status":400

The filling-in was confirmed only for ingestion via an ingestion job.

The announcement says that documents with a missing or invalid timestamp are rejected. This time, documents with invalid values were rejected as announced, but documents with a missing timestamp were not rejected, and the S3 update time was filled in.

The timestamp field documentation says that documents are rejected if @timestamp is missing. The result of the direct ingestion matches this description.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-timestamp-field.html

The S3 ingestion documentation does not describe how a missing @timestamp is handled via a job. Logs with a missing timestamp appear in time range searches by the S3 update time rather than the original event time. Because you cannot rely on a job to reject missing ones, you need to add @timestamp to all logs before placing them in S3.

Summary

With intelligent ingestion in Amazon OpenSearch Serverless, you can now ingest NDJSON and JSON array logs in S3 into a time series collection without a pipeline.

If you are building pipelines for time series data ingestion or ETL, I recommend evaluating it once.

Share this article

AWSのお困り事はクラスメソッドへ