Using Structured Outputs with Amazon Bedrock Batch Inference Too: Stabilizing AI Output

On our technical blog "DevelopersIO," we are attempting to use AI to generate summaries for more than 60,000 articles. This time, we needed to reprocess a large number of summaries to reflect improvements to our prompt. Before processing everything at once, here are the results of a pilot run on the most recent ~7,000 articles.

So far, we have covered Amazon Bedrock batch inference and Structured Outputs separately.

https://dev.classmethod.jp/articles/amazon-bedrock-batch-inference-efficiency/

https://dev.classmethod.jp/articles/amazon-bedrock-structured-outputs-json/

This article is a practical record of combining the two.
Processing articles one by one on demand takes about 3 seconds per article x 60,000 articles, or roughly 50 hours, but Batch Inference is expected to greatly shorten the processing time and reduce costs by 50%. For the basic setup of Batch Inference, see the first article.

Problem: JSON output from batch inference was unstable

In the first batch inference, we instructed the system prompt to "output in JSON format" and parsed the results with regular expressions. With this approach, adding JSON output instructions for batch use made the prompt diverge from the production prompt, and problems occurred such as the JSON breaking because of double quotes or line breaks in Japanese text, the latter fields of multiple-field outputs going missing, and the closing bracket of the JSON being missing in long outputs.

Along with this prompt improvement, the on-demand side has already migrated to Structured Outputs. If the batch side can also process with the same schema, we can achieve both centralized prompt management and type-safe JSON output.

Solution: Structured Outputs worked with batch inference too

To get straight to the conclusion, Structured Outputs can be used with Batch Inference as is. At first I assumed it would not be usable in batch and proceeded with the conventional approach, but the official documentation clearly states, "Batch inference - Use structured outputs within batch inference without any additional setup."

The prompts and schemas below are minimal samples for reproducing the behavior. The details of the production prompt are not disclosed.

Sample schema

A Japanese and English summary and detail, four fields in total, are generated in a single request.

SCHEMA = {
    "type": "object",
    "properties": {
        "summary_ja": {"type": "string"},
        "summary_en": {"type": "string"},
        "detail_ja": {"type": "string"},
        "detail_en": {"type": "string"}
    },
    "required": ["summary_ja", "summary_en", "detail_ja", "detail_en"],
    "additionalProperties": False
}

Because output of all fields specified in required is guaranteed, missing fields and JSON syntax errors cannot occur in principle.

Only three changes to the JSONL

For the existing batch inference JSONL, you just remove the JSON output instructions from the system prompt and add output_config. Result parsing also becomes just json.loads().

 {
   "recordId": "article_id",
   "modelInput": {
     "anthropic_version": "bedrock-2023-05-31",
     "max_tokens": 4096,
-    "system": "... Generate a JSON with summary_ja, summary_en ... Output ONLY valid JSON.",
+    "system": "... Generate summaries in both Japanese and English.",
     "messages": [{"role": "user", "content": "..."}],
+    "output_config": {
+      "format": {
+        "type": "json_schema",
+        "schema": { ... }
+      }
+    }
   }
 }

On-demand and batch use the same prompt and the same schema, which guarantees consistent quality.

Differences in schema format between APIs

When using Structured Outputs with multiple APIs, how the schema is passed differs for each API.

It is practical to manage the schema centrally as a dict and call json.dumps() only on the Converse API side.

# Converse API -- stringify the schema, name required
outputConfig={"textFormat": {"type": "json_schema", "structure": {
    "jsonSchema": {"name": "my_schema", "schema": json.dumps(SCHEMA)}}}}

# Batch Inference (inside JSONL) -- keep as dict, no name
"output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}

When batch is not a good fit: comparison with prompt caching

When the system prompt is large and the same prompt is used repeatedly, on-demand + prompt caching is more cost-efficient.

In our system, the system prompt for tag classification includes the tag master CSV (about 3,000 entries, about 15,000 tokens). For this process, prompt caching reduces input cost by 90% from the second request onward.

system=[
    {"text": tag_prompt},
    {"cachePoint": {"type": "default"}}
]

Here is a cost comparison for 1,000 tag classifications (system prompt of about 15,000 tokens, article input of about 500 tokens, output of about 100 tokens).

※Prices are calculated for Claude 3.5 Haiku (input $0.80 / 1M, output $4.00 / 1M)

When the system prompt is large and used repeatedly, the 90% discount from prompt caching beats the 50% discount from batch. However, prompt caching has a TTL constraint (5 minutes), and if the interval between requests exceeds 5 minutes, the cache expires and you return to regular pricing. It presupposes a workload that can be processed continuously. Batch inference has no such time constraint, so it offers the operational convenience of being able to submit a job at any time and leave it.

Decision flowchart

The minimum requirement for Batch Inference is 1,000 records. Below that, waiting time alone takes 15 minutes or more, so it is not suited for processing small numbers of records. With 1,000 or more records, parallel processing keeps the required time almost constant even as the count increases (measured: 17 minutes for 1,000 records, 21 minutes for 7,950 records).

How we use each method in our system is as follows.

Implementation points

Adding the us. prefix to the model ID to enable a Cross-Region Inference Profile lets you avoid capacity shortages in a particular region through load balancing across regions. Also, checking before submission that there are no duplicate recordIds, that the file size is 200MB or less, and that the record count is within the range of 1,000-50,000 helps prevent failures at the Validating stage.

Execution results

Run 1: Pilot (1,000 records)

There were 0 JSON syntax errors in the successful records.
The 26 errors were not reproducible, and judging from the symptoms, they may have been timeouts caused by temporary load on the underlying infrastructure or cold starts.

Run 2: Scale-up (6,284 records)

Even with six times the number of records, the processing time hardly changed. The error rate also improved from 2.7% to 0.05%, so some kind of warm-up on the underlying infrastructure may have taken effect.

Retrying the failures

The 30 failures in total from the first and second runs were retried individually with the on-demand Converse API, and all were resolved. Since the minimum requirement for batch is 1,000 records, it was more reasonable to retry a small number of failures immediately on demand.

Summary of effects

We saw improvements in processing time, cost, and quality alike. In particular, being able to centralize prompt management was a major gain for improving maintainability going forward.

Conclusion

By combining Batch Inference and Structured Outputs, we achieved type-safe JSON output and a 50% cost reduction at the same time. Because the prompt and schema can be fully shared between on-demand and batch, we avoided the operational burden of "managing a separate prompt for batch."

In the future, when bulk update maintenance of existing data becomes necessary, such as with the release of a new model or prompt tuning, we plan to make use of Batch Inference and Structured Outputs.