Structuring LLM Output with Tool Use in Bedrock Claude — No More JSON.parse() or Regex
This page has been translated by machine translation. View original
Introduction
Have you ever written code like this when you want an LLM to respond in JSON?
raw_text = llm_response["output"]["message"]["content"][0]["text"]
# Using regex to extract because the LLM sometimes wraps the response in a markdown code block
json_match = re.search(r"```(?:json)?\s*([\s\S]*?)```", raw_text)
json_str = json_match.group(1).strip() if json_match else raw_text.strip()
parsed = json.loads(json_str)
"Have the LLM return JSON and parse it with json.loads()" — it looks simple at first glance, but it will definitely break in production.
In this article, I'll introduce an approach using Amazon Bedrock's Tool Use (toolChoice enforcement) that eliminates the need for regex and json.loads(). I'll specifically show the problems I actually encountered while developing an in-house service desk AI (RAG + Claude), and the code that could be deleted after migrating to Tool Use.
Why "Having the LLM Return JSON" Breaks
The approach of instructing via system prompt to "always return JSON" and parsing the response with json.loads() has three structural problems.
1. Wrapped in Markdown Code Blocks
Out of helpfulness, the LLM may wrap the JSON in ```json ... ```. Since json.loads() doesn't understand code block notation, it fails. You end up writing code to strip it with regex, but that itself is a fragile parser.
2. Extra Commentary Gets Mixed In
Prefatory text like "Here are the results:" can appear before the JSON. Even if you emphasize "return only JSON" in the prompt, it can recur with model updates or changes in input.
3. Truncation at max_tokens
This is the most troublesome problem. When the response reaches the token limit, the JSON gets cut off midway.
{
"results": [...],
"general_advice": "The solution is as follows.\n\n## Pattern 1\n\n1. Admin panel
json.loads() naturally fails. Since we had a fallback of general_advice = raw_text, a bug occurred where the truncated raw JSON was displayed directly to the user. Japanese text consumes approximately 1.5–2× more tokens than English, making max_tokens estimation particularly difficult.
Solution: Structured Output with Tool Use
Claude (and Claude on Bedrock) has a Tool Use feature. It's originally designed for "having the LLM call external tools," but by enforcing a specific tool with tool_choice, you can force the LLM's output to always conform to a specified JSON Schema.
The key point is that you don't actually need to execute the tool. You're simply borrowing the Tool Use mechanism to tell the LLM "output according to this schema."
How It Works
When you pass tools (schema definition) and tool_choice (enforcement) to the API, the LLM returns the response as a tool_use block. The input field of this block is guaranteed at the API level to be JSON conforming to the schema.
Implementation
Step 1: Define the Schema with a Pydantic Model
First, define the structure you want the LLM to return using a Pydantic model.
from pydantic import BaseModel, Field
from typing import Literal
class TicketResult(BaseModel):
issue_id: int
subject: str
relevance: Literal["high", "medium", "low"]
summary: str
class ChatToolInput(BaseModel):
"""Fields generated by the LLM via tool use."""
response_type: Literal["clarification", "results"]
general_advice: str = Field(
default="", description="Response for the operator (in markdown format)"
)
question: str = ""
results: list[TicketResult] = []
Field(description=...) is reflected in the JSON Schema's description and serves as a hint for the LLM to understand the meaning of the field.
Step 2: Enforce Output with tool_choice
Here's an example using the Anthropic SDK (anthropic[bedrock]).
from anthropic.lib.bedrock import AsyncAnthropicBedrock
client = AsyncAnthropicBedrock(aws_region="ap-northeast-1")
tool_schema = ChatToolInput.model_json_schema()
response = await client.messages.create(
model="ap-northeast-1.anthropic.claude-sonnet-4-5-20250514-v1:0",
messages=[{"role": "user", "content": user_message}],
system=system_prompt,
max_tokens=8192,
temperature=0.3,
tools=[{
"name": "submit_response",
"description": "Submit the structured helpdesk response",
"input_schema": tool_schema,
}],
# This is the key. Forces the LLM to use this tool
tool_choice={"type": "tool", "name": "submit_response"},
)
tool_choice={"type": "tool", "name": "submit_response"} is the key. This forces the LLM to always respond in the form of calling the submit_response tool, generating JSON that conforms to the schema defined in input_schema.
Step 3: Retrieve the Response
for block in response.content:
if block.type == "tool_use" and block.name == "submit_response":
result = ChatToolInput(**block.input)
# result.response_type, result.general_advice, result.results can be
# retrieved in a type-safe manner
No regex or json.loads() needed. block.input is already a parsed dict, and simply passing it to Pydantic gives you an object with type validation.
When Using boto3
If you use the boto3 Converse API instead of the Anthropic SDK, the parameter names differ.
client = boto3.client("bedrock-runtime", region_name="ap-northeast-1")
response = client.converse(
modelId="ap-northeast-1.anthropic.claude-sonnet-4-5-20250514-v1:0",
messages=messages,
system=[{"text": system_prompt}],
inferenceConfig={"maxTokens": 8192, "temperature": 0.3},
toolConfig={
"tools": [{
"toolSpec": {
"name": "submit_response",
"description": "Submit the structured helpdesk response",
"inputSchema": {"json": tool_schema},
}
}],
"toolChoice": {"tool": {"name": "submit_response"}},
},
)
Note: The boto3 Converse API does not support $defs / $ref in JSON Schema. Since Pydantic v2's model_json_schema() generates nested models using $defs, inline expansion is required.
def _resolve_schema_refs(schema: dict) -> dict:
"""Inline-expand $ref for Bedrock compatibility (resolved recursively)."""
defs = schema.pop("$defs", {})
def _resolve(obj):
if isinstance(obj, dict):
if "$ref" in obj:
ref_name = obj["$ref"].split("/")[-1]
return _resolve(dict(defs[ref_name]))
return {k: _resolve(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_resolve(item) for item in obj]
return obj
return _resolve(schema)
# Expansion is required for boto3
tool_schema = _resolve_schema_refs(ChatToolInput.model_json_schema())
When using the Anthropic SDK (anthropic[bedrock]), the Messages API is used internally and $defs / $ref are natively supported, so this expansion process is not needed.
Applying the Same Pattern to Multiple Use Cases
Structured output via Tool Use can be reused for multiple purposes within a single project. Just by changing the schema (Pydantic model), the same pattern can be applied.
Combining with Streaming
Tool Use can also be used in combination with streaming. When streaming with tool_choice enforced, the JSON arrives partially (token by token).
async with client.messages.stream(
model=model_id,
messages=messages,
system=system_prompt,
max_tokens=8192,
tools=[{
"name": "submit_response",
"description": "Submit the structured helpdesk response",
"input_schema": tool_schema,
}],
tool_choice={"type": "tool", "name": "submit_response"},
) as stream:
async for event in stream:
if event.type == "input_json":
chunk = event.partial_json # A fragment of JSON arrives
However, what arrives during streaming is a fragment of a JSON string. If you want to display text like "general_advice": "The solution is..." in real time, you'll need to implement a state machine that parses the JSON structure. I'll cover this in more detail another time.
Fallback When max_tokens Is Reached
Even with Tool Use, there's a possibility that the JSON gets cut off midway when max_tokens is reached. However, you can use data already retrieved during streaming as a fallback.
try:
tool_input = ChatToolInput.model_validate_json(full_tool_input)
result = ChatResponse(
response_type=tool_input.response_type,
general_advice=tool_input.general_advice,
results=tool_input.results[:3],
)
except Exception:
# When JSON is incomplete due to max_tokens being reached
# Use the general_advice already retrieved during streaming as a fallback
result = ChatResponse(
response_type="results",
general_advice=streamed_advice or "An error occurred while generating the response.",
)
With the traditional approach, raw JSON was displayed directly to the user when json.loads() failed. With the combination of Tool Use + streaming, you can use the general_advice text received partway through, significantly improving the user experience.
Before / After
Before: Regex + json.loads() Parser
# Instructing JSON format in the system prompt
RESPONSE_FORMAT = """{
"results": [...],
"general_advice": "...",
}"""
# Parsing the response
raw_text = "".join(block.get("text", "") for block in output)
try:
json_match = re.search(r"```(?:json)?\s*([\s\S]*?)```", raw_text)
json_str = json_match.group(1).strip() if json_match else raw_text.strip()
parsed = json.loads(json_str)
except (json.JSONDecodeError, AttributeError):
return [], raw_text # Return the raw text as-is (the cause of raw JSON being displayed)
results = [TicketResult(**r) for r in parsed.get("results", [])]
general_advice = parsed.get("general_advice", "")
After: Tool Use
tool_schema = ChatToolInput.model_json_schema()
response = await client.messages.create(
...,
tools=[{
"name": "submit_response",
"description": "Submit the structured helpdesk response",
"input_schema": tool_schema,
}],
tool_choice={"type": "tool", "name": "submit_response"},
)
for block in response.content:
if block.type == "tool_use":
result = ChatToolInput(**block.input) # Type-safe. No parse errors.
import re and import json are no longer needed, and the JSON format instruction section (approximately 30 lines) could also be removed from the system prompt.
Summary
| Traditional (Regex + json.loads) | Tool Use (tool_choice enforcement) | |
|---|---|---|
| Schema compliance | Depends on prompt (not guaranteed) | Guaranteed at API level |
| Markdown fences | Need to strip with regex | Does not occur |
| Extra commentary | Risk of inclusion | Does not occur |
max_tokens truncation |
json.loads() fails → raw JSON displayed |
Fallback strategy is easier to implement |
| Schema changes | Dual management of prompt and parser | Only change the Pydantic model |
Structured output via Tool Use is not a feature for "executing tools" — it can be used as a means to "enforce the LLM's output format at the API level." Since the same pattern can be applied whether you're using Bedrock or the Anthropic API directly, it's worth considering for any case where you want to structure the LLM's output.
