Blind Majority Voting with Three LLMs! Testing the Claude Sonnet 4.6 Translation Pipeline I'm Building

Since the end of 2025, DevelopersIO has been offering automatic English translation of articles using Amazon Bedrock and Claude.

https://dev.classmethod.jp/articles/developersio-bedrock-translation-launch/

Our previous pipeline already achieved Markdown-structure-preserving translation to a certain degree, but in this article I introduce our effort to adopt Claude's latest model, Sonnet 4.6, to further improve translation quality and use the API more efficiently.

Architecture

This is the architecture of the translation pipeline currently under construction.

Translation + Summary + Recommendation in One Call with Structured Outputs

We wanted to obtain information beyond the translation, such as summaries useful for search, from a single LLM run. Bedrock's Structured Outputs turned out to be the ideal means for this.

https://dev.classmethod.jp/articles/amazon-bedrock-structured-outputs-json/

Schema Design

Using Bedrock's output_config.format.json_schema, we have it return 9 fields in a single API call.

{
  "type": "object",
  "properties": {
    "title":          { "type": "string" },
    "excerpt":        { "type": "string" },
    "content":        { "type": "string" },
    "summary_en":     { "type": "string" },
    "rag_en_s":       { "type": "string" },
    "rag_en_m":       { "type": "string" },
    "rag_en_l":       { "type": "string" },
    "recommendation": { "type": "string" },
    "confidence":     { "type": "integer" }
  },
  "required": ["title", "excerpt", "content", "summary_en",
               "rag_en_s", "rag_en_m", "rag_en_l", "recommendation",
               "confidence"],
  "additionalProperties": false
}

Specifying additionalProperties: false prevents fields outside the schema from slipping in.

Calling Bedrock

body = {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 60000,
    "system": [{"type": "text", "text": system_prompt,
                "cache_control": {"type": "ephemeral"}}],
    "messages": [{"role": "user", "content": user_prompt}],
    "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}
}
response = bedrock.invoke_model(modelId="us.anthropic.claude-sonnet-4-6", body=json.dumps(body))

Key Points of the Translation Prompt

The translation prompt is placed in the system prompt. Since the full text is long, I'll excerpt the key rule instructions.

- Preserve all Markdown formatting symbols (#, *, _, [], (), etc.) in their exact positions
- Never edit, modify, or translate URLs. Keep all URLs exactly as they appear in the original
- Preserve the line break structure of the original document
- Do not start the content translation with a heading

We explicitly instruct the model to preserve the Markdown structure. The ban on modifying URLs is to prevent the translation model from translating Japanese paths inside URLs into English.

For the summary fields, we specify the word count and level of detail for each purpose.

- summary_en: 3 lines in English, max 50 words per line, Markdown bullet format (- )
- rag_en_s: ~200 words, 1 paragraph. First 80 words MUST contain what the article covers and key outcome
- rag_en_m: ~150 words, dense factual summary. Include technologies used, problem solved, approach taken

Execution Results

We validated with a 13,479-character technical article (8 code blocks, a Mermaid diagram, and many tables).

Thanks to Structured Outputs, there were zero parse errors even for JSON containing long Markdown. There is no worry that backticks or escape characters inside the Markdown will break the JSON, which keeps downstream processing simple.

Mechanical QA: Four Quality Checks

The following four checks are run automatically on the translation results.

As preprocessing, line endings are normalized (\r\n → \n).

In addition, we combine these with the confidence score (0-100) mentioned earlier for anomaly detection. Articles with low confidence are subject to human review.

The pilot article passed all checks, with a confidence of 93.

LLM-as-a-Judge: Quality Verification by Blind Majority Vote of 3 Models

Why LLM-as-a-Judge

Mechanical QA can detect structural problems such as "a heading disappeared" or "a URL changed," but it cannot measure whether "the translation is natural" or "the technical terms are well chosen." Human evaluation, on the other hand, costs time and money even at a scale of 50 articles.

So we adopted LLM-as-a-Judge, in which multiple LLMs score translation quality and a majority vote decides the outcome.

Selecting the Evaluation Models

We used three models as evaluators. Nova Pro and gpt-oss-120b are available on Amazon Bedrock, while Gemini 3.1 Pro was run manually outside Bedrock. The characteristics of each model are described in detail in the evaluation results section.

Designing Out Bias

LLM-as-a-Judge has two strong biases.

Version number bias (anchoring): If the prompt says "Sonnet 3.7" and "Sonnet 4.6," the LLM may be swayed by the prior knowledge that "the newer version must be better."

Position bias: It has been reported that the order in which translations are presented (A/B) tends to influence the judgment.

To eliminate these, we took the following measures.

1. Model name masking: The prompt only says "Translation A" and "Translation B," with no model names or version information whatsoever
2. A/B order randomization: The A/B assignment is randomly swapped for each article, and the script automatically reverses the results after evaluation

Evaluation Prompt

The evaluation prompt is written in English. Evaluation models tend to be stronger in English, so we prioritized evaluation accuracy and stability.

The following is the full text of the prompt we actually used.

You are an expert bilingual (Japanese-English) technical translation evaluator.
Compare two machine translations of the same Japanese technical blog article.
You MUST choose a winner — ties are NOT allowed.

## Evaluation Criteria (1-10 scale)
1. Accuracy — faithful meaning, correct technical terms, preserved numbers/data
2. Naturalness — reads like native English engineer writing, not "translated Japanese"
3. Technical Precision — AWS/API/code terms handled correctly
4. Structural Fidelity — Markdown structure (headings, tables, code blocks) preserved
5. Readability — easy to follow, logical flow

## Response Format
Return ONLY valid JSON (no markdown fences):
{
  "accuracy_a": <int>, "accuracy_b": <int>,
  "naturalness_a": <int>, "naturalness_b": <int>,
  "technical_a": <int>, "technical_b": <int>,
  "structural_a": <int>, "structural_b": <int>,
  "readability_a": <int>, "readability_b": <int>,
  "winner": "A_much_better" | "A_slightly_better" | "B_slightly_better" | "B_much_better",
  "summary": "<one paragraph explaining the key differences that determined the winner>"
}

---
## Original Japanese

### Title
{ja_title}

### Excerpt
{ja_excerpt}

### Body
{ja_content}

---
## Translation A

### Title
{a_title}

### Excerpt
{a_excerpt}

### Body
{a_content}

---
## Translation B

### Title
{b_title}

### Excerpt
{b_excerpt}

### Body
{b_content}

The model is asked to give a score of 1-10 on each of five axes (Accuracy, Naturalness, Technical Precision, Structural Fidelity, Readability), and to return the overall winner and a summary of the differences in JSON.

Evaluation Results

For quality verification, we compared translations by Sonnet 4.6 against those from the existing translation pipeline in an A/B comparison.

Nova Pro + gpt-oss-120b Evaluation

The first evaluation used a schema that allowed Ties (draws), but Nova Pro judged 72% of articles as a Tie, so it did not function as an evaluation. LLMs seem to have a bias toward choosing the safest option, a Tie, when the difference is slight.

So we switched to a Forced Choice (forced binary choice) that eliminates Ties, and re-evaluated with a 4-level scale (A_much_better / A_slightly_better / B_slightly_better / B_much_better).

Here is the breakdown by model.

With Forced Choice, Nova Pro gave a clear verdict for every article and overwhelmingly favored Sonnet 4.6. gpt-oss-120b also favored Sonnet 4.6, but there were 21 articles where it chose the old translation.

Analysis of Why gpt-oss-120b Chose the Old Translation

When we analyzed the summaries of the 21 articles where gpt-oss-120b chose the old translation, clear patterns emerged.

In particular, the hallucination in empty Excerpts is likely to be fixed in 7 cases just by adding one line to the prompt, which made it a useful finding as a concrete improvement point.

Gemini 3.1 Pro Detailed Evaluation

We had the AI assistant Kiro (Opus 4.6) select 8 samples according to the following policy, and entered the resulting judging prompts into the web browser version of Gemini for evaluation.

• Include the longest article (33,330 characters) and the shortest article (415 characters)
• Include articles that failed mechanical QA
• Include not only technical articles but also non-technical articles (a new-hire blog post)
• Include contested articles on which Nova and gpt-oss disagreed

Sonnet 4.6's Strengths Pointed Out by Gemini

Free-translation ability: It converted Japanese abbreviations and colloquial expressions into expressions that work in the English-speaking world.

"LT" is a common abbreviation in the Japanese tech community, but it is not understood in the English-speaking world. Sonnet 4.6 expanded it to "lightning talk slides," which Gemini rated highly. Regarding the translation of the Japanese phrase meaning "detected as a difference" as "detected as a diff," Gemini commented that it "reads as if it were translated by someone who actually writes Terraform."

Structure preservation: Sonnet 4.6 accurately preserved custom Markdown blocks such as :::message and :::details. In the old translation, there were cases where a :::message block was deleted entirely.

Restructuring of text: It went beyond mere translation by editing, for example splitting a long Japanese sentence into "First... Second... Third..." to match English-language conventions.

Cases Needing Improvement

Specific improvement points were found in the NocoDB article (1,589 characters), where the old translation won in Gemini 3.1 Pro's detailed evaluation.

• Hallucination in an empty field: Even though the original Excerpt was empty, Sonnet 4.6 copied the beginning of the body text to fill in the Excerpt
• Unconverted full-width parentheses: It did not convert （） to half-width ()

Hallucination in empty fields was also confirmed in 7 cases in the batch evaluation described earlier, and it can be handled either by stating "return an empty string if empty" in the prompt or by preprocessing in code. Converting full-width parentheses is a problem that should be solved mechanically with str.replace in post-processing, and should not be left to the LLM.

Prompt Tuning and Re-evaluation

Based on the evaluation results above, we added the following four items to the translation prompt.

- If a field (title, excerpt) is empty in the source, return an empty string. Never fabricate content
- Translate image alt-text (e.g. ![日本語テキスト](...) → ![English text](...))
- Adapt section headings to natural English conventions rather than translating word-for-word
- Proofread your output for typos and grammatical errors before responding

After deployment, we re-translated the NocoDB article that lost to the old translation and the FSx article that was a Tie, and re-evaluated them with the same evaluation script.

Here are the specific improvements in the NocoDB article.

Incidentally, when we checked the content of the Excerpt generated through hallucination, it was a suitable summary of the article. This time we prioritized translation fidelity and tuned the prompt to "return empty if empty," but we also plan to adjust the upstream process so that a Japanese Excerpt exists before translation.

Comparison of the Evaluation Models' Characteristics

Nova Pro: Fast (2-5 seconds) and low cost. With a schema that allowed Ties, 72% escaped to Tie, but when we switched to Forced Choice it gave a clear verdict for every article. However, most were slightly_better, so the degree of difference is modest. It seems well suited to primary screening of large numbers of articles.

gpt-oss-120b: Comes with reasoning (thought process) and compares carefully section by section. Its strength is that concrete improvement points, such as hallucination in empty Excerpts and slight structural differences, can be read from its summaries. On the other hand, due to the output format peculiar to reasoning models, there were cases of JSON parse failures (15/51 on the first run, improved to 1/51 with retries), so output stability needs attention.

Gemini 3.1 Pro: Returned the most detailed evaluations. It quoted specific phrases and explained "why that translation is good." It included typo detection, pointing out hallucinations, and even context-aware comments such as "reads as if it were translated by someone who actually writes Terraform." In cases where judgment was difficult, Gemini's evaluation was the most useful reference.

Summary

For the Sonnet 4.6 translation pipeline under construction, we have found a path to surpassing the quality of the previous version. It showed particularly excellent results in free-translation ability, preservation of custom Markdown blocks, and adaptation to English-language writing conventions.

On the other hand, the evaluations by gpt-oss-120b and Gemini also revealed specific issues to be addressed through preprocessing and prompt tuning, such as hallucination caused by empty fields. When we actually revised the prompt and re-evaluated, even the articles that had lost to the old translation flipped to Sonnet 4.6 wins. The LLM-as-a-Judge introduced this time can quickly judge quality after a fix, so we can expect a more efficient improvement cycle.

Alongside the update to Sonnet 4.6, we have been able to establish a foundation for LLM-powered evaluation and improvement. Going forward, through continuous improvement of translation quality, we hope to deliver technical information to even more people.