Nova for Intent Understanding, Titan for Stability: Comparing the Search Accuracy of Four Embedding Models Using LLM-as-a-Judge

In the previous article, I registered about 60,000 DevelopersIO blog articles in Amazon S3 Vectors and built a serverless semantic search environment.

https://dev.classmethod.jp/articles/s3-vectors-bedrock-semantic-search/

This time, I used that environment to compare search accuracy across four embedding models. I automatically scored the search results for identical queries with an LLM-as-a-Judge, and I'll explain how we arrived at the conclusion of "using Nova Embed and Titan V2 together, each where it fits best."

Models Compared

I compared two Bedrock managed models and two OSS models.

For the roughly 60,000 blog articles above, I vectorized a Japanese summary of about 165 characters and an English summary of about 60 words with each model, and registered them in S3 Vectors as separate indexes (4 models × Japanese/English = 8 indexes).

Evaluation Method

Search Queries (33 queries × 12 patterns)

Assuming real use cases, I prepared the following 12 patterns. They are broadly divided into "same-language search," where the query and the search target are in the same language, and "cross-lingual search," where the search spans different languages.

Same-Language Search:

Cross-Lingual Search:

Evaluation Metric: nDCG@10 × LLM-as-a-Judge

I gave the Top-10 of each search result to an LLM and had it score them from 0 to 100. To improve judgment accuracy, I had it read both the Japanese and English RAG summaries, not just the article titles.

From the scores, I calculated nDCG (Normalized Discounted Cumulative Gain). It is a metric that gets higher the more good results appear near the top, with 1.0 being ideal.

I used Gemini 3.0 Pro as the Judge. For reference, I also scored the same data with Claude Sonnet 4.5 (Amazon Bedrock) and compared the Judges.

Test Environment

Results

Overall Ranking

Overall, E5-Large came in first, but the picture changes greatly when looking at each test pattern.

Same-Language Search

In basic keyword searches (test1, 3, 4, 5), E5-Large won them all. On the other hand, Nova won overwhelmingly in notation variation (test6) and intent understanding (test8).

Cross-Lingual Search

For en→ja, Nova took first place in 3 of 4 patterns, and its ability to find Japanese articles from English queries stood out.

On the other hand, in the follow-up test ja→en (test12: Japanese query → English index), the results reversed. For the query "RAG hallucination countermeasures," Nova's Top-6 were all unrelated articles (conference reports, anime introductions, etc.), and it collapsed to nDCG@10 = 0.33. Titan (0.95) and E5 (0.96) maintained stable accuracy. This showed that Nova's cross-lingual ability is asymmetric between en→ja and ja→en.

Same-Language vs. Cross-Lingual

Placing same-language and cross-lingual searches of the same category side by side revealed some interesting trends.

• Nova: Accuracy barely dropped for en→ja (Basic: 0.74→0.83, Synonym: 0.90→0.88). However, some queries collapsed badly in ja→en, so its cross-lingual ability is directionally asymmetric
• Titan / E5: Maintained stable accuracy even for ja→en. Probably because the English summaries contain more information, there were cases where ja→en scored higher than ja→ja
• Ruri: Accuracy dropped significantly in cross-lingual search (Intent: 0.60→0.45, ja→en for react_hooks had nDCG=0.0)
• Niche: All models struggled in both same-language and cross-lingual search

Heatmap (All Patterns)

Characteristics by Model

Nova Embed: Strong in intent understanding and notation variation, weak in niche topics

Nova overwhelmed the other models in notation variation (test6: 0.8956) and intent understanding (test8: 0.7187).

Below is a concrete example of a notation-variation query (test6). The search uses the phrase "Change serverless function timeout," which avoids the official AWS service name.

Nova Embed (Top-3):

1. I tried fixing the issue where a Lambda function runs multiple times by setting the timeout and retry count → score: 100 ✅
2. Please tell me how to confirm from a Lambda function's logs that it timed out → score: 75
3. I created a Lambda that recovers Lambda functions that errored due to a timeout → score: 65

Titan V2 (Top-3):

1. How can I change the execution timeout of AWS Systems Manager RunCommand? → score: 75
2. Please tell me how to change the integration timeout in API Gateway → score: 95
3. [New Feature] Changing the timeout value of AWS Elastic Load Balancing (ELB) → score: 40

Nova understood that "serverless function" = "Lambda," and its Top-3 were all Lambda-related. Titan stayed faithful to the words "timeout change," and services other than Lambda, such as SSM and API Gateway, crept in.

Below is a concrete example of an intent-understanding query (test8). This is a pattern where a "troubling situation," "Container keeps restarting on its own," is entered in natural language.

Nova Embed (Top-3):

1. An unintended reboot occurred when creating an AMI → score: 20
2. Automatically restarting on exceptions with systemd → score: 75
3. ECS task gets CannotPullContainerError on ECR image update → score: 95 ✅

Titan V2 (Top-3):

1. An unintended reboot occurred when creating an AMI → score: 30
2. How to handle EC2 maintenance that involves a reboot → score: 35
3. How to keep Windows containers running on ECS → score: 75

Nova succeeded in including a high-scoring ECS-related article (score: 95) in its Top-3. Titan stopped at word matching on "restart" and "container," and couldn't bring articles matching the intent to the top.

On the other hand, for niche technology (test10: 0.1576), the situation reverses. These are the results for "Project management using Projen."

Nova Embed (Top-3):

1. Backlog management (product / sprint) using GitHub Projects → score: 20
2. How to use Projects to manage Issues and things smaller than Issues in a well-balanced way → score: 20
3. I tried incorporating project management into the work of (non-engineer) staff → score: 15

Titan V2 (Top-3):

1. Even regular AWS CDK App development was comfortable with projen → score: 95 ✅
2. [Report + I Tried It] projen – a CDK for software project configuration #CDK Day → score: 95 ✅
3. Project Management #1 → score: 15

Nova was pulled toward the generic meaning of "project management" and couldn't find a single Projen article. Titan accurately captured the proper noun "Projen" and hit relevant articles in its Top-2. This is a case where Nova's semantic understanding backfires and Titan complements it well.

Titan Embed V2: A stable runner-up

Titan has no standout strength, but it consistently placed 2nd–3rd across all patterns. In niche technology (test10: 0.5701), it took first place while Nova collapsed, making it a well-balanced model with few weak areas.

OSS Models (E5-Large / Ruri-Base): High basic performance, but passed over for operational reasons

E5-Large took first place in Japanese basic search (test1: 0.8711), English basic search (test3: 0.8953), and single-word search (test4/5). It is the model with the highest accuracy for textbook-style search queries. However, it fell behind Nova in notation variation and intent understanding.

Ruri-Base was last in all 12 patterns, but considering the handicap of having 768 dimensions, lower than the other models (1024 dimensions), it could be said to have done respectably.

However, the OSS models had operational challenges. Real-time vectorization of search terms requires loading the model and running inference. In a CPU-only environment like Lambda, it is difficult to achieve practical latency, and keeping a GPU instance running at all times isn't worth the cost.
There was room to use them for similar-article search between pre-vectorized data, but since we have no plans for fine-tuning or custom optimization of the models, we decided to forgo OSS models this time and limit ourselves to Bedrock managed models.

Vectorization Speed

The models via the Bedrock API required no initialization and were fast. The OSS models required starting a Docker container and loading the model, and E5-Large in particular had a bottleneck of about 1 minute for initialization.

Reference: Sonnet's Second Opinion

To check for Judge bias, I also scored the same data with Claude Sonnet 4.5 (Amazon Bedrock).

Gemini used the five levels (0/20/50/80/100) as instructed, and clear differences between models emerged. Sonnet, on the other hand, used continuous values from 0 to 100 (69% of 1,640 scores were values outside those specified), and all models clustered at nDCG@10 = 0.90–0.95, making differences hard to see.

The agreement rate for the first-place model was 6/11 (55%). The patterns where Nova is strong (cross-lingual, notation variation, intent understanding) and the pattern where Titan is strong (niche technology) agreed between both Judges, but for basic search and single-word search, the Judges split between E5 and Titan. Since rankings can change depending on the choice of Judge in LLM-as-a-Judge, please treat these results as relative trends.

Summary

Guidelines for Choosing a Bedrock Model

Overall Impressions

This comparison made the strengths and weaknesses of each model clear.

By a simple "overall score," the OSS model E5-Large came in first. However, for the semantic search platform we are building, we place the highest priority not on searches with clear keywords but on use cases where users vaguely enter "what's troubling them" in natural language.
Therefore, rather than the height of the overall score, we valued strength in "intent understanding (test8)" and "notation variation (test6)," and decided to adopt Nova Embed, a Bedrock managed model that requires no infrastructure management such as GPUs, as our main model.

On the other hand, the verification also highlighted areas where Nova is weak, such as niche technology and single-word keyword search. For these areas, we are considering using Titan V2, which has few weaknesses and is stable, alongside it, so that the two models complement each other and are used where each fits best.