DeepEvalの公式Jev対応でjudgeを判定特化モデル「Jev」に置き換えて検出力・安定性・コストを比べてみる

Python 3.13.15
deepeval 4.2.6
typesafe-sdk 0.7.1
litellm 1.102.0
langgraph 1.2.11
langchain 1.4.2
langchain-litellm 0.7.2

$ uv pip install deepeval typesafe-sdk litellm langgraph langchain langchain-litellm
$ export OPENAI_API_KEY="sk-..."      # GPT judge と被評価エージェント用
$ export TYPESAFE_API_KEY="..."       # Jev 用

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, HallucinationMetric
from deepeval.models import OpenAIModel, TypeSafeModel

JEV_MODEL = "jev-1.13.0"  # 料金はモデルの版に紐づくので jev-latest ではなく版を固定する

def jev_metrics(include_reason: bool):
    """標準メトリクスを Jev だけで判定する（eval_mode="system_one"）。理由は Jev の答えから機械的に組み立てられる。"""
    jev = TypeSafeModel(model=JEV_MODEL)
    kw = {"system_one_model": jev, "eval_mode": "system_one", "include_reason": include_reason}
    return [
        AnswerRelevancyMetric(threshold=0.7, **kw),
        FaithfulnessMetric(threshold=0.7, **kw),
        HallucinationMetric(threshold=0.5, **kw),
    ]

def hybrid_metrics(include_reason: bool):
    """LLM が主張の抽出と理由を担当し、主張ごとの判定だけ Jev に聞く（eval_mode="hybrid"）。"""
    kw = {
        "model": OpenAIModel(model="gpt-5.4"),
        "system_one_model": TypeSafeModel(model=JEV_MODEL),
        "eval_mode": "hybrid",
        "include_reason": include_reason,
    }
    return [
        AnswerRelevancyMetric(threshold=0.7, **kw),
        FaithfulnessMetric(threshold=0.7, **kw),
        HallucinationMetric(threshold=0.5, **kw),
    ]

[
  {
    "type": "noul",
    "statement": "Every factual claim in `actual_output` is supported by the facts in `retrieval_context`.",
    "weight": 2
  },
  {
    "type": "noul",
    "statement": "No claim in `actual_output` contradicts the facts in `retrieval_context`."
  },
  {
    "type": "score",
    "question": "How much of `actual_output` is grounded in the facts in `retrieval_context`?",
    "levels": ["Fabricated", "Mostly fabricated", "Mostly grounded", "Fully grounded"]
  }
]

from deepeval.metrics import JevEval
from deepeval.metrics.jev_eval import Noul
from deepeval.test_case import SingleTurnParams

def jev_eval_metrics(include_reason: bool):
    """JevEval に、LLM 版の標準メトリクスと同じ定義を自分の言葉で書いた版。質問は 1 メトリクス 1 問。"""
    jev = TypeSafeModel(model=JEV_MODEL)
    P = SingleTurnParams
    kw = {"system_one_model": jev, "include_reason": include_reason}
    return [
        JevEval(
            name="Jev Answer Relevancy",
            evaluation_params=[P.INPUT, P.ACTUAL_OUTPUT],
            questions=[Noul("`actual_output` は `input` への回答として関係がある（無関係な話題ではない）")],
            threshold=0.7,
            **kw,
        ),
        JevEval(
            name="Jev Faithfulness",
            evaluation_params=[P.ACTUAL_OUTPUT, P.RETRIEVAL_CONTEXT],
            questions=[Noul("`actual_output` に `retrieval_context` の内容と矛盾する記述は無い")],
            threshold=0.7,
            **kw,
        ),
        JevEval(
            name="Jev Hallucination",
            evaluation_params=[P.ACTUAL_OUTPUT, P.CONTEXT],
            questions=[Noul("`actual_output` に `context` の文書と矛盾する記述は無い")],
            threshold=0.5,
            **kw,
        ),
    ]

agent = create_agent(
    model=ChatLiteLLM(model="openai/gpt-5-mini"),
    tools=[search_doc],
    system_prompt="必ず search_doc で社内ドキュメントを検索し、その結果だけを根拠に 1〜2 文で答える。",
)

from deepeval.models.llms.constants import OPENAI_MODELS_DATA, make_model_data

# 単価は https://developers.openai.com/api/docs/pricing（USD / 1M tokens）
for name, in_price, out_price in [("gpt-6-luna", 0.10, 0.50), ("gpt-5.4-nano", 0.20, 1.25)]:
    OPENAI_MODELS_DATA[name] = make_model_data(
        supports_log_probs=False,
        supports_multimodal=True,
        supports_structured_outputs=True,
        supports_json=True,
        supports_temperature=False,
        input_price=in_price / 1e6,
        output_price=out_price / 1e6,
    )

EXPECT = {
    "standard": {
        "正解: 有給": {"relevancy": True, "faithfulness": True, "hallucination": True},
        "正解: リモート": {"relevancy": True, "faithfulness": True, "hallucination": True},
        "矛盾: 週3日を週5日と答える": {"relevancy": True, "faithfulness": False, "hallucination": False},
        "無関係: 有給の質問にリモートの話": {"relevancy": False, "faithfulness": True, "hallucination": None},
        "際どい: 6ヶ月後を6ヶ月以内と言い換える": {"relevancy": True, "faithfulness": False, "hallucination": False},
        "根拠なし: 正しい一文に作り話を足す": {"relevancy": True, "faithfulness": None, "hallucination": None},
    },
}
EXPECT["grounded"] = {
    **EXPECT["standard"],
    "無関係: 有給の質問にリモートの話": {"relevancy": False, "faithfulness": False, "hallucination": None},
    "根拠なし: 正しい一文に作り話を足す": {"relevancy": True, "faithfulness": False, "hallucination": None},
}

$ python compare_judges.py --runs 3

    reason[矛盾: 週3日を週5日と答える / faithfulness]: Decided by jev-1.13.0 (TypeSafe AI), minimum confidence 0.50. | 1. Every factual claim in `actual_output` is supported by the facts in `retrieval_context`. -> clearly fails (P(yes)=0.04, confidence=0.92, weight=2) | 2. No claim in `actual_o…

    reason[際どい: 6ヶ月後を6ヶ月以内と言い換える / faithfulness]: The score is 1.00 because there are no contradictions; the actual output is fully faithful to the retrieval context.
    reason[際どい: 6ヶ月後を6ヶ月以内と言い換える / hallucination]: The score is 0.00 because the output says the 10 days are granted within six months of joining, while the context says they are granted six months after joining.

    reason[際どい: 6ヶ月後を6ヶ月以内と言い換える / faithfulness]: Decided by jev-1.13.0 (TypeSafe AI), minimum confidence 0.00. | 1. Every factual claim in `actual_output` is supported by the facts in `retrieval_context`. -> unclear (P(yes)=0.50, confidence=0.00, weight=2) | 2. No claim in `actual_output`…

    reason[根拠なし: 正しい一文に作り話を足す / faithfulness]: The score is 1.00 because there are no contradictions, so the actual output appears fully consistent with the retrieval context—great job staying faithful to the source.
    reason[根拠なし: 正しい一文に作り話を足す / faithfulness]: Decided by jev-1.13.0 (TypeSafe AI), minimum confidence 0.86. | 1. Every factual claim in `actual_output` is supported by the facts in `retrieval_context`. -> clearly fails (P(yes)=0.04, confidence=0.92, weight=2) | 2. No claim in `actual_o…

    reason[矛盾: 週3日を週5日と答える / faithfulness]: The score is 0.00 because the actual output directly conflicts with the retrieval context by stating P(yes)=0.02.
    reason[根拠なし: 正しい一文に作り話を足す / faithfulness]: The score is 0.50 because the actual output appears to conflict with the retrieval context on a key probability value, stating P(yes)=0.08.

$ python compare_judges.py --runs 3 --no-reason gpt-5.4 gpt-6-luna gpt-5.4-nano hybrid jev

from deepeval.models import TypeSafeModel

jev = TypeSafeModel(model="jev-1.13.0")

def test_answer_quality(question: str, expected_substr: str) -> None:
    ...
    assert_test(
        test_case,
        [
            AnswerRelevancyMetric(threshold=0.7, system_one_model=jev, eval_mode="system_one"),
            FaithfulnessMetric(threshold=0.7, system_one_model=jev, eval_mode="system_one"),
        ],
    )

$ pytest test_agent_pytest_jev.py -v
============================= test session starts ==============================
plugins: deepeval-4.2.6, repeat-0.9.4, xdist-3.8.0, asyncio-1.4.0, rerunfailures-16.7, anyio-4.15.1, langsmith-0.13.0
collected 2 items

test_agent_pytest_jev.py::test_answer_quality[有給休暇は何日？-10日] PASSED [ 50%]
test_agent_pytest_jev.py::test_answer_quality[入社後いつから有給？-6ヶ月] PASSED [100%]

======================== 2 passed, 6 warnings in 9.45s =========================