LiteLLMとDeepEvalでLangGraphエージェントの応答品質を自動評価してみる

Python 3.13
litellm 1.83.14
langgraph 1.1.10
langchain-litellm 0.6.4
deepeval 2.0.0
pytest 8.0.0

$ uv pip install litellm langgraph langchain-litellm deepeval pytest
$ export OPENAI_API_KEY="sk-..."  # DeepEvalの審判モデル用
$ export ANTHROPIC_API_KEY="sk-ant-..."

"""DeepEval で LangGraph エージェントの応答品質を評価する。

メトリクス:
- AnswerRelevancyMetric: 質問に関連した回答か
- FaithfulnessMetric: コンテキストに忠実か（ハルシネーションがないか）
- HallucinationMetric: 矛盾した情報を含まないか
"""

from deepeval import evaluate
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
)
from deepeval.test_case import LLMTestCase
from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_litellm import ChatLiteLLM

@tool
def search_doc(query: str) -> str:
    """社内ドキュメントを検索します。"""
    docs = {
        "有給": "有給休暇は入社6ヶ月後に10日付与されます。",
        "リモート": "リモートワークは週3日まで可能です。",
    }
    for k, v in docs.items():
        if k in query:
            return v
    return "情報が見つかりませんでした"

llm = ChatLiteLLM(model="openai/gpt-5-mini")
agent = create_agent(model=llm, tools=[search_doc])

def run_agent(question: str) -> tuple[str, list[str]]:
    """エージェントを実行し、(回答, 取得したコンテキスト) を返す。"""
    result = agent.invoke({"messages": [{"role": "user", "content": question}]})
    final = str(result["messages"][-1].content)
    # ツール出力（コンテキスト）を抽出
    contexts = [
        str(m.content) for m in result["messages"] if getattr(m, "type", None) == "tool"
    ]
    return final, contexts

# 評価対象のテストケース
test_cases: list[LLMTestCase] = []
for q, expected in [
    ("有給休暇のルールを教えて", "入社6ヶ月後に10日付与"),
    ("リモートワークの可能日数は？", "週3日まで可能"),
]:
    answer, contexts = run_agent(q)
    test_cases.append(
        LLMTestCase(
            input=q,
            actual_output=answer,
            expected_output=expected,
            retrieval_context=contexts,
            context=contexts,
        )
    )

# 評価実行: judge モデルは OPENAI_API_KEY を使った DeepEval デフォルト
# （高性能な GPT が自動選択される）。
# 別プロバイダーに切替えたい場合は LiteLLMModel を import して
# AnswerRelevancyMetric(model=LiteLLMModel(model="anthropic/claude-sonnet-4-6"))
# のように渡せる。
results = evaluate(
    test_cases=test_cases,
    metrics=[
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.7),
        HallucinationMetric(threshold=0.5),
    ],
)
print(results)

$ python agent_eval.py
✨ You're running DeepEval's latest Answer Relevancy Metric! (using gpt-5.4, strict=False, async_mode=True)...
✨ You're running DeepEval's latest Faithfulness Metric! (using gpt-5.4, strict=False, async_mode=True)...
✨ You're running DeepEval's latest Hallucination Metric! (using gpt-5.4, strict=False, async_mode=True)...

======================================================================

Metrics Summary

  - ✅ Answer Relevancy (score: 1.0, threshold: 0.7, evaluation model: gpt-5.4, reason: The score is 1.00 because the response appears fully relevant to the question, with no irrelevant statements noted.)
  - ✅ Faithfulness (score: 1.0, threshold: 0.7, evaluation model: gpt-5.4, reason: The score is 1.00 because there are no contradictions, so the actual output appears fully consistent with the retrieval context.)
  - ✅ Hallucination (score: 0.0, threshold: 0.5, evaluation model: gpt-5.4, reason: The score is 0.00 because the actual output fully aligns with the context, correctly stating that remote work is possible up to 3 days per week.)

For test case:
  - input: リモートワークの可能日数は？
  - actual output: 原則、週3日まで可能です。
  - expected output: 週3日まで可能
  - retrieval context: ['リモートワークは週3日まで可能です。']

======================================================================

Overall Metric Pass Rates
Answer Relevancy: 100.00% pass rate
Faithfulness:     100.00% pass rate
Hallucination:    100.00% pass rate

✓ Evaluation completed 🎉! (time taken: 30.32s | token cost: 0.1013725 USD)
» Test Results (2 total tests):
   » Pass Rate: 100.0% | Passed: 2 | Failed: 0

"""pytest 経由で DeepEval を回す。CIに組み込みやすい。"""

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_litellm import ChatLiteLLM

@tool
def search_doc(query: str) -> str:
    """社内ドキュメントを検索します。"""
    return "有給休暇は入社6ヶ月後に10日付与されます。"

llm = ChatLiteLLM(model="openai/gpt-5-mini")
agent = create_agent(model=llm, tools=[search_doc])

@pytest.mark.parametrize(
    ("question", "expected_substr"),
    [
        ("有給休暇は何日？", "10日"),
        ("入社後いつから有給？", "6ヶ月"),
    ],
)
def test_answer_quality(question: str, expected_substr: str) -> None:
    result = agent.invoke({"messages": [{"role": "user", "content": question}]})
    answer = str(result["messages"][-1].content)
    contexts = [
        str(m.content) for m in result["messages"] if getattr(m, "type", None) == "tool"
    ]
    test_case = LLMTestCase(
        input=question,
        actual_output=answer,
        expected_output=expected_substr,
        retrieval_context=contexts,
    )
    assert_test(
        test_case,
        [
            AnswerRelevancyMetric(threshold=0.7),
            FaithfulnessMetric(threshold=0.7),
        ],
    )

$ pytest test_agent_pytest.py -v
============================= test session starts ==============================
plugins: deepeval-3.9.9, asyncio-1.3.0, langsmith-0.7.38, ...
collected 2 items

test_agent_pytest.py::test_answer_quality[有給休暇は何日？-10日] PASSED  [ 50%]
test_agent_pytest.py::test_answer_quality[入社後いつから有給？-6ヶ月] PASSED  [100%]

=================== 2 passed, 1 warning in 60.21s (0:01:00) ====================