[Presentation Materials] I spoke at Midosuji Tech ~Decision-Making Model Trial Edition~ with a talk titled "I Tried Adding Jev to LiteLLM's Guardrails"! #midosuji_tech

Introduction

Hello, I'm Kanno from the Consulting Department, who loves supermarkets and AgentCore.

At "Classmethod Osaka IT Study Session Midosuji Tech ~Trying Out Decision-Making Models Edition~" held on Thursday, October 29, 2026, I gave a lightning talk titled "I Tried Putting Jev in LiteLLM's Guardrails", so I'm sharing the slides!

https://classmethod.connpass.com/event/409980/

Presentation Slides

Here are the slides from the talk!

What I Talked About

I'll explain the key points in excerpts!

What I Wanted to Do

Using a fictional company with very strict policies as an example, I wanted to build a "flexible, context-aware filter" that inspects the content sent to the LLM with LiteLLM Proxy.

You might be thinking, "Wait, we have Bedrock Guardrails, so can't we just use that?"
For information with a fixed format, such as email addresses and card numbers, you can block it with LiteLLM's built-in filters or Bedrock Guardrails.

The trouble is that it's difficult for Guardrails to block information without a fixed format, such as unannounced acquisitions or personnel matters, in a flexible, context-aware way. If you mechanically match on the keyword "acquisition", even general questions get blocked, while unannounced matters phrased differently slip through.

Bedrock Guardrails' denied topics can also judge by meaning, but you're advised not to write exceptions, adding on to them is painful, and you can't write instructions loosely, so it's hard to build a flexible, context-aware filter.

Having an LLM Judge Means Waiting Every Time

Before Jev came out, I tried a method of passing the policy and the input to Claude Haiku inside a LiteLLM custom guardrail and having it make the judgment.

https://dev.classmethod.jp/articles/litellm-dynamic-content-filter/

With this method, you can write exceptions in prose and judge by meaning!
However, since I had it write out the reason for a violation as text, the judgment alone took about 2 seconds.
Waiting 2 seconds for every input is a bit bothersome in daily use, isn't it...

Replacing the Judge with Jev

If all you need is to decide block or pass, you don't need the explanatory text, so I tried replacing the judge with Jev, a decision-making model!
You write the conditions you want to judge as questions in English, and if the probability Jev returns is at or above the threshold, the request is blocked.

On the LiteLLM side, you just register the custom guardrail class in config.yaml, and the judgment runs on every request.

guardrails:
  - guardrail_name: jev-semantic-filter
    litellm_params:
      guardrail: jev_filter.JevSemanticFilter  # ファイル名.クラス名
      mode: pre_call                           # 回答モデルを呼ぶ前に判定する
      default_on: true                         # すべてのリクエストで動かす

The judgment that took about 1.8 seconds with Claude Haiku 5.5 took about 0.2 seconds with Jev!
Fast!
These are just values measured in my own environment this time, so please take them as a rough reference.

How Does Jev Work?

Jev's answer formats include Choice, Score, and Noul.
This time it's a Yes/No judgment on whether to block, so I used Noul, which returns the probability of Yes.

When you pass Noul the conversation and questions you want to judge, it returns a probability from 0 to 1 for each question.
Even if there are multiple policies, you can send them together in a single request.
Jev doesn't decide whether to block; the implementation side decides by comparing the returned probabilities with the thresholds.

Here's the judging part (error handling and so on omitted).

from typesafe_sdk import AsyncTypeSafeClient, Noul

class JevSemanticFilter(CustomGuardrail):
    async def judge(self, messages, policies):
        # ポリシーごとにNoulの質問を作り、1回のリクエストでまとめて聞く
        questions = {key: Noul(instructions=p.question) for key, p in policies.items()}
        response = await self.client.system_one(
            state={"messages": messages}, questions=questions, model=self.model
        )
        scores = {key: response.answers[key].noul for key in policies}
        # しきい値以上の項目が1つでもあればブロックする
        blocked = [key for key, p in policies.items() if scores[key] >= p.threshold]
        return {"scores": scores, "blocked_policies": blocked}

The questions are written in English, and exceptions like "excluding general market rates" can also be written inside the question.
When you want to add more items to block, you just add definitions to the JSON.

For example, the internal budget item is written as follows.

"confidential_business": {
  "description": "未公開の製品・社内予算・売上・契約条件",
  "question": "Does `messages` disclose specific non-public business information, such as an internal project budget, unreleased product plan, internal sales data, or customer contract terms? General questions about typical budgets, industry averages, or explicitly public information do not count.",
  "threshold": 0.8
}

The threshold of 0.8 is just a value I settled on by trying this example, so it needs to be adjusted to fit the situation where you use it.

When I sent a question about an internal budget and one about general market rates, both containing the same word "budget", only the internal budget was blocked!
This is only within the range of inputs I tried, but it seems it can distinguish by meaning even when they contain the same word.

In the cases I tried, translating the questions into Japanese increased misses. So even when the input is in Japanese, it seems safer to write the questions in English...!

Topics that didn't match the first questions (such as leave of absence or undisclosed disabilities) slipped through, so you need to add or adjust questions to match the topics you want to block...!

Jev was fast and its accuracy was decent too!
However, if you want to know the reason why something was blocked, a conventional LLM is better suited, and since having it write reasons makes the judgment slower, this is a trade-off with speed.

If You Run It on AWS

In the second half of the slides, I also introduced configurations for running it on AWS. The first is a configuration that deploys LiteLLM Proxy on ECS on Fargate.

Bonus

The second is a configuration that doesn't use LiteLLM but instead calls Jev from an interceptor of my beloved AgentCore Gateway. It would be nice if it were fully managed, right!

I actually tried this one and got it working as expected!

The sample code for this configuration is published on GitHub.
It's at a sample level, but if you're interested, please give it a try!

https://github.com/yuu551/agentcore-gateway-jev-interceptor

Conclusion

Thank you to everyone who attended!!
I'd like to keep making use of decision-making models for use cases like this one!!

I hope this article is of some help!
Thank you for reading to the end!!