Tried running "Kev", an OSS model compatible with the judgment-specialized AI "Jev", on Docker

Introduction

"Jev" is a decision-focused AI that has been gaining attention for its low cost and fast response times.

https://dev.classmethod.jp/articles/jev-guide-with-examples/

"Kev," an OSS decision-focused AI (decision model) that implements the same API as Jev, has been released under the Apache-2.0 license. It comes in multiple sizes, and the smallest, Kev-0.8B (based on Qwen3.5-0.8B-Base), is said to run even in environments without a GPU.

This article presents the results of testing Kev-0.8B on Docker running on a first-generation M1 Mac (16GB memory, 8-core CPU) with Linux installed.

Kev API and Model Sizes

https://github.com/jaredpalmer/kev

Kev is a decision model developed under jaredpalmer/kev. Since it accepts the same POST /v1/systemone as Jev, the TypeSafe SDK can be used with just an endpoint change.

Model sizes are four: 0.8B / 4B / 9B / 27B. The README recommends 4B as the standard, but 4B and above require a Mac with 32GB of memory or a GPU. Since 0.8B is the only size that achieves practical speeds on CPU, this article evaluates 0.8B.

Operating Environment

Since this runs on Linux rather than macOS, the GPU (Metal) is unavailable, and PyTorch uses the CPU path.

Starting Kev-0.8B with Docker

To pin the version and model, build from the official repository source and start it.

docker run -d --name kev-server \
  -p 127.0.0.1:8009:8009 \
  -e KEV_API_KEY=localkey \
  -e OMP_NUM_THREADS=4 \
  python:3.12-slim \
  sh -c "
    apt-get update -qq && apt-get install -qq -y git build-essential g++
    pip install --quiet uv
    git clone --quiet https://github.com/jaredpalmer/kev /kev
    cd /kev && git checkout --quiet f153596
    uv sync --quiet --extra serve
    uv run python -m kev.serve --run jaredpalmer/kev-0.8b --host 0.0.0.0 --port 8009
  "

f153596 is the latest commit at the time of writing. The model jaredpalmer/kev-0.8b is automatically downloaded at startup and requires no authentication. Including dependency resolution and model download, it took about 90 seconds to start.

Fetching 9 files: 100%|██████████| 9/9 [00:02<00:00]
Loading weights: 100%|██████████| 320/320 [00:00<00:00]
INFO:     Started server process [1434]
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8009 (Press CTRL+C to quit)

After startup, calling /v1/models returns the execution configuration.

run=jaredpalmer/kev-0.8b device=cpu backend=torch dtype=float32 temperature=2.3510958125672174

The probabilities included in the response are values calibrated with the temperature bundled per checkpoint. The displayed 2.35 is for 0.8B.

Setting KEV_API_KEY

If KEV_API_KEY is not set, Kev responds without authentication (kev/serve.py, commit f153596).

API_KEY = os.environ.get("KEV_API_KEY")   # unset = open server; set = require Authorization: Bearer <key>

In a container without this set, a 200 was returned even without an authorization header or with an incorrect key (when the key is set, a 401 was returned without an authorization header). The startup command above passes the key with -e KEV_API_KEY=localkey.

Also, the default bind address is 127.0.0.1, but --host 0.0.0.0 is required for use with Docker. To compensate, -p 127.0.0.1:8009:8009 restricts the host-side listening address to loopback.

Baking the Model into the Dockerfile

If you don't want to wait for clone and uv sync every time, create an image with the model baked in at build time. With this approach, startup took about 30 seconds.

FROM python:3.12-slim

RUN apt-get update -qq \
 && apt-get install -qq -y --no-install-recommends git build-essential g++ \
 && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir uv==0.9.7

RUN git clone https://github.com/jaredpalmer/kev /kev
WORKDIR /kev
RUN git checkout f153596
RUN uv sync --extra serve

# Bake the model in at build time (no waiting on first startup)
RUN uv run python -c "from huggingface_hub import snapshot_download; snapshot_download('jaredpalmer/kev-0.8b')"

# Pass KEV_API_KEY and OMP_NUM_THREADS via docker run -e, not hardcoded in ENV
#   KEV_API_KEY     : Must be passed since the server responds without authentication if unset. Writing it in ENV causes docker build to warn SecretsUsedInArgOrEnv
#   OMP_NUM_THREADS : The optimal value varies depending on the host CPU configuration

EXPOSE 8009
CMD ["uv", "run", "python", "-m", "kev.serve", \
     "--run", "jaredpalmer/kev-0.8b", "--host", "0.0.0.0", "--port", "8009"]

docker build -f Dockerfile-kev -t kev-local .
docker run -d --name kev-server -p 127.0.0.1:8009:8009 -e KEV_API_KEY=localkey -e OMP_NUM_THREADS=4 kev-local

Sending Questions to the API

Pass the text to evaluate (state) and questions (questions) to POST /v1/systemone. The answer format is specified per question using type. There are three formats: choice for selecting from options, noul for returning a yes/no as a single probability, and score for rating scales. For choice, list the options in criteria. For authentication, pass the KEV_API_KEY set at startup as a Bearer token.

Here is an example asking a single choice question.

curl -s -X POST http://localhost:8009/v1/systemone \
  -H "Authorization: Bearer localkey" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "インフラ設計のレビューをしたい",
    "model": "kev-latest",
    "questions": {
      "skill": {
        "type": "choice",
        "instructions": "Which skill fits this message?",
        "criteria": {
          "aidlc-infrastructure-design": null,
          "aidlc-bugfix": null,
          "aidlc-market-research": null,
          "none": null
        }
      }
    }
  }'

{
    "model": "kev-latest",
    "answers": {
        "skill": {
            "type": "choice",
            "choice": "aidlc-infrastructure-design",
            "confidence": 0.705,
            "probabilities": {
                "aidlc-infrastructure-design": 0.7788,
                "aidlc-bugfix": 0.0078,
                "aidlc-market-research": 0.0016,
                "none": 0.2118
            }
        }
    },
    "usage": {
        "input_tokens": 39,
        "output_tokens": 89
    },
    "latency_ms": 1029.4
}

The response format is the same as Jev, containing choice and probabilities. confidence is not an accuracy rate, but a value the README defines as (p_max − 1/K) / (1 − 1/K) (where K is the number of options).

Here is a response assuming support ticket routing, asking for the responsible department (choice), whether escalation is needed (noul), and the degree of frustration (score) in a single request.

{
    "model": "kev-latest",
    "answers": {
        "department": {
            "type": "choice",
            "choice": "shipping",
            "confidence": 0.2625,
            "probabilities": {"returns": 0.338, "shipping": 0.5083, "billing": 0.1537}
        },
        "escalate": {"type": "noul", "noul": 0.4363},
        "frustration": {
            "type": "score",
            "score": 1.3551,
            "legend": {"0": "Calm", "1": "Frustrated", "2": "Very angry"},
            "probabilities": {"0": 0.041, "1": 0.5629, "2": 0.3961},
            "confidence": 0.3444
        }
    },
    "usage": {"input_tokens": 102, "output_tokens": 180},
    "latency_ms": 994.8
}

Resources to Allocate to the Container

Minimum Memory Requirement

Results from changing the docker run --memory value at startup.

With the default fp32, the weights are 0.8B × 4 bytes = approximately 3.2GB (3.0GiB), and with buffers added, it exceeded 3.5GiB. Allocate 4GB or more of memory to the container.

Reference: Response Time Variation by CPU Settings

latency_ms measured on M1 CPU. To exclude prompt cache effects, all values are from the first time the same input was sent (number of measurements is shown in column headers).

Match the thread count to the number of performance cores. Changing from the default 8 threads to OMP_NUM_THREADS=4 made single-question responses about 3x faster. This is why OMP_NUM_THREADS=4 is passed at startup.

The default fp32 dtype is the fastest. KEV_DTYPE=bf16 was 7.6x slower compared to the same default thread count. The M1 CPU lacks bf16 arithmetic instructions, so low-precision matrix multiplication is not optimized. fp16 was also 6x slower.

On the other hand, using bf16 reduces memory requirements, allowing 0.8B to start with a 2GB limit and 4B with an 11GB limit. However, the response time for 4B running in bf16 was 26–93 seconds. Using 4B in a CPU environment was ruled out due to the long response times.

Verifying Answer Accuracy

In the README's model table (commit f153596), the accuracy for new sources (development column) is 0.648 for Kev-0.8B, 0.817 for Kev-4B, and 0.857 for Jev. However, it is also noted that this is not a controlled comparison since Jev's training data is unknown.

Locally, 5 cases (7 answers) were sent to both 0.8B and 4B in the same format, and the 3 cases with a unique correct answer are shown in the table below.

For classification tasks where the decision cues are explicitly stated in the input—such as skill selection for a clear request or knowledge questions—0.8B returned the same answer as 4B. On the other hand, for problems requiring reasoning such as determining sentence entailment, 0.8B assigned a probability exceeding 0.9 to the wrong answer. This is the type of failure referred to as confident errors in the README. Even a high probability does not necessarily mean the answer is correct.

Summary

Kev, the OSS model with API compatibility with Jev, has been confirmed to work even in a Docker environment with CPU only and 4GB of memory, despite limitations in model size and performance.

This time, the smallest model was evaluated due to execution environment constraints, but on a host with a GPU or in a macOS environment where Apple Silicon native (MLX) is available, higher-performance models may also be usable.

Please try Kev if you want to evaluate a model specialized for judgment and classification, or if you do not want to send inference input data to an external API.