Tried running "Kev", an OSS implementation of a compatible API for the judgment-specialized AI "Jev", on Docker

Introduction

"Jev" is a decision-specialized AI that has been attracting attention for its low pricing and fast response times.

https://dev.classmethod.jp/articles/jev-guide-with-examples/

"Kev," an OSS decision-specialized AI (decision model) that implements the same API as Jev, has been released under the Apache-2.0 license. It comes in multiple sizes, and the smallest, Kev-0.8B (based on Qwen3.5-0.8B-Base), is said to run even in environments without a GPU.

This article presents the results of testing Kev-0.8B on Docker running on a first-generation M1 Mac (16GB memory, 8-core CPU) with Linux installed.

Kev's API and Model Sizes

https://github.com/jaredpalmer/kev

Kev is a decision model being developed at jaredpalmer/kev. Since it accepts the same POST /v1/systemone as Jev, the TypeSafe SDK can be used with just an endpoint change.

The model sizes are four: 0.8B / 4B / 9B / 27B. The README recommends 4B as the standard, but 4B and above require a Mac with 32GB of memory or a GPU. Since 0.8B is the only size that achieves practical speeds on CPU, this article evaluates 0.8B.

Operating Environment

Since this is running on Linux rather than macOS, the GPU (Metal) cannot be used, and PyTorch falls back to the CPU path.

Starting Kev-0.8B with Docker

To pin the version and model, we build and start from the source in the official repository.

docker run -d --name kev-server \
  -p 127.0.0.1:8009:8009 \
  -e KEV_API_KEY=localkey \
  -e OMP_NUM_THREADS=4 \
  python:3.12-slim \
  sh -c "
    apt-get update -qq && apt-get install -qq -y git build-essential g++
    pip install --quiet uv
    git clone --quiet https://github.com/jaredpalmer/kev /kev
    cd /kev && git checkout --quiet f153596
    uv sync --quiet --extra serve
    uv run python -m kev.serve --run jaredpalmer/kev-0.8b --host 0.0.0.0 --port 8009
  "

f153596 is the latest commit at the time of writing. The model jaredpalmer/kev-0.8b is automatically downloaded at startup and requires no authentication. Including dependency resolution and model fetching, startup took approximately 90 seconds.

Fetching 9 files: 100%|██████████| 9/9 [00:02<00:00]
Loading weights: 100%|██████████| 320/320 [00:00<00:00]
INFO:     Started server process [1434]
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8009 (Press CTRL+C to quit)

After startup, calling /v1/models returns the execution configuration.

run=jaredpalmer/kev-0.8b device=cpu backend=torch dtype=float32 temperature=2.3510958125672174

The probabilities included in the response are values calibrated with the temperature bundled with each checkpoint. The displayed value of 2.35 is for 0.8B.

Setting KEV_API_KEY

Kev responds without authentication if KEV_API_KEY is not set (kev/serve.py, commit f153596).

API_KEY = os.environ.get("KEV_API_KEY")   # unset = open server; set = require Authorization: Bearer <key>

With a container where it is not set, 200 was returned even without an authorization header or with a wrong key (when the key is set, 401 was returned without an authorization header). In the startup command above, the key is passed with -e KEV_API_KEY=localkey.

Also, the default bind address is 127.0.0.1, but --host 0.0.0.0 is required for use with Docker. Instead, -p 127.0.0.1:8009:8009 restricts the host-side listening address to loopback.

Baking the Model into a Dockerfile

If you don't want to wait for the clone and uv sync every time, you can build an image with the model baked in at build time. In this form, startup took approximately 30 seconds.

FROM python:3.12-slim

RUN apt-get update -qq \
 && apt-get install -qq -y --no-install-recommends git build-essential g++ \
 && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir uv==0.9.7

RUN git clone https://github.com/jaredpalmer/kev /kev
WORKDIR /kev
RUN git checkout f153596
RUN uv sync --extra serve

# Bake the model in at build time (no waiting on first startup)
RUN uv run python -c "from huggingface_hub import snapshot_download; snapshot_download('jaredpalmer/kev-0.8b')"

# Pass KEV_API_KEY and OMP_NUM_THREADS via docker run -e, not baked into ENV
#   KEV_API_KEY     : Must be passed since the server responds without authentication if unset. Writing in ENV causes docker build to warn SecretsUsedInArgOrEnv
#   OMP_NUM_THREADS : The optimal value varies depending on the host CPU configuration

EXPOSE 8009
CMD ["uv", "run", "python", "-m", "kev.serve", \
     "--run", "jaredpalmer/kev-0.8b", "--host", "0.0.0.0", "--port", "8009"]

docker build -f Dockerfile-kev -t kev-local .
docker run -d --name kev-server -p 127.0.0.1:8009:8009 -e KEV_API_KEY=localkey -e OMP_NUM_THREADS=4 kev-local

Sending Questions to the API

Pass the text to be evaluated (state) and questions (questions) to POST /v1/systemone. The answer format is specified per question using type. There are 3 formats: choice for selecting from options, noul for returning a yes/no as a single probability, and score for graduated ratings. For choice, list the options in criteria. For authentication, pass the KEV_API_KEY set at startup as a Bearer token.

Here is an example asking a single choice question.

curl -s -X POST http://localhost:8009/v1/systemone \
  -H "Authorization: Bearer localkey" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "インフラ設計のレビューをしたい",
    "model": "kev-latest",
    "questions": {
      "skill": {
        "type": "choice",
        "instructions": "Which skill fits this message?",
        "criteria": {
          "aidlc-infrastructure-design": null,
          "aidlc-bugfix": null,
          "aidlc-market-research": null,
          "none": null
        }
      }
    }
  }'

{
    "model": "kev-latest",
    "answers": {
        "skill": {
            "type": "choice",
            "choice": "aidlc-infrastructure-design",
            "confidence": 0.705,
            "probabilities": {
                "aidlc-infrastructure-design": 0.7788,
                "aidlc-bugfix": 0.0078,
                "aidlc-market-research": 0.0016,
                "none": 0.2118
            }
        }
    },
    "usage": {
        "input_tokens": 39,
        "output_tokens": 89
    },
    "latency_ms": 1029.4
}

The response format is the same as Jev, containing choice and probabilities. confidence is not an accuracy rate, but a value the README defines as (p_max − 1/K) / (1 − 1/K) (where K is the number of options).

Here is a response assuming a support ticket routing scenario, asking for the responsible department (choice), whether escalation is needed (noul), and the degree of frustration (score) in a single request.

{
    "model": "kev-latest",
    "answers": {
        "department": {
            "type": "choice",
            "choice": "shipping",
            "confidence": 0.2625,
            "probabilities": {"returns": 0.338, "shipping": 0.5083, "billing": 0.1537}
        },
        "escalate": {"type": "noul", "noul": 0.4363},
        "frustration": {
            "type": "score",
            "score": 1.3551,
            "legend": {"0": "Calm", "1": "Frustrated", "2": "Very angry"},
            "probabilities": {"0": 0.041, "1": 0.5629, "2": 0.3961},
            "confidence": 0.3444
        }
    },
    "usage": {"input_tokens": 102, "output_tokens": 180},
    "latency_ms": 994.8
}

Resources to Allocate to the Container

Minimum Memory Required

Results from changing the docker run --memory value at startup.

With the default fp32, the weights are 0.8B × 4 bytes = approximately 3.2GB (3.0GiB), and with buffers added, it was just over 3.5GiB. Allocate 4GB or more of memory to the container.

Reference: Response Time Changes with CPU Settings

latency_ms values measured on M1 CPU. To exclude the effect of prompt caching, all values are from the first time the same input was sent (number of measurements is shown in the column headers).

Match the thread count to the number of performance cores. Changing from the default 8 threads to OMP_NUM_THREADS=4 made processing about 3x faster for a single question. This is why OMP_NUM_THREADS=4 is passed at startup.

The default fp32 is the fastest dtype. KEV_DTYPE=bf16 was 7.6x slower compared to the default threads. This is because the M1's CPU does not have bf16 arithmetic instructions, so low-precision matrix multiplication is not optimized. fp16 was also 6x slower.

On the other hand, switching to bf16 reduces the required memory, and 0.8B could start with a 2GB limit, and 4B with an 11GB limit. However, the response time for 4B running in bf16 was 26 to 93 seconds. Using 4B in a CPU environment was ruled out due to the long response times.

Verifying Answer Accuracy

In the README's model table (commit f153596), the accuracy for new sources (development column) is 0.648 for Kev-0.8B, 0.817 for Kev-4B, and 0.857 for Jev. However, it is also noted that this is not a controlled comparison because Jev's training data is unknown.

Locally, 5 cases (7 answers) were sent to both 0.8B and 4B in the same format, and the 3 cases with a uniquely determined correct answer are shown in the table below.

For classification tasks where the decision cues are explicitly present in the input, such as skill selection for a clear request or knowledge questions, 0.8B returned the same answer as 4B. On the other hand, for problems requiring inference such as sentence entailment judgment, 0.8B assigned a probability exceeding 0.9 to an incorrect answer. This is the failure referred to as confident errors in the README. A high probability does not necessarily mean the answer is correct.

Summary

Kev, an OSS model API-compatible with Jev, was confirmed to work even in a Docker environment with CPU only and 4GB of memory, though with limitations in model size and performance.

This time, the smallest model was evaluated due to execution environment constraints, but if a host with a GPU or a macOS environment where Apple Silicon native (MLX) can be used is available, higher-performance models may also work.

Please try Kev if you want to evaluate a model specialized for judgment and classification, or if you do not want to send input data during inference to an external API.