I Tried Decision 2.0 on an M4 Mac's MPS Using the Same 20 Questions as Strands Decider

Introduction

Decision 2.0 is a series of decision-only models for vLLM Semantic Router. The Hugging Face repositories were created between September 28 and 29, 2026. Six sizes, from Kai-0.6B to Vega-27B, have been released.

https://huggingface.co/vllm-sr/Decision-2.0-Eos-0.8B

I ran three of them, Eos-0.8B, Sol-2B, and Nox-4B, on the same 20 questions and the same decision prompt as in the previous article on Strands Decider 2B.

https://dev.classmethod.jp/articles/strands-decider-2b-m4-mac-kiro-crew/

Differences from Kev and Strands Decider

For details on Kev and Strands Decider, see their respective model cards and announcements.

https://github.com/jaredpalmer/kev

https://strandsagents.com/blog/introducing-strands-decider/

Conditions tested

The decision prompt uses the same instructions as model.route in Kiro Crew. Because the criteria in Crew's decision prompt is null, Decider rejected it with HTTP 422 in the previous article. Eos-0.8B also returned invalid_question without running inference. Therefore, as in the previous article, I used B_desc, which adds the tier descriptions as criteria. The state was passed in the form {"message": body text}. The 20 Japanese questions are also identical to the previous article. For each condition, I ran one question first as a warm-up and excluded it from measurement.

INSTR_B = (
    "How hard is this request for an AI coding assistant? Answer one of: "
    "simple = a short, local, mechanical request answerable in one step -- a rename, a lookup, a small edit, a direct factual question; "
    "medium = ordinary work over a few files or steps -- write a function, explain some code, fix a bug whose cause is already named; "
    "complex = work needing a plan, a trade-off or reasoning across a whole system -- design, architecture, a diagnosis with no named cause, a risky refactor. "
    "Being wrong in the two directions does not cost the same. A turn put in a lower tier than it needs gets less room to work in and a worse answer; "
    "a turn put higher only costs more. So answer simple only when the request is self-contained and you can see the whole of it -- "
    "a request that is a plan, a judgement call, several instructions at once, or that names work you cannot see, is not simple however short it is."
)

TIER_DESC = {
    "simple": "a short, local, mechanical request answerable in one step -- a rename, a lookup, a small edit, a direct factual question",
    "medium": "ordinary work over a few files or steps -- write a function, explain some code, fix a bug whose cause is already named",
    "complex": "work needing a plan, a trade-off or reasoning across a whole system -- design, architecture, a diagnosis with no named cause, a risky refactor",
}

The execution environment was a Mac mini (M4 Pro, 24GB, macOS 27.0.1) with Python 3.12.14, torch 2.14.1, and transformers 5.18.0. The 20 questions were run 3 times for each of the 3 models.

I pinned the commit of each model (3594047d for Eos-0.8B, 23cbe9f9 for Sol-2B, and 25e8f67d for Nox-4B). The Decider and Kev-4B values in the results tables are from the logs of the previous article, measured on the same Mac.

How to run it

Create a virtual environment with uv and install the dependencies.

uv venv --python 3.12
uv pip install torch==2.14.1 transformers==5.18.0 safetensors==0.8.0

I saved the above decision prompt to prompts.py and called it from the following Python code.

from transformers import AutoModel
from prompts import INSTR_B, TIER_DESC

model = AutoModel.from_pretrained(
    "vllm-sr/Decision-2.0-Eos-0.8B",
    trust_remote_code=True,
    revision="3594047d69f476f1d01cf84c593e213fc3a4dfe0",
    device="mps",
)
model.eval()

text = "このリポジトリで使っている Node.js のバージョンは？"
result = model.system_one(
    state={"message": text},
    questions={"tier": {"type": "choice", "instructions": INSTR_B, "criteria": TIER_DESC}},
)

trust_remote_code=True executes Python code on Hugging Face, so I pinned the commit by specifying revision.

Every time the model loaded, a tokenizer warning, incorrect regex pattern … fix_mistral_regex=True, appeared. To check the effect of the warning, I replaced the runtime's internal tokenizer (runtime.backend.tokenizer) with one reloaded using fix_mistral_regex=True. When I re-ran Eos-0.8B on the 20 B_desc questions on the M4 Pro's MPS, the tiers were unchanged compared with before the replacement, and the probability difference was at most 0.046. The values in the results tables are from before the replacement.

Results

Decisions

"Within expected" is the number of questions where the tier fell in the "Acceptable" column, and "Exact match" is the number of questions where the model returned the same tier as the expected answer. Eos-0.8B's decisions were identical across the 3 runs.

The number of exact matches with the expected answer for Eos-0.8B is 14, the same as Decider. On the other hand, Eos-0.8B and Decider returned the same tier on 12 of the 20 questions. I also ran Sol-2B and Nox-4B under the same conditions. The number within expected did not increase as the size grew, and all 4 out-of-expectation results for Nox-4B are simple decisions.

Response time

The Decider and Kev-4B values were measured on the client side by sending requests to the HTTP server, in a session of the previous article separate from this one. Kev-4B ran on MLX (bf16). For Eos-0.8B, Sol-2B, and Nox-4B, I measured the system_one() call within the process.

The mean, median, and min–max for Eos-0.8B, Sol-2B, and Nox-4B are values aggregated over the 3 runs (60 responses). The mean, median, and 20-question total for Decider are ranges of the per-run values over the 3 runs. Kev-4B had the same values in both runs. Eos-0.8B was about 100ms longer than Decider on average, and Sol-2B and Nox-4B were longer still than Eos-0.8B.

Memory

footprint is the Physical footprint from the macOS footprint command, and includes the GPU portion in unified memory. RSS is the resident memory from ps. For Eos-0.8B, Sol-2B, and Nox-4B, the value outside the parentheses is at process exit, and the value inside the parentheses is the peak during measurement. The Decider values were measured on the server process from the previous article right after the evaluation finished.

The footprint was comparable between Eos-0.8B and Decider. On the 24GB Mac mini, Nox-4B's footprint reached 21GB at its peak.

Summary

Decision 2.0 comes in 6 sizes from 0.6B to 27B, and like Kev, you can choose the size.

On these 20 questions, Eos-0.8B and Sol-2B achieved decision accuracy comparable to Strands Decider 2B.
The differences showed up in speed and memory. Eos-0.8B was about 100ms slower than Decider, with comparable memory. Sol-2B responded even more slowly and used more memory.

Decision 2.0 is one of the decision-only models, including Jev covered in the previous article, that offers a wide range of size choices in an OSS implementation. I'd like to keep an eye on it going forward.