I tried to organize the state of local LLMs in 2026
This page has been translated by machine translation. View original
Introduction
Hello, I'm Moroshige from Classmethod's Manufacturing Business Technology Division.
From 2025 to 2026, the local LLM scene has been moving so fast that it's become hard to keep up. The impact of DeepSeek-R1, expanded Ollama features, improved Japanese language performance in Qwen2.5, and the news that Claude Code can now run on local LLMs. Feeling like "I really need to catch up properly," I wrote this article partly to organize my own thoughts.
This article is aimed at developers who regularly use coding AIs like Claude Code, GitHub Copilot, and Cursor, and organizes the local LLM options available as of January 2026. I'll address questions like "which model should I choose," "are the licenses okay," and "what kind of specs do I need."
Why API Users Are Moving to Local LLMs
Improvements in Open-Source LLM Performance
Entering 2026, the performance of open-source LLMs has been improving rapidly. Particularly in the coding domain, some models are now achieving benchmark scores comparable to commercial models. The perception that "open-source lags behind commercial models in accuracy" may gradually be becoming a thing of the past.
Differences in Cost Structure
The cost structure differs significantly between API billing and investing in your own GPU.
API billing is a pay-as-you-go model, which is inexpensive if usage is low. On the other hand, a self-hosted GPU environment requires a large initial investment, but the per-unit cost decreases as usage increases. Once monthly token processing volume exceeds a certain threshold, a self-hosted environment can have a lower total cost.
Which is better depends on usage patterns, but it's great to be in a situation where you have "options" rather than being limited to "API only."
Handling Data Confidentiality
When dealing with "data that can't go outside," local LLMs are a strong option. For data that can't be sent to the cloud for compliance reasons—such as medical data, financial data, and internal confidential information—you can process it with confidence in a local environment.
Comparing Major Models in 2026
Now for the main topic. I've organized the major models garnering attention as of January 2026 by use case.
Recommended Models by Use Case
| Use Case | Recommended Model | Reason for Selection |
|---|---|---|
| Coding completion | Qwen3 or Qwen2.5-Coder | Stable JSON output, Apache 2.0 license |
| Coding (mid-size) | Devstral Small 2 | SWE-bench 68%, 256K context |
| General chat | Llama 3.3 | 128k context support, wide range of sizes |
| Cost efficiency | Qwen3-30B-A3B | MoE structure with effectively 3B active, Apache 2.0 |
| Lightweight / Edge | gpt-oss-20b or Qwen3-1.7B | Runs on 16GB/8GB, reasoning model or lightweight general-purpose |
| Multimodal | Gemma 3-27B | Text, image, and video support, 140 languages |
| Math / Reasoning | Nemotron 3 Nano | AIME 89.1%, 1M context |
| Reasoning tasks | DeepSeek-V3.2 | GPT-5 level, integrated reasoning and agent capabilities |
Detailed Model Comparison
| Model | Strengths | Size | License | Features |
|---|---|---|---|---|
| Qwen3-14B | General purpose / Japanese | 14B | Apache 2.0 | Performance equivalent to Qwen2.5-32B |
| Qwen3-30B-A3B | Cost efficiency | 30B (3B active) | Apache 2.0 | Lightweight operation via MoE |
| Qwen2.5-Coder | Code generation / JSON | 0.5B–32B | Apache 2.0 | Supports 29 languages |
| Qwen3-Coder | Agent / Code generation | 480B (35B active) | Apache 2.0 | 256K context |
| Devstral Small 2 | Coding | 24B | Apache 2.0 | SWE-bench 68%, 256K context |
| gpt-oss-20b | Reasoning | 21B (3.6B active) | Apache 2.0 | OpenAI's first open-weight model |
| GLM-4.7-Flash | Reasoning / Fast generation | 30B (3B active) | MIT | 24GB recommended, stability challenges |
| Gemma 3-27B | Multimodal | 27B | Gemma proprietary | 140 languages, 128K context |
| Nemotron 3 Nano | Math / Reasoning | 31.6B (3.6B active) | NVIDIA proprietary | Hybrid Mamba-Transformer MoE, supports 20 languages |
| DeepSeek-V3.2 | Reasoning / Agent integration | 671B (37B active) | MIT | Integrated reasoning and agent, R1 successor |
| Kimi K2.5 | Agent / Multimodal | 1T (32B active) | Modified MIT | Agent Swarm, 256K context |
| Llama 3.3 | General chat | 1B–405B | Meta proprietary | 128k context |
How Qwen3 Changed Things
Released in April 2025, the Qwen3 series significantly changed the local LLM landscape. Compared to the previous generation Qwen2.5, performance has improved even with the same parameter count. Roughly speaking, Qwen3-1.7B delivers performance equivalent to Qwen2.5-3B, and Qwen3-14B delivers performance equivalent to Qwen2.5-32B. In other words, you can now get equivalent quality with less VRAM.
Qwen3 includes Dense models (0.6B–32B) and MoE models (30B-A3B, 235B-A22B). Notably, Qwen3-30B-A3B has a MoE configuration where only 3B out of a total of 30B parameters are active, and it can run on 16GB VRAM. It's a strong option if you prioritize cost efficiency. It supports 119 languages and has high Japanese language performance, firmly inheriting the Japanese language strengths of Qwen2.5.
Also, Qwen3-Coder, released in July 2025, uses a MoE configuration that activates only 35B out of 480B total parameters, and has recorded a score on SWE-Bench Verified comparable to Claude Sonnet 4. However, running it locally as an individual is currently not realistic (requires 290GB VRAM). We'll have to wait for future releases of smaller Coder-specific variants.
Personally, I think Qwen series is a good entry point if you're just getting started with local LLMs.
Why GLM-4.7-Flash Is Getting Buzz Overseas
On January 19, 2026, GLM-4.7-Flash from Tsinghua University was released and is attracting attention in overseas AI communities. With a MoE configuration of 30B total parameters (3B active), it has recorded a high SWE-bench score of 59.2% (approximately 2.7 times that of Qwen2.5-Coder).
The reasons it's getting attention are as follows:
- Fully open source (MIT license)
- Runs on 24GB+ VRAM (practical on an RTX 4090 or M3 Max)
- OpenAI/Claude API compatible, immediately usable with existing tools (Cursor, etc.)
- Extremely low API pricing (input $0.07/1M, output $0.40/1M)
Local execution is also easy, and it has been confirmed to work with llama.cpp and text-generation-webui. It looks set to become a strong option for personal local environments alongside Qwen2.5-Coder.
Notes for Japanese Language Environments
One thing that matters here is Japanese language performance. Many of the above models are primarily trained on English, and there are strengths and weaknesses when using them in Japanese.
| Model | Japanese Performance | Notes |
|---|---|---|
| Qwen3 series | ◎ | Supports 119 languages, inherits Qwen2.5's strengths |
| Qwen2.5 series | ◎ | Industry-leading class, with scores exceeding GPT-4 |
| Qwen3-Coder | ◎ | Qwen3-based with 119-language support, good Japanese performance |
| Gemma 3 | ○ | Supports 140 languages, Japanese fine-tuned versions available |
| Nemotron 3 | ◎ | Supports 20 languages, trained on 682.8B Japanese tokens. JCommonsenseQA 92.5% (1035/1119) |
| DeepSeek-V3.2 | ○ | Official Japanese support since V3 |
| Kimi K2.5 | ○ | Multilingual support, no publicly available Japanese-specific benchmarks |
| gpt-oss | ○ | Multilingual support, reasonably usable in Japanese |
| GLM-4.7-Flash | ◎ | High Japanese performance, reportedly better than DeepSeek |
| Devstral Small 2 | △ | English and code-focused, limited Japanese support |
| Llama 3.3 | △ | English-centered, may respond in English even when asked in Japanese |
For Japanese use, it's practical to focus on the Qwen3 series. I listed Llama 3.3 for "general chat" use, but in a Japanese environment, Qwen3-8B or Qwen3-14B will likely give better results. (My Japanese performance assessments are quite subjective...)
Nemotron 3 Nano supports 20 languages and has abundant Japanese training data at 682.8B tokens. When I actually ran JCommonsenseQA (1,119 questions, 3-shot) on a DGX Spark, the result was 92.5% (1035/1119). This is comparable to Gemma 3 12B (91.8%) and gpt-oss:20b (92.7%), and it's impressive to achieve this accuracy with only 3.6B active parameters. NVIDIA has also published a synthetic persona dataset called Nemotron-Personas-Japan (1 million records, CC BY 4.0) for Japan, and its use for fine-tuning Japanese LLMs is progressing.
If you want to use the Llama series, Japanese fine-tuned versions like ELYZA-Llama-3-JP-8B and Swallow are also options. They are published on Hugging Face and can also be used with Ollama.
License Points to Watch Out For
When using open-source LLMs commercially, it's worth paying attention to licenses. Even if something is called "free," the conditions differ, so let's get organized.
License Classification
| License | Representative Models | Commercial Use | Modification / Redistribution |
|---|---|---|---|
| Apache 2.0 | Qwen, Mistral, gpt-oss | Completely free | Unrestricted |
| Meta proprietary | Llama series | Basically OK (with conditions) | With conditions |
| Other free | DeepSeek, GLM, etc. | Basically free (check terms recommended) | Depends on terms |
If you're unsure, sticking with Apache 2.0 (Qwen/Mistral series) is a safe bet. Llama is usually fine in practice, but if there's a possibility of scaling up significantly in the future, Apache 2.0 licensed models are more reassuring from the start.
Hardware Requirements and Selection Guidelines
"Which models can I run on my own setup?" is something many people want to know. I've organized a rough guide for which models you can run based on VRAM capacity.
Model Selection Guide by VRAM
| VRAM | Runnable Models | Quantization | Use Case |
|---|---|---|---|
| 8GB | Qwen3-1.7B, Qwen 7B | 4bit | Light use, experimentation |
| 16GB | gpt-oss-20b, Qwen3-14B, Nemotron 3 Nano, Llama 8B | 8bit | Personal development, code completion |
| 24GB | Devstral Small 2, Gemma 3-27B, GLM-4.7-Flash | 4bit | Coding, reasoning |
| 24GB | Llama 70B | 4bit | Serious use, RAG |
| 48GB+ | Llama 405B | 8bit | Large-scale production |
| 150GB+ | Qwen3-Coder-480B (quantized) | 4bit | Enterprise |
About Quantization
"Quantization" is a technique that reduces required memory at the cost of some model precision. With 4-bit quantization, you can sometimes run a model that originally required 48GB on just 24GB. The impact on quality depends on the use case, but for coding assistance, 4-bit is sufficiently practical.
Tool Selection Guide
There are several tools available for running local LLMs. Here are my recommendations as of 2026.
Comparison of Major Tools
| Tool | Recommendation | Features | CPU MoE Support | Best For |
|---|---|---|---|---|
| Ollama | ★★★★★ | Works with one command, API compatible | ✗ | People who want to quickly try it out via CLI |
| vLLM | ★★★★☆ | High throughput, production-optimized | △ (supplementary) | Production environments requiring large numbers of simultaneous requests |
| LM Studio | ★★★★☆ | Visually operable with GUI | ◎ | Non-engineers, people who prefer a GUI |
| llama.cpp | ★★★☆☆ | Lightweight with high customizability | ◎ | Advanced users, edge devices |
About MoE Models and CPU MoE Offloading
MoE models (Qwen3-30B-A3B, gpt-oss-120b, etc.) activate only some experts when processing each token. Using llama.cpp's --n-cpu-moe option, you can place expert weights in CPU RAM while keeping the attention layers in GPU VRAM, allowing large MoE models to run at practical speeds even with limited VRAM. LM Studio has also supported this feature via GUI since v0.3.23.
Community reports have shown cases where gpt-oss-120b ran at approximately 3.5 times the speed of Ollama (GitHub Issue #11772). If you're planning to use MoE models seriously, consider trying llama.cpp or LM Studio.
Note that in unified memory environments like Apple Silicon Macs or NVIDIA DGX Spark, where the CPU and GPU share the same memory pool, the memory savings from CPU MoE Offloading don't apply. On the other hand, since there's no PCIe transfer bottleneck when loading experts, unified memory is structurally well-suited to MoE models.
Why I Recommend Ollama
If you're trying local LLMs for the first time, Ollama is the easiest option. If you need to handle large numbers of simultaneous requests in a serious production environment, vLLM becomes an option (setup isn't as simple as Ollama, but throughput under heavy load can be several times to nearly 20 times higher than Ollama).
# Installation (Mac/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Get and run a model
ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b
Running ollama run automatically downloads the model as well. Japanese responses are also smooth.
That's all it takes to run a local LLM. Installation via Homebrew is also supported.
# Install with Homebrew
brew install ollama
Ollama also provides an OpenAI-compatible API endpoint, making migration from existing code easy. The approach of "start with Ollama to get a feel for it" is recommended.
For those who want to manage models and chat via a web UI, LM Studio is also a great option. You can search, download, and run models entirely through a GUI, and it includes a local server with OpenAI compatibility. If you prefer visual operation over CLI, give it a try.
About API Compatibility
In addition to the OpenAI API-compatible endpoint (/v1/chat/completions), Ollama also provides an Anthropic API-compatible endpoint (/v1/messages). Since v0.15, the ollama launch claude command has also been added, allowing you to try integration with Claude Code.
However, there are still some challenges with Claude Code integration at this time. For details, see the related article "Trying to Run Claude Code Locally with Ollama v0.15" where I've summarized my verification results.
Summary
Finally, here are concrete steps for getting started with local LLMs.
Step 1: Choose a Model
| Use Case | Recommended Model |
|---|---|
| Coding (24GB+) | Devstral Small 2 or Qwen2.5-Coder |
| Coding (16GB) | Qwen3-14B or gpt-oss-20b |
| General chat (Japanese) | Qwen3-14B |
| General chat (English) | Llama 3.3 |
| Lightweight / Trial | Qwen3-1.7B or gpt-oss-20b |
Step 2: Check Your Hardware
If you have less than 16GB of VRAM, choose a 4-bit or 8-bit quantized model. MacBook unified memory works too, but inference speed will be slower than dedicated GPU.
Step 3: Get Started with Ollama
# Installation
curl -fsSL https://ollama.com/install.sh | sh
# To try general chat (Japanese)
ollama pull qwen3:8b
ollama run qwen3:8b
# To try coding completion
ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b
This year feels like the moment when open-source LLMs are transitioning from "hobby experiments" to "practical tools." If you have API experience, moving to local LLMs is easier than you might imagine.
In a Japanese language environment, the Qwen3 series is currently the top recommendation. Qwen3-14B delivers performance equivalent to Qwen2.5-32B at less than half the VRAM, making it excellent in terms of cost performance. Why not start by running Qwen3 on Ollama to get a feel for it? You may find yourself experiencing the unique benefits of local LLMs, such as cost reduction and ensuring data confidentiality.
Reference Links
- Ollama Official Site
- LM Studio Official Site
- vLLM Official Documentation
- Hugging Face Open LLM Leaderboard
- Qwen3 Official Blog
- Qwen2.5-Coder Model Card
- Qwen3-Coder Official Blog
- Qwen3-Coder GitHub
- gpt-oss Hugging Face
- Devstral Small 2 Official Blog
- GLM-4.7-Flash Official (Z.AI)
- GLM-4.7-Flash Hugging Face
- Llama 3.3 Official Page
- DeepSeek-V3.2 Official Blog
- Kimi K2.5 Hugging Face
- Gemma 3 Official Blog
- Gemma 3 Hugging Face
- Nemotron 3 Nano Official Blog
- Nemotron-Personas-Japan (HuggingFace)
- ELYZA Official Site
