Open-weight model team AI environment that covers everything from development to business use
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.
When a team starts using AI coding agents, two concerns quickly emerge: costs and data handling. Relying entirely on powerful models makes billing unpredictable, and leaving settings unconfigured means nobody can explain where internal information mixed into prompts is being sent.
I'm validating a team AI environment in our internal development team that addresses both of these issues through a combination of "open-weight models + automatic routing + leak-proof configuration." The router portion is also published as a Docker bundle.
The core of the architecture is connecting opencode to Fireworks AI via NVIDIA's LLM routing infrastructure NeMo Switchyard, which determines for each request whether a lighter model will suffice, automatically switching between a strong model and a light model. To state the conclusion upfront: in actual measurements across 52 runs of a synthetic coding benchmark, automatic routing completed tasks at $0.0009/run compared to $0.035/run with strong-fixed, without any drop in task completion rate (including classification costs). Since the benchmark is at a difficulty level where the light model alone can achieve a perfect score, this figure demonstrates the effect of "not routing lightweight tasks to expensive models" — but the actual savings vary depending on the content of your traffic, and I've written honestly about that in the main text.
The mechanics of Switchyard itself and its setup procedure are introduced in the following article.
This article introduces the overall picture of the team environment built on top of that routing. It's a record of design decisions — what goes where, why Fireworks, how we narrowed down to two models, settings for preventing data leaks, and the current state of observability — so I hope it resonates with people responsible for distributing AI agents to their teams.
The Full Team Environment on One Diagram
First, the overall picture. The current operational scope places a router on each developer's local machine and consolidates LLMs in Fireworks serverless, while placing team-shared RAG and an observability stack on the DGX Spark — the latter still being under validation.
The left half of the diagram — the route from opencode through Switchyard to Fireworks — is the main subject of this article. Switchyard runs as a Docker container resident on each person's local machine, and for each request it uses a classifier to estimate the "probability that a light model can complete the task," currently switching automatically between a strong model (Kimi K3) and a light model (DeepSeek V4 Flash-0731). The router itself is Switchyard's standalone Rust server (switchyard-server), and its configuration is a single routes.toml file.
Claude Code's LLM traffic goes directly to Anthropic and does not pass through Switchyard. Routing targets only the opencode side that uses open-weight models, and the structure shares only the observability layer (NeMo Relay) with Claude Code.
The service layer and inference layer on the right half are the location for the team's shared knowledge base (RAG) and local LLMs. Of these, the RAG is published in a separate article, so this article only provides an introduction at the entry point.
Placement Is Determined by State and Keys
The first decision when building this architecture was whether to "place each component on a shared server or on each person's local machine." Two criteria guided the decision: whether it holds state that needs to be shared, and who holds the API keys.
| Component | State to Share | API Key | Placement |
|---|---|---|---|
| Switchyard router | None (logs in each person's local volume) | Fireworks (individually issued) | Each person's local (127.0.0.1) |
| Web search | None | Search API (individually issued) | Each person's local (skill) |
| RAG (including vector DB) | Corpus and index | Consolidated on server side | Dedicated machine (DGX Spark) |
| Observability (Tempo / Grafana) | Traces | Not required (receiving end) | Dedicated machine (DGX Spark) |
The decision not to put the router on a shared server was the first fork in the road. Since Fireworks API keys are issued per individual, consolidating them in a shared proxy would break the alignment between key ownership and billing ownership. The router itself is stateless, and having a single Docker container reside on a laptop creates no burden. So it makes more sense to place it on each person's local machine, bound to 127.0.0.1 — cleaner for both key management and attack surface. Distribution is handled through a git repository with docker compose up -d, and updates only require git pull and a restart. Only updates that change the router implementation itself require an image rebuild, which takes 10–20 minutes for the initial Rust compilation (subsequent builds take tens of seconds with caching).
Conversely, data like the RAG's vector DB and observability traces are only meaningful when the entire team shares a single state. Since keys can also be consolidated on the server side, these are placed on the dedicated machine.
Along the way, I also tried a "consolidate web search on a single team-shared server" approach, but ultimately reverted to placing it on each person's local machine. Search API keys can be issued per individual with minimal friction, and if you can eliminate a server, it's better to do so for lighter operations. The distinction I formalized then — "if there's a server you can eliminate, distribute it as a skill; if the server is essential and must remain, expose it via MCP" — is directly reflected in the difference between web search (skill) and RAG (MCP) distribution methods described in later sections.
Why Run Open-Weight Models on Fireworks
There are three reasons I chose Fireworks AI serverless as the LLM provider.
The first is freshness. New versions of open-weight models are available on serverless on their release day. In fact, the official DeepSeek V4 Flash 0731 version was listed on its July 31, 2026 release day, and we were able to swap out the team's weak tier within that same week. With managed cloud services, open-weight model listings can often lag by a generation, so freshness matters for use cases that prioritize keeping up with the latest models.
The second is that data handling can be verified at the contract level. Fireworks' DPA (Data Processing Addendum) section 4.3(f) contractually prohibits using Covered Data, including prompts and inputs/outputs, for model training and improvement. Zero Data Retention is also enabled by default, meaning prompts and generated results only exist in volatile memory during request processing. Their Trust Center also shows certifications including SOC 2 Type II, ISO 27001:2022, and the AI management standard ISO/IEC 42001:2023. For team use, I prioritized being able to confirm in a contract document — not just a line on a policy page — that "inputs won't be used for training."
The third is pay-as-you-go billing. With no GPUs or containers to maintain and payment only for tokens used, the savings from the routing described later are directly reflected in the bill.
One caveat: region selection requires attention. Serverless offers no region selection and no contractual guarantee about where processing occurs (infrastructure is primarily US-based). If you have requirements for domestic processing or latency, you'll need to use on-demand dedicated deployment and select AP_TOKYO_1 / AP_TOKYO_2.
| Use Case | Offering | Region | Billing |
|---|---|---|---|
| Everyday development (scope of this environment) | Serverless | Not selectable (primarily US) | Token-based |
| Domestic processing / latency requirements | On-demand (dedicated) | AP_TOKYO_1/2 selectable | GPU-hour billing |
Since our use cases center on public information and code, we've organized this as "serverless for everyday development, on-demand Tokyo when requirements arise." Note that information equivalent to PCI or PHI is contractually prohibited from being input, so this boundary is established as an operational rule before any configuration.
Narrowing Down to Two Open-Weight Models
The models used by the team were narrowed down to two: Kimi K3 for strong, and DeepSeek V4 Flash-0731 for weak and classifier. The Fireworks serverless Standard tier pricing (per 1 million tokens, as of August 2026) and roles are as follows.
| Role | Model | Input | Cached Input | Output |
|---|---|---|---|---|
| Strong | Kimi K3 | $3.00 | $0.30 | $15.00 |
| Weak + classifier | DeepSeek V4 Flash-0731 | $0.14 | $0.028 | $0.28 |
| (Reference) Pre-switch strong | DeepSeek V4 Pro | $1.74 | $0.145 | $3.48 |
The unit price difference between weak and strong is 21x for input and 54x for output. This gap is exactly what funds the automatic routing — the bill decreases proportionally to how many lightweight tasks can be routed to weak.
Flash-0731 supports a 1M context and function calling, and streaming tool_call delta — the lifeline of agentic loops — has been validated. According to vendor-published figures, agentic benchmarks improved significantly from the preview version (e.g., Terminal Bench went from 61.8 to 82.7), making it a reliable choice to handle both the weak tier and classifier roles with a single model.
Additionally, I've confirmed that this model can also run locally on the DGX Spark. I've validated both standalone operation with llama.cpp and 2-node vLLM parallel inference in past articles.
The fact that the same open-weight model can run on both cloud serverless and a local DGX Spark means that if governance requirements or cost structures change in the future, you can escape to local inference simply by swapping Switchyard's connection target. The value of committing to open-weight models lies largely in the existence of this escape route.
I'll also explain why we narrowed down to two models. Adding more candidates multiplies the explanation costs for "which model to use when," the validation combinations, and the tracking of price changes. Given that Switchyard's routing follows a two-tier strong/weak structure, the most operationally manageable approach is to fix the team standard at two models and handle exceptions through per-route opt-in (like the auto-esc described later).
Switching Strong to Kimi K3
The strong model at launch was DeepSeek V4 Pro. It scored perfectly on coding benchmarks with no quality complaints, but for deep design discussions and planning, Kimi K3's responses felt noticeably better. I first added it as an opt-in fixed route called k3-only, used it extensively myself, and then switched the entire strong tier to K3.
The switch itself is a single line in the configuration file routes.toml. Just edit and restart — no image rebuild required. The classifier and weak continue to run with Flash-0731.
[targets.strong]
id = "accounts/fireworks/models/kimi-k3" # ← changed from deepseek-v4-pro
However, K3 is not a free upgrade — it's clearly a trade-off. In a pre-switch estimate applying K3's unit pricing to the actual tokens from already-run benchmarks, the cost for strong-fixed was expected to be approximately 3.0x higher. Since routing works against the ladder of unit prices, the more expensive the strong model becomes, the greater the value of "tasks routed to weak." On the other hand, through extensive pre-switch use, I observed that for brainstorming and strategic questions, thinking tokens tend to run long, making actual costs balloon beyond the unit price difference alone (7.1x compared to Pro), with latency also approximately 2.5x higher. Given these characteristics, the switch was carried out alongside re-measurement of the benchmarks.
Post-switch measurements consisted of 52 total runs across 4 arms: code_fix ×10 + tool_calls ×3. In addition to the standard auto arm, the opt-in auto-esc mode was included. This mode starts all requests with weak, and only escalates to strong for the entire session when it detects a "stuck trajectory" such as repeatedly hitting the same error. Before escalation, responses are returned as a complete batch rather than streamed, making it suited for non-interactive workloads like cron jobs.
| Arm (code_fix, n=10) | Quality (pytest score, 5-point scale) | Cost / run | vs. strong-fixed | Wall-clock median |
|---|---|---|---|---|
| Strong-fixed (Kimi K3) | Perfect | $0.0353 | — | 25.5 sec |
| Weak-fixed (Flash-0731) | Perfect | $0.00066 | −98.1% | 8.7 sec |
| Auto (including classification cost) | Perfect | $0.00088 | −97.5% | 9.1 sec |
| Auto-esc (including classification cost) | Perfect | $0.0021 | −94.1% | 20.9 sec |
All 52 runs completed, with perfect scores across all arms. For tool_calls (n=3), auto also landed approximately 90% cheaper. In the code_fix auto arm, all 10 runs fell to weak, with zero Kimi K3 calls. Two notes to add: this benchmark is at a difficulty level where weak alone can achieve a perfect score, so this table is proof of "not routing lightweight tasks to expensive models." Also, repeated classification calls for identical content were skipped via deduplication, meaning the classification cost in real-world operation will run higher than shown here.
These numbers didn't appear from the start. Early measurements (with strong being V4 Pro) showed approximately 27% reduction, and immediately after switching to K3, the routing pin shifted entirely to strong, nearly eliminating any reduction. The cause was not K3 itself, but a change in the classification model's interpretation that occurred around the same time. Afterward, Switchyard revamped its classification approach from "task complexity classification" to "estimating the probability that weak can complete the task," and the numbers in the table above resulted from the same benchmark with the same models. This "silent change in classification" is planned for a deep-dive in a separate article, alongside experiments across multiple classification models.
Team rollout follows the same process as before: a PR to the distribution repository with an announcement. Route IDs (auto / strong-only / weak-only, etc.) are preserved across configuration updates, so team members' opencode settings require no changes. A strong-only fixed fallback route and revert instructions are also included for cases where the feel doesn't match expectations.
Routing Escapes for Areas That Can't Be Left to Automatic Routing
Automatic routing is not a silver bullet. Through actual operation, I've identified specific areas where leaving everything to the classifier results in misses.
The current classifier, by default, reads only the first and most recent user messages to estimate "the probability that weak can complete this task," then compares against a threshold to determine routing. For tasks involving writing or fixing code, the threshold is 0.75. For tasks outside of code — like brainstorming or strategic discussion — if they don't match the classification rules, a stricter threshold is automatically applied. When in doubt, the design falls back to strong.
When I measured 13 deep discussion questions through the same classification path as production, the default threshold of 0.5 only routed 9 questions to strong. Calibrating to 0.75 improved this to 11–12 questions (with 1 borderline question varying between runs). Still, 1–2 questions fall through to weak. "Can catch about 90%+ but not all" is the current state of automatic classification.
The range of context shown to the classifier can be changed in settings. Adding recent_turn_window to a route includes the last N conversation turns (including assistant responses and tool results) as classification input, in addition to the initial task. Across measurements with 40 real-world code work sessions, varying the window between off / 2 / 4 / 8 showed almost no change in routing decisions — the misses described above were not fixed by this. What it did affect was different: classification time dropped from a median of 17.5 seconds to around 3 seconds, and the classifier's thinking tokens fell from 1,713 to 227. The long deliberation from insufficient context was eliminated. Since swapping the arm order produced the same results, recent_turn_window = 4 has been incorporated into the distributed configuration.
For areas where misses are unacceptable, I use fixed routing outside of the automatic system. This is done by assigning models per mode or agent in opencode's configuration.
{
// Default is automatic routing
"model": "switchyard/auto",
// Auxiliary calls like title generation use weak-fixed
"small_model": "switchyard/weak-only",
"agent": {
// Deep planning goes directly to strong (Kimi K3). Don't let classifier decide.
"plan": { "model": "switchyard/strong-only" },
// Sub-agents inherit the calling model's setting, so explicitly cut this off
"explore": { "model": "switchyard/weak-only" },
"scout": { "model": "switchyard/weak-only" }
}
}
This agent block is enabled by default in the distributed bundle's configuration example, so team members get the same boundaries just by pasting the configuration.
To elaborate on the intent behind each: the plan mode is set to strong direct-connect because brainstorming sessions are infrequent and quality-dominant — this isn't a scenario where saving a few cents through routing makes sense. The explicit weak assignment for explore/scout is defensive in the opposite direction: opencode sub-agents inherit the calling model, so without explicit assignment, even simple tasks like reading grep results would incur the $15/1M output unit price. The small_model weak-fixed assignment came from actual measurements — there were records of auxiliary calls like title generation being routed to strong via auto, and fixing this eliminated the leakage to high-cost tiers.
Once you can see the boundary between "what to leave to automatic routing" and "what to escape with fixed routing," using the routing system becomes much more comfortable.
Preventing Data Leaks with strict-privacy and Web Search
Even if you consolidate LLM traffic to a single Fireworks endpoint, opencode still has external transmission paths other than LLMs. Before distributing to the team, I addressed each of these one by one. The configuration base is the recommended setup from opencode-with-strict-privacy, maintained internally, and the bundle includes a merged, complete opencode.jsonc.example.
The first issue is the built-in websearch. opencode's standard web search connects to the search provider Exa's hosted MCP via an anonymous connection with no API key, meaning it falls outside corporate contract (data processing agreement) exclusions and is subject to general privacy policy. Exa's policy explicitly states that search queries are used for "model training and fine-tuning," with no opt-out mechanism. Since search queries during coding can include error messages and internal context, this is disabled with tools.websearch: false and permission.websearch: "deny". I also disabled conversation share link generation and auto-update.
Next is what I consider the most important pitfall in this section. If opencode's Global ~/.config/opencode/AGENTS.md doesn't exist, it falls back for compatibility by loading ~/.claude/CLAUDE.md as the global rules (official specification). If you're using Claude Code alongside opencode, personal settings and notes written for Claude will be injected into all requests to opencode's connected models. In my own environment, this fallback was indeed triggered, and my entire Claude Code personal configuration was hitching a ride on requests to DeepSeek. No matter how carefully you lock down configuration files, this won't be closed unless you place an AGENTS.md. The fix is simply to place a minimal Global AGENTS.md.
mkdir -p ~/.config/opencode
cat > ~/.config/opencode/AGENTS.md <<'EOF'
# Global rules
- Do not include customer names, internal project names, or unpublished code names in web search queries
EOF
Another pitfall is configuration scope. Always place privacy settings in Global scope (~/.config/opencode/opencode.json). Project-level opencode.json overrides Global, so project-specific configurations can inadvertently nullify your privacy settings. Having both .json and .jsonc files is also prohibited since only one will be read. Furthermore, some disable flags live on the environment variable side rather than in config files.
export OPENCODE_ENABLE_EXA=0 # Disable Exa search
export OPENCODE_EXPERIMENTAL=0 # Disable experimental features in bulk
export OPENCODE_EXPERIMENTAL_EXA=0 # Legacy Exa flag
export OPENCODE_AUTO_SHARE=0 # Disable auto-sharing
Environment variables are a separate system from config files and easy to miss — in fact, on my main machine, there was a period where only the config file side was applied while the rc side was unconfigured. In the team distribution, I've added env | grep OPENCODE_ to the onboarding checklist so it can be verified mechanically.
As a replacement for disabled websearch, a search API that contractually guarantees no use for training is distributed as a skill, called with each person's own key. The backend is a choice of Gemini (Google AI Studio, with free tier) or OpenAI (Responses API web_search), selected not on search provider features but on "whether there's a contract stating queries won't be used for training." There's an important distinction: for Gemini, a key issued from a GCP project with billing enabled is required (free tier keys are subject to training data use), while OpenAI defaults to no training use via the API. This is the one part I specifically ask team members not to skip in the onboarding instructions.
Team-Shared RAG as the Entry Point for Business Use
Up to this point, I've focused on the developer's entry point — but the value of a team AI environment isn't limited to coding. For business use cases like searching across documents, summarizing, and answering with citations, I've built a team-shared RAG based on NVIDIA RAG Blueprint on the DGX Spark.
I'll defer the details to the article above (published 2026-08-07), but what I want to highlight in the context of this article is the connection method. The RAG is exposed as an MCP server, allowing developers to pull team knowledge directly from opencode or Claude Code. For non-engineers, the entry point is a web UI. Since I've verified that the search and generation pipeline can run locally on DGX Spark, the "data doesn't leave" guarantee can be maintained even when internal documents are ingested.
The full picture of this environment is: opencode + routing as the development entry point, and RAG + MCP as the knowledge entry point.
Two Systems for Observability
A team environment isn't "done" once distributed — improvement cycles only work when you can observe how it's actually being used and how much it costs. There are currently two observability systems running.
The first is NVIDIA's NeMo Relay. It records agent execution in Claude Code and Codex CLI (model calls, tool execution, sub-agent branching) and sends it in OpenTelemetry format from OTel Collector to Tempo / Grafana. Setup and caveats are introduced in the following article (published 2026-08-06).
One thing to mention about Relay up front: by default, prompt text reaches the observability backend. For team deployment, I'm using the first-party pii-redaction plugin to drop the text content, keeping only the metadata needed for routing and cost analysis.
The second is Switchyard's own routing logs. On each person's local machine, request-level classification decisions, model selections, and token counts are recorded to routing.jsonl (stored in a Docker named volume, so it survives config changes and key rotations). A weekly snapshot script collects these and aggregates the team's data. Per-route aggregation can distinguish not only the tier distribution for auto routes, but also fixed-route usage as separate entries like pinned:kimi-k3, making it possible to track "how much was used by automatic vs. fixed routing." In the current server, classification calls are also recorded as independent entries with tier="classifier", so classification costs (approximately 8% of total in actual measurements) that could only be estimated in the old architecture are now directly included in weekly aggregations. A Prometheus-format /metrics endpoint is also available, providing health metrics like fail-open classification counts. The cost reconciliation target is the Fireworks dashboard's per-model usage data. Since strong and weak are separate models, per-model usage directly maps to tier distribution and cost breakdown.
The Honest Current State of Observability Coverage
The observability goal is to "consolidate everything under Relay," but we haven't reached that point yet. Here's an honest breakdown of coverage per route:
| Route | NeMo Relay | Switchyard stats (routing.jsonl) | Fireworks Dashboard |
|---|---|---|---|
| Claude Code | ✅ | ❌ (goes directly to Anthropic) | ❌ |
| Codex CLI | ✅ | ❌ | ❌ |
| opencode | ❌ | ✅ | ✅ |
The reason opencode — the most important one — is not on Relay is the support method on Relay's side. The passive plugin approach for opencode was abandoned in June 2026, and NVIDIA has announced a wrapped execution approach for opencode support (NeMo Relay PR #73 close comment, 2026-06-03). Relay itself was officially released as the 0.7 series in August 2026. The integration with Switchyard is mid-transition: the experimental integration examples in the Relay repository (where Relay calls Switchyard's classification API) were removed in 0.8 and replaced by a native plugin on the Switchyard side — this is explicitly documented (NeMo Relay Epic #401, Switchyard PR #270). The plan is to verify and deploy to the team as soon as that support lands, completing the consolidation.
Also worth noting: Switchyard itself underwent a major redesign to a standalone Rust server implementation in August 2026 and is still marked as pre-alpha ("Not for production use") in terms of maturity. Since it's not yet published to any package registry, the bundle pins to a specific commit SHA for tracking. This is a configuration that requires willingness to keep pace with upstream changes — an important caveat for anyone planning to try this.
In the meantime, opencode is covered through routing.jsonl and the Fireworks dashboard as alternative observability. While the trace granularity is inferior to Relay, for actual cost figures this path is actually more accurate. routing.jsonl is primary data recorded by the router on a per-request basis, and Fireworks' per-model usage data is the billing itself. Even with uneven-looking observability, the ability to answer "how much did it cost?" hasn't degraded — that's my assessment of the current state.
Summary
I introduced a team AI environment that connects opencode to Fireworks via NeMo Switchyard, automatically routing between two open-weight models — Kimi K3 and DeepSeek V4 Flash-0731. In synthetic benchmarks, automatic routing completed tasks at $0.0009/run compared to $0.035/run with strong-fixed, without any drop in task completion rate. On the other hand, there was a period where savings nearly disappeared due to a classification model update. The effectiveness of reduction is not a fixed spec of the environment — it's a "variable to keep observing" that fluctuates with configuration and traffic. That's the real lesson from running this in production. Rather than relying solely on routing, I escape areas the classifier can't handle reliably by using per-mode fixed routing, close data exit points with strict-privacy settings and AGENTS.md placement, and observe actual costs through routing.jsonl and the Fireworks dashboard. Packaging all of this into a git repository for distribution is the key point of this as a team environment.
I'll also state the honest limitations. The synthetic benchmark used for quality evaluation is at a difficulty level where weak alone scores perfectly, so it remains a weak demonstration of "routing protecting quality." Classification calls add waiting time to affected turns, and actual savings vary significantly depending on traffic content. Observability also remains split between two systems while waiting for opencode's Relay support. I plan to update these areas as real traffic accumulates and upstream development progresses.
The team-shared RAG construction article is already published, so reading it alongside this one should connect the full picture. I also plan to cover the details of Switchyard's new Rust server implementation in a separate article aligned with upstream releases. Next, I'd like to try mounting Hermes Agent as a team assistant and building an entry point for pulling knowledge from Slack using natural language.
Reference Links
- himorishige/switchyard-opencode-bundle — Router distribution bundle from this article (Apache-2.0)
- NVIDIA-NeMo/Switchyard
- NVIDIA/NeMo-Relay
- cm-dyoshikawa/opencode-with-strict-privacy — Recommended privacy configuration for opencode
- Fireworks AI — Serverless Pricing
- Fireworks AI — Data Handling (Zero Data Retention)
- Fireworks AI — DPA (Section 4.3(f) is the contractual prohibition on training use)
- Kimi K3 on Fireworks
- DeepSeek-V4-Flash-0731 (Hugging Face)
- Trying Out NVIDIA's New LLM Routing Infrastructure NeMo Switchyard (Published 2026-07-03, covers the initial Python-based router. Has addendum.)
- Building a Team-Shared RAG with NVIDIA RAG Blueprint × DGX Spark and Connecting via MCP (Published 2026-08-07)
- NeMo Relay first-touch (Published 2026-08-06)
- Running DeepSeek V4 Flash-0731 on a Single DGX Spark with llama.cpp (Published 2026-08-02)
- Running DeepSeek V4 Flash-DSpark on Two DGX Spark Nodes
