Open-weight model team AI environment covering everything from development to business use

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

When a team starts using AI coding agents, two concerns quickly surface: costs and data handling. Delegating everything to a powerful model makes billing hard to predict, and leaving settings unchecked means nobody can explain where internal information mixed into prompts is being sent.

I'm verifying the operation of a team AI environment that solves both of these issues through a combination of "open-weight models + automatic routing + leak-proof configuration" with an in-house development team. The router portion is also published as a Docker bundle.

https://github.com/himorishige/switchyard-opencode-bundle

The core of the architecture is connecting opencode to Fireworks AI via NVIDIA's LLM routing infrastructure NeMo Switchyard, and automatically switching between a strong model and a light model by determining "whether a lighter model is sufficient" for each request. To state the conclusion upfront, in actual measurements across 52 runs of a synthetic coding benchmark, automatic routing completed tasks at $0.0009/run compared to $0.035/run with strong-only, without dropping task completion rate (including classification costs). Since the benchmark is at a difficulty level where the light model alone can achieve a perfect score, these numbers demonstrate the effect of "not routing lightweight tasks to expensive models" — though I've written honestly in the body about how the actual savings amount varies depending on what's in the traffic.

The mechanism behind Switchyard itself and its installation procedure are introduced in the following article.

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-v020-rust-first-touch/

This article introduces the overall picture of the team environment built on top of that routing. It's a record of design decisions — where to place what, why Fireworks, how we narrowed down to two models, settings to prevent data from leaking outside, and the current state of observability — so I hope it resonates with people who are in the position of deploying AI agents to their team.

Summarizing the Team Environment in One Diagram

First, the overall picture. The current operating scope has the router on each developer's local machine with LLMs hosted on Fireworks serverless, while placing the team-shared RAG and observability stack on the DGX Spark is still in the verification stage.

The left half of the diagram — the path from opencode through Switchyard to Fireworks — is the main subject of this article. Switchyard runs as a Docker container on each person's local machine, and for each request it estimates "the probability that the lighter model can complete the task" using a classifier, currently automatically switching between the strong model (Kimi K3) and the light model (DeepSeek V4 Flash-0731). The router itself is Switchyard's standalone Rust server (switchyard-server), and configuration is a single routes.toml file.

Claude Code's LLM traffic goes directly to Anthropic and does not pass through Switchyard. The routing target is only the opencode side using open-weight models, and the structure is that Claude Code only shares the observability layer (NeMo Relay).

The right half — the service layer and inference layer — is the location for the team's shared knowledge base (RAG) and local LLM. The RAG portion has been published as a separate article, so this article only provides an introduction to it.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-rag-blueprint-mcp/

Placement is Determined by State and Keys

When building this architecture, the first thing we decided was "whether to place each component on a shared server or on each person's local machine." The decision criteria were two: whether it holds state that needs to be shared, and who holds the API keys.

The decision not to put the router on a shared server was the first fork in the road. Since Fireworks API keys are issued per individual, consolidating them into a shared proxy would break the attribution of keys and billing. The router itself is stateless, and a single Docker container running on a laptop is not a burden. In that case, placing it on each person's local machine and binding it to 127.0.0.1 is cleaner for key management and attack surface. Distribution was set up on a git repository with docker compose up -d, and updates only require git pull and restart. Only updates that change the router implementation itself require image rebuilding, with Rust compilation taking 10–20 minutes the first time (subsequent builds take tens of seconds with caching).

Conversely, the vector DB for RAG and traces for observability are data that only have meaning when the team shares a single state. These can also have keys consolidated on the server side, so they are placed on the dedicated machine.

At one point we also tried a "consolidate on a team-shared server with a single contract" approach for web search, but ultimately brought it back to each person's local. Search API keys have low friction even with individual issuance, and if a server can be eliminated, it's better to do so for lighter operations. The distinction we clarified at that time — "if a server can be eliminated, distribute it as a skill; if a server inherently remains, publish it via MCP" — is reflected directly in the difference between web search (skill) and RAG (MCP) distribution methods that appear in later sections.

Why Run Open Weights on Fireworks

There are three reasons we chose Fireworks AI's serverless as our LLM provider.

The first is freshness. New versions of open-weight models are available on serverless from their release day. In fact, the official version 0731 of DeepSeek V4 Flash was listed on the day of its July 31, 2026 release, and we were able to swap out the team's weak tier within that week. With some managed cloud services, open-weight model listings can lag by a generation, so this matters for freshness-sensitive use cases.

The second is that data handling can be verified at the contract level. Fireworks' DPA (Data Processing Addendum) Section 4.3(f) contractually prohibits using Covered Data, including prompts and outputs, for model training and improvement. Zero Data Retention is also enabled by default, with prompts and generated results only existing in volatile memory during request processing. The Trust Center also shows certifications including SOC 2 Type II, ISO 27001:2022, and AI management standard ISO/IEC 42001:2023. Since the whole team would be using this, we prioritized being able to confirm that "inputs won't be used for training" through contractual documents rather than a single line on a policy page.

The third is pay-as-you-go pricing. Since we pay only for the tokens used without holding any GPU or containers, reductions from routing described later are directly reflected in billing.

One point to be aware of: there are caveats about regions. Serverless does not allow region selection, and there is no contractual guarantee of processing location (infrastructure is US-centric). If domestic processing or latency requirements exist, you would need to choose AP_TOKYO_1 / AP_TOKYO_2 with dedicated on-demand deployment.

Since our use cases are mainly public information and code, we've organized this as "everyday development on serverless, on-demand Tokyo when requirements arise." Note that information equivalent to PCI/PHI is contractually prohibited from being input, so this line is drawn as an operational rule before any configuration.

Narrowed Down to Two Open-Weight Models

The models used by the team were narrowed down to two: Kimi K3 for strong, and DeepSeek V4 Flash-0731 for both weak and classifier. The Fireworks serverless Standard tier prices (per million tokens, as of August 2026) and roles are as follows.

The unit price difference between weak and strong is 21x for input and 54x for output. This gap is the very resource that automatic routing leverages — the more lightweight tasks we can route to weak, the lower the billing.

Flash-0731 supports 1M context and function calling, and we've verified streaming tool_call delta, which is the lifeline of the agent loop. According to vendor-published figures, agentic benchmarks have improved significantly from the preview version (Terminal Bench 61.8 → 82.7, etc.), reaching a level where we can confidently assign both the weak tier and classifier roles to a single model.

One more thing: we've confirmed that this model can also run locally on the DGX Spark. We've verified both standalone operation with llama.cpp and 2-node vLLM parallelism in past articles.

https://dev.classmethod.jp/articles/dgx-spark-deepseek-v4-flash-0731-llama-cpp/

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-0731/

The fact that the same open-weight model can run on both cloud serverless and local DGX Spark means that if governance requirements or cost structures change in the future, we can escape to local inference simply by swapping Switchyard's connection target. The value of leaning toward open weights is largely in the existence of this escape route.

Let me also explain why we narrowed down to two models. The more candidates you add, the greater the explanation cost of "which one to use when," the combinations of verification, and the tracking of price revisions. Given that Switchyard's routing has a strong/weak 2-tier structure, the operational experience is that fixing the team standard to 2 models and providing opt-in escapes at the route level (like auto-esc described later) for exceptions is the easiest to manage.

Switching Strong to Kimi K3

At the start of operations, the strong model was DeepSeek V4 Pro. It achieved perfect scores on coding benchmarks and had satisfactory quality, but for deep design discussions and planning, Kimi K3's responses felt superior. We first added it as an opt-in fixed route called k3-only, used it extensively personally, then switched the entire strong tier to K3.

The switch itself is just one line in the configuration file routes.toml. Just edit and restart — no image rebuild required. The classifier and weak continue to run on Flash-0731.

[targets.strong]
id = "accounts/fireworks/models/kimi-k3" # ← Changed from deepseek-v4-pro

However, K3 is not a free upgrade — it's clearly a trade-off. In pre-switch estimates applying K3 pricing to actual tokens from already-run benchmarks, the projected cost for strong-only was about 3.0x higher. Since routing works against the ladder of unit prices, the more expensive the strong model becomes, the more valuable "what could be routed to weak" becomes. On the other hand, preliminary hands-on use showed that thinking tokens tend to run long for brainstorming and strategic questions, making actual costs balloon beyond the unit price difference (7.1x vs. Pro), with latency also about 2.5x — so the switch was carried out together with re-measurement of benchmarks.

Post-switch measurements ran code_fix ×10 + tool_calls ×3 across 4 arms for a total of 52 runs. In addition to the standard auto arm, the opt-in auto-esc is also included. This mode starts all requests with weak and only upgrades to strong on a per-session basis when it detects "stuck trajectories" like repeated identical errors. Before the upgrade, responses are returned all at once, making it suitable for non-interactive workloads like cron jobs.

All 52 runs completed successfully with perfect scores across all arms, and even for the tool calling category (n=3), auto landed about 90% cheaper. All 10 runs of code_fix auto fell to weak, with zero Kimi K3 calls. Two notes: this benchmark is at a difficulty where weak alone achieves a perfect score, so this table demonstrates the effect of "not routing lightweight tasks to expensive models." Also, same-content re-classification skipping was active for the classification calls, meaning they barely ran, so actual operational classification costs will be higher than shown in this table.

These numbers didn't appear from the start. Actual measurements at operation launch (when strong was V4 Pro) showed about 27% reduction, and immediately after switching to K3, routing pinned entirely to strong and reduction almost disappeared. The cause was not K3 but a change in how the classification model that was swapped at the same time was reading requests. Subsequently, Switchyard overhauled the classification method from "task complexity categorization" to "estimating the probability that weak can complete the task," and with the same benchmark and same models, the numbers in the table above resulted. This "silent change in classification" will be explored in depth in a separate article along with experiments across multiple classification models.

Team deployment follows the same process as before — a PR to the distribution repository and an announcement. Route IDs (auto / strong-only / weak-only, etc.) are preserved across configuration updates, so team members' opencode settings remain unchanged. As a safety valve for when something feels off, a strong-only fixed escape route and revert procedure are also included.

Fixed Routing for Areas That Can't Be Left to Automatic Routing

Automatic routing is not a silver bullet. Through operation, we've come to understand specifically which areas the classifier tends to miss when fully delegated to.

The current classifier, by default, reads only the first and most recent user messages to estimate "the probability that weak can complete this task," then compares against a threshold to decide the destination. For tasks involving writing or fixing code, the threshold is 0.75; for non-code tasks like brainstorming or strategic discussion that don't match the classification rules, a stricter threshold is automatically applied. The design defaults to strong when uncertain.

When we actually measured 13 deep discussion questions through the same classification path as production, at the default threshold of 0.5 only 9 questions reached strong, but calibrating to 0.75 allowed 11–12 questions to be picked up (the boundary question fluctuates between runs). Even so, 1–2 questions still fall to weak. Around 90% can be caught but not all — that's the current state of automatic classification.

The range shown to the classifier can be changed in configuration. Adding recent_turn_window to a route includes the most recent N turns of conversation (including assistant responses and tool results) as classification material in addition to the initial task. Across the range of my measurements — varying the window off / 2 / 4 / 8 on 40 actual code work conversation shapes — the routing decisions barely moved, and the missed cases described above were not fixed by this. Where it did help was elsewhere: classification time dropped from a median of 17.5 seconds to around 3 seconds, and the classification model's thinking tokens dropped from 1,713 to 227. It seems the prolonged deliberation from insufficient material was eliminated. Since the same results held when the arm order was varied, recent_turn_window = 4 was incorporated into the distributed configuration.

Therefore, areas where misses are unacceptable are handled with fixed routing outside of automatic routing. The approach is to assign models per mode/agent in the opencode configuration.

{
  // Default is automatic routing
  "model": "switchyard/auto",
  // Auxiliary calls like title generation are fixed to weak
  "small_model": "switchyard/weak-only",
  "agent": {
    // Deep planning goes directly to strong (Kimi K3). Don't let classifier decide
    "plan": { "model": "switchyard/strong-only" },
    // Sub-agents inherit the calling model's, so explicitly cut that off
    "explore": { "model": "switchyard/weak-only" },
    "scout": { "model": "switchyard/weak-only" }
  }
}

This agent block is enabled by default in the distributed bundle's example configuration, so team members get the same boundary just by copying the settings.

Let me elaborate on the intent behind each. The reason plan mode is directly connected to strong is that brainstorming is infrequent and quality-dominant — this isn't a scenario where routing saves a few cents. The explicit weak assignment for explore/scout is defensive in the opposite direction: since opencode sub-agents inherit the calling model, leaving this unspecified means even simple tasks like reading grep results would be billed at the $15/1M output unit price. The weak fix for small_model comes from actual measurement — there were records of auxiliary calls like title generation flowing to strong via auto, and after fixing it, leakage to high unit prices disappeared.

Once you can see the boundary between "what to leave to automatic routing" and "what to escape with fixed routing," you can use the routing with much more confidence.

Preventing Data Leakage with strict-privacy and Web Search

Even if LLM traffic is converged to a single point at Fireworks, external transmission paths other than LLM remain in opencode. Before distributing to the team, we addressed these one by one. The configuration base is the recommended setup from opencode-with-strict-privacy, which has been organized internally, and the bundle includes a complete opencode.jsonc.example already merged with this.

The first challenge was the built-in websearch. opencode's standard web search connects anonymously without an API key to the search provider Exa's hosted MCP, which means it doesn't fall under corporate contract (data processing agreement) exceptions and is subject to the general privacy policy. And Exa's policy explicitly states that search queries will be used for "model training and fine-tuning" with no opt-out mechanism. Since search queries during coding may include error messages and internal context, we've disabled this with tools.websearch: false and permission.websearch: "deny". We also disabled conversation share link generation (share) and auto-updates.

Next is the pitfall I most want to convey in this section. If opencode's Global ~/.config/opencode/AGENTS.md does not exist, it loads ~/.claude/CLAUDE.md as a global rule as a compatibility fallback (official specification). If you're also using Claude Code, the content of personal settings and notes written for Claude will be injected into all requests to the models opencode connects to. In fact, in my own environment this fallback was triggering, and my entire personal Claude Code configuration was riding along in requests to DeepSeek. No matter how tightly you lock down the configuration files, this won't be closed unless you place an AGENTS.md. The fix is just to place a minimal Global AGENTS.md.

mkdir -p ~/.config/opencode
cat > ~/.config/opencode/AGENTS.md <<'EOF'
# Global rules

- Do not include customer names, internal project names, or unreleased code names in web search queries
EOF

Another trap is configuration scope. Privacy settings must always be placed in the Global scope (~/.config/opencode/opencode.json). Since project-side opencode.json overrides Global settings, disablement can be unintentionally bypassed when pulled along by per-repository configurations. Dual placement of .json and .jsonc is also prohibited since only one will be read. Furthermore, some disablement flags are on the environment variable side rather than in configuration files.

export OPENCODE_ENABLE_EXA=0        # Disable Exa search
export OPENCODE_EXPERIMENTAL=0      # Bulk disable experimental features
export OPENCODE_EXPERIMENTAL_EXA=0  # Legacy Exa flag
export OPENCODE_AUTO_SHARE=0        # Disable auto-sharing

Environment variables are a separate system from configuration files and easy to miss — in fact, on my main machine there was a period where only the configuration file side was applied and the rc side was not set. In team distribution, we've added env | grep OPENCODE_ to the onboarding checklist so it can be verified mechanically.

As a replacement for disabled websearch, we distribute a search API whose non-use for training is contractually confirmed as a skill that each person calls directly with their own key. The backend is a choice of Gemini (Google AI Studio, with a free tier) or OpenAI (Responses API's web_search), and the selection criteria is not search provider features but "whether there is a contract stating that queries won't be used for training." Gemini requires a key issued from a GCP project with billing enabled (free tier keys are subject to training use), while OpenAI does not use API calls for training by default — so this is the one part of onboarding I make sure people don't skip.

Team-Shared RAG as the Entry Point for Business Use

Up to this point I've been talking about the developer entry point, but the value of a team AI environment is not limited to coding. As an entry point for business use cases — searching across documents, summarizing, and answering with citations — we've built a team-shared RAG based on NVIDIA RAG Blueprint on DGX Spark.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-rag-blueprint-mcp/

I'll defer the details to the above article (published 2026-08-07), but what's important to note in the context of this article is the form of connection. RAG is published as an MCP server, so developers can access team knowledge directly from opencode or Claude Code. For non-engineers, the Web UI is the entry point. We've verified that the search and generation pipeline can run locally on DGX Spark, so the design can maintain the "data doesn't leave the premises" line even when internal documents are loaded.

The developer entry point is opencode + routing, and the knowledge entry point is RAG + MCP — these two together form the overall picture of this environment.

Running Observability on Two Systems

A team environment doesn't end at "deployed." Improvement only happens when you can observe how it's actually being used and how much it costs. Currently we run observability on two systems.

The first is NVIDIA's NeMo Relay. It records Claude Code and Codex CLI agent execution (model calls, tool executions, sub-agent branching) and forwards them in OpenTelemetry format from OTel Collector to Tempo / Grafana. Setup and caveats are introduced in the following article (published 2026-08-06).

https://dev.classmethod.jp/articles/nvidia-nemo-relay-first-touch/

One point to mention about Relay upfront: by default, the full prompt text reaches the observability infrastructure. For team deployment, we use the first-party pii-redaction plugin to drop the full text and retain only the metadata needed for routing and cost analysis.

The second is Switchyard's own routing logs. On each person's local machine, per-request classification decisions, models, and token counts are recorded in routing.jsonl (in a Docker named volume, so they survive configuration changes and key rotations), and a weekly snapshot script collects and aggregates the team's data. Per-route aggregation can separate not just the tier distribution of the auto path, but also fixed-route usage in the form of pinned:kimi-k3, making it possible to track "how much auto and fixed were each used." The current server now records classification calls as independent rows with tier="classifier", so the classification cost (actual measurement: about 8% of total) that could only be estimated in the old architecture is now directly reflected in weekly aggregation. There's also a Prometheus-format /metrics endpoint that provides health metrics like fail-open counts. The billing reconciliation target is the Fireworks dashboard's per-model usage. Since strong and weak are different models, per-model usage directly becomes the tier distribution and cost breakdown.

Honest Current State of Observability Coverage

The observability goal is "consolidating all harnesses into Relay," but we haven't gotten there yet. Being honest about coverage per path:

The reason the key opencode path isn't on Relay is related to Relay's support approach. The passive plugin method for opencode was shelved in June 2026, and NVIDIA has announced opencode support via the wrapped execution method (NeMo Relay PR #73 close comment, 2026-06-03). Relay itself was officially released as version 0.7.x in August 2026. The integration with Switchyard is in the middle of a handover — the experimental integration example in the Relay repository (where Relay calls Switchyard's classification API) was removed in 0.8 and replaced by a native plugin that Switchyard holds, as explicitly stated in the documentation (NeMo Relay Epic #401, Switchyard PR #270). The plan is to verify and deploy to the team as soon as that lands, completing the consolidation.

I should also mention that Switchyard itself underwent a major redesign to a standalone Rust server implementation in August 2026, and its maturity is still marked as pre-alpha ("Not for production use"). Since it hasn't been published to a package registry yet, the bundle tracks it with a pinned commit SHA. This is a configuration that requires willingness to keep up with upstream changes, which is something to note for those who want to try it.

In the meantime, opencode is observed via the alternative of routing.jsonl and the Fireworks dashboard. While it's inferior to Relay in trace granularity, for the actual cost figures this path is actually more accurate. routing.jsonl is primary data recorded by the router per request, and Fireworks' per-model usage is the billing itself. Even if the observability picture is uneven, the ability to answer "how much did it cost" hasn't degraded — that's the current assessment.

Summary

I've introduced a team AI environment that connects opencode to Fireworks via NeMo Switchyard and automatically routes between two open-weight models, Kimi K3 and DeepSeek V4 Flash-0731. In synthetic benchmarks, we were able to complete tasks at $0.0009/run compared to $0.035/run with strong-only without dropping task completion rate, while also experiencing a period when reduction almost disappeared due to a classification model update. The takeaway from operation is that reduction effectiveness is not a fixed spec of the environment, but a "variable to keep observing" that fluctuates with configuration and traffic. The package — not relying solely on routing, escaping areas where the classifier can't be trusted with per-mode fixed routing, closing data exit points with strict-privacy settings and AGENTS.md placement, and observing actual costs with routing.jsonl and the Fireworks dashboard — built on top of git repository distribution is the key point of this as a team environment.

Let me also be honest about the limitations. The synthetic benchmark used for quality evaluation is at a difficulty level where weak alone achieves a perfect score, so it remains a weak proof that "routing protected quality." Classification turns add waiting time for classification calls, and the actual savings amount varies greatly depending on what's in the traffic. Observability is also still split across two systems while waiting for opencode's Relay support. I intend to update these aspects as real traffic accumulates and upstream progress is made.

The team-shared RAG construction piece has already been published, so reading it alongside this should connect the overall picture. I also plan to cover the new Rust server implementation internals of the router in a separate article aligned with upstream releases. Next, I'm thinking about trying to load Hermes Agent as a team assistant and create an entry point where you can pull knowledge from Slack in natural language.

Reference Links

• himorishige/switchyard-opencode-bundle — Router distribution bundle from this article (Apache-2.0)
• NVIDIA-NeMo/Switchyard
• NVIDIA/NeMo-Relay
• cm-dyoshikawa/opencode-with-strict-privacy — Recommended opencode privacy configuration
• Fireworks AI — Serverless Pricing
• Fireworks AI — Data Handling (Zero Data Retention)
• Fireworks AI — DPA (Section 4.3(f) is the contractual prohibition on training use)
• Kimi K3 on Fireworks
• DeepSeek-V4-Flash-0731 (Hugging Face)
• Trying Out NVIDIA's New LLM Routing Infrastructure NeMo Switchyard (Published 2026-07-03, covers the early Python router; includes addenda)
• Building a Team-Shared RAG with NVIDIA RAG Blueprint × DGX Spark and Connecting via MCP (Published 2026-08-07)
• NeMo Relay first-touch (Published 2026-08-06)
• Running DeepSeek V4 Flash-0731 on a Single DGX Spark with llama.cpp (Published 2026-08-02)
• Running DeepSeek V4 Flash-DSpark on 2 DGX Spark Nodes