Open-weight model team AI environment covering everything from development to business use

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

When a team starts using AI coding agents, two problems quickly surface: cost and data handling. Delegating everything to a powerful model makes billing hard to predict, and leaving settings unconfigured means no one can explain where internal information mixed into prompts is being sent.

I'm validating a team AI environment in our internal development team that solves these two issues with a combination of "open-weight models + automatic routing + leak-proof configuration." The router component is also published as a Docker bundle.

https://github.com/himorishige/switchyard-opencode-bundle

The core of the architecture is connecting opencode to Fireworks AI via NVIDIA's LLM routing infrastructure NeMo Switchyard, automatically switching between strong and lightweight models depending on task complexity. To give the conclusion upfront: in actual measurements across 39 runs of a synthetic coding benchmark, we achieved approximately 27% cost reduction compared to strong-fixed, without dropping task completion rate (including classifier costs; figures as of when strong was DeepSeek V4 Pro and weak was DeepSeek V4 Flash preview). The trade-off is latency — responses are longer by the amount of routing overhead (wall time median 30 seconds vs 21 seconds).

The mechanism of Switchyard itself and the setup procedure are introduced in the following article (published 2026-07-03, with actual team environment measurements added on 2026-08-05).

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article introduces the overall picture of the team environment built on top of that routing. It's a record of design decisions — where to place what, why Fireworks, how we narrowed down to two models, settings to prevent data leakage, and the current state of observability — so I hope it resonates with people in the position of distributing AI agents to their team.

Summarizing the Team Environment in One Diagram

First, the overall picture. The current operational scope covers placing a router locally on each developer's machine and routing LLM traffic to Fireworks serverless, while loading a team-shared RAG and observability stack onto DGX Spark is still at the validation stage.

The left half of the diagram — the path from opencode through Switchyard to Fireworks — is the main subject of this article. Switchyard runs as a Docker container on each person's local machine, classifying task complexity per request, and currently automatically switches between a strong model (Kimi K3) and a lightweight model (DeepSeek V4 Flash-0731).

Claude Code's LLM traffic goes directly to Anthropic and does not pass through Switchyard. The routing target is exclusively the opencode side using open-weight models, and the structure only shares the observability layer (NeMo Relay) with Claude Code.

The right half — the service layer and inference layer — is where the team-shared knowledge base (RAG) and local LLM reside. Since the RAG is published as a separate article, I'll limit coverage here to an introduction.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-rag-blueprint-mcp/

Placement is Determined by State and Keys

The first decision when building this architecture was "should each component go on a shared server or on each person's local machine?" The two criteria were: does it hold state that should be shared, and who holds the API key.

The decision not to put the router on a shared server was the first fork in the road. Since Fireworks API keys are issued per individual, consolidating them into a shared proxy would break the correspondence between key ownership and billing ownership. The router itself is stateless, and having one Docker container resident on a laptop is not a burden. In that case, placing it on each person's local machine and binding it to 127.0.0.1 is cleaner for both key management and attack surface. Distribution is handled via a git repository and docker compose up -d, with updates requiring just git pull and restart.

Conversely, the RAG vector DB and observability traces are data that only make sense when sharing a single state across the team. Since keys can also be consolidated server-side for these, they live on the always-on machine.

At one point I tried consolidating web search onto a "shared server with one team-wide contract," but ultimately moved it back to each person's local machine. The friction of issuing search API keys individually is low, and if a server can be eliminated, it should be — operations are lighter that way. The resulting distinction I arrived at — "if the server can be eliminated, distribute it as a skill; if the server essentially needs to remain, expose it via MCP" — is directly reflected in the different distribution methods for web search (skill) and RAG (MCP) that appear in later sections.

Why We Run Open-Weight Models on Fireworks

There are three reasons we chose Fireworks AI serverless as our LLM provider.

The first is freshness. New versions of open-weight models are available on serverless from day one of release. In fact, the official DeepSeek V4 Flash 0731 version was listed on the day of its July 31, 2026 release, and we were able to swap out the team's weak tier within that same week. With managed cloud services, open-weight model listings can often be a generation behind, so this matters for freshness-sensitive use cases.

The second is that data handling can be confirmed at the contract level. Fireworks' DPA (Data Processing Addendum) section 4.3(f) contractually prohibits using Covered Data, including prompts and inputs/outputs, for model training and improvement. Zero Data Retention is also enabled by default, meaning prompts and generated results only exist in volatile memory during request processing. The Trust Center also shows certifications including SOC 2 Type II, ISO 27001:2022, and AI management ISO/IEC 42001:2023. For team use, we prioritized being able to confirm that "inputs won't be used for training" in contract documents rather than a single line on a policy page.

The third is pay-per-use billing. With no GPU or container to manage and payment only for tokens used, cost reductions from routing described below are directly reflected in the bill.

One point requiring attention is regions. Serverless offers no region selection and no contractual guarantee of processing location (infrastructure is primarily US-based). If there are requirements for domestic processing or latency, you would need on-demand dedicated deployment with AP_TOKYO_1 / AP_TOKYO_2.

Since our use cases are primarily public information and code, we've organized things as "serverless for regular development, on-demand Tokyo when requirements arise." Note that information equivalent to PCI or PHI is contractually prohibited from being submitted, so this line is drawn as an operational rule before any configuration.

We Narrowed Down to Two Open-Weight Models

For team use, we narrowed model selection to two: Kimi K3 for strong, and DeepSeek V4 Flash-0731 for both weak and classifier. The Fireworks serverless Standard tier prices (per 1 million tokens, as of August 2026) and roles are as follows.

The unit price difference between weak and strong is 21x for input and 54x for output. This gap is the source of automatic routing's savings — the more lightweight tasks are routed to weak, the lower the bill.

Flash-0731 supports 1M context and function calling, and streaming tool_call deltas — the lifeline of agent loops — have been validated. Vendor-published figures show major improvements in agentic benchmarks from the preview version (Terminal Bench 61.8 → 82.7, etc.), making it comfortably suitable for the dual role of weak tier and classifier in a single model.

One more point: we've confirmed this model can also run locally on DGX Spark. We've previously validated both single-node operation with llama.cpp and two-node vLLM parallelism.

https://dev.classmethod.jp/articles/dgx-spark-deepseek-v4-flash-0731-llama-cpp/

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-dspark/

The fact that the same open-weight model runs on both Fireworks serverless and local DGX Spark means that if governance requirements or cost structures change in the future, we can escape to local inference simply by swapping Switchyard's connection target. The value of committing to open-weight models is largely the existence of this escape route.

I'll also explain why we narrowed down to two models. The more candidates you add, the more explanation overhead there is for "which one to use when," the more validation combinations, and the more price change tracking is needed. Given that Switchyard's routing is a two-tier strong/weak structure, the most operationally comfortable approach is to fix the team standard at two models and handle exceptions through per-route opt-in (like the k3-only form described later).

Trying the Switch to Kimi K3 as Strong

When we started operations, the strong model was DeepSeek V4 Pro. Its quality on coding benchmarks was perfect and satisfactory, but for deep design discussions and planning, Kimi K3's responses felt better. I first added it as an opt-in fixed route called k3-only, used it personally for a while, and then switched the entire strong tier to K3.

The switch itself is just changing two places in route.yaml. The classifier and weak continue running on Flash-0731.

routes:
  auto:
    strong:
      model: accounts/fireworks/models/kimi-k3 # ← changed from deepseek-v4-pro
  strong-only:
    type: model
    target: accounts/fireworks/models/kimi-k3 # ← same

However, K3 is not a free upgrade — it is clearly a trade-off. Pre-switching estimates applying K3 pricing to actual tokens from the already-run benchmark showed that costs would be approximately 3.0x for both strong-fixed and auto, while the auto reduction rate was estimated to remain roughly the same. Since routing works against the price ladder, the more expensive strong becomes, the more valuable "what could be routed to weak" becomes. On the other hand, from pre-switch hands-on use, we also saw that reasoning tokens tend to run long for brainstorming and strategy-type questions (making actual cost 7.1x vs Pro), and latency was about 2.5x — so the switch was carried out alongside a re-run of the benchmark.

The results of re-running the same benchmark (code_fix ×10 + tool_calls ×3 across 3 arms, 39 runs total) after the switch are in the table below. Quality and reliability were unchanged from before the switch: all 39 runs completed, zero failures, perfect scores across all arms.

The strong-fixed side landed at 3.7x, close to the pre-estimate of 3.0x. What was unexpected was the auto side. In this run, all 10 runs of code_fix were pinned to strong, making it nearly identical in cost to strong-fixed including classifier. When I separated out just the classification calls and re-ran to investigate, the cause was not K3's response text, but rather the classifier-side model update (V4 Flash preview → 0731) deployed at the same time as K3 had changed judgment tendencies for conversation transcripts, skewing them toward strong. Since single-prompt calibration showed matching judgments between both versions, this wasn't detectable at swap time. I recalibrated the judgment prompt, reflected it in the distributed bundle, and confirmed restoration of weak-side routing for actual agent loop-shaped judgments.

On the other hand, for tool calling tasks (n=3), auto landed about 40% cheaper than strong-fixed, showing that reductions do appear when flow goes to weak. Reduction rate is not a fixed property but "a variable that swings from zero to over 40% depending on pin distribution," and presenting single-run benchmark figures as a promise for production use is dangerous — that's the biggest lesson from this switch. Where things settle in actual production will be determined by continuously measuring real team traffic.

Team rollout follows the same pattern as before: a PR and announcement on the distribution repository, with revert instructions (git pull to restore route.yaml and restart) included in case the feel isn't right.

Use Fixed Routing to Escape What Routing Can't Handle

Automatic routing is not omnipotent. Through operation, I've gained a concrete understanding of areas that should not be left to the classifier.

Switchyard's coding_agent preset judges task complexity along "coding work axes" — scope of code changes, number of tool calls, whether codebase context is needed. As a result, deep business discussions and architecture brainstorming can be read as "no code touched, no tools needed" and get pushed to the lightweight task side.

When I actually measured 17 questions including strategy and planning types through the same judgment path as production, 11 of the 13 deep discussion questions reached strong, but the 2 that missed were misclassified as "simple" with a high confidence of 0.95. This is inverted from the correctly-classified 11, which had confidence of 0.70–0.85. Since the pattern shows more hesitation for correct answers and more confidence for wrong ones, adjusting the confidence threshold (min_confidence) cannot rescue these 2 questions.

So for areas where the classifier is structurally weak, I bypass routing and use fixed assignments. This takes the form of assigning models per mode/agent in opencode's configuration.

{
  // Default is automatic routing
  "model": "switchyard/auto",
  // Auxiliary calls like title generation use weak-fixed
  "small_model": "switchyard/weak-only",
  "agent": {
    // Deep planning goes directly to strong (Kimi K3). Don't let classifier judge this.
    "plan": { "model": "switchyard/strong-only" },
    // Sub-agents inherit the caller's model, so explicitly break the chain
    "explore": { "model": "switchyard/weak-only" },
    "scout": { "model": "switchyard/weak-only" }
  }
}

This agent block is enabled by default in the distribution bundle's example configuration, so team members get the same delineation just by pasting the config.

Some notes on the intent behind each. Plan mode is set to strong-direct because brainstorming is infrequent and quality-dominant — this isn't a scenario where we're saving a few cents through routing. The explicit weak assignment for explore/scout is a defensive measure in the opposite direction: since opencode sub-agents inherit the caller's model, without explicit assignment you'd end up paying the $15/1M output price even for simple tasks like reading grep results. The small_model weak-fixed assignment came from actual measurements — there were records of auxiliary calls like title generation flowing through auto and reaching strong, and after fixing it, leakage to high-price tiers disappeared.

Once you can see the line between "what to leave to automatic" and "what to escape with fixed," routing becomes much more comfortable to use.

Preventing Data Leakage with strict-privacy and Web Search

Even if LLM traffic is converged to a single point at Fireworks, opencode still has external transmission paths other than LLM. Before team distribution, I closed these one by one. The base for our settings is the recommended configuration from internally organized opencode-with-strict-privacy, and the bundle includes a completed opencode.jsonc.example merged with this.

The first issue is the built-in websearch. opencode's standard web search connects anonymously without an API key to the search provider Exa's hosted MCP, which means it doesn't fall under enterprise contract (data processing agreement) carve-outs and is subject to the general privacy policy. Exa's policy explicitly states that search queries will be used for "model training and fine-tuning," with no opt-out mechanism. Since search queries during coding can contain error messages and internal context, I've disabled this with tools.websearch: false and permission.websearch: "deny". I also disabled conversation share link generation (share) and auto-updates.

Next is the most important gotcha in this section. When opencode's Global ~/.config/opencode/AGENTS.md doesn't exist, it falls back to reading ~/.claude/CLAUDE.md as the global rule (official specification). If you're also using Claude Code, the personal settings and notes you wrote for Claude get injected into every request to opencode's connected model. In my own environment, this fallback was indeed triggered, and my entire Claude Code personal settings were riding along in DeepSeek requests. No matter how tightly you configure settings files, this won't be closed unless you place an AGENTS.md. The fix is simply placing a minimal Global AGENTS.md.

mkdir -p ~/.config/opencode
cat > ~/.config/opencode/AGENTS.md <<'EOF'
# Global rules

- Do not include customer names, internal project names, or unpublished code names in web search queries
EOF

Another trap is configuration scope. Always place privacy settings in Global scope (~/.config/opencode/opencode.json). Since project-side opencode.json overrides Global, settings can slip through and get invalidated when dragged along by per-repository configurations. Double-placing .json and .jsonc is also prohibited since only one will be read. Furthermore, some disable flags are on the environment variable side rather than in configuration files.

export OPENCODE_ENABLE_EXA=0        # Disable Exa search
export OPENCODE_EXPERIMENTAL=0      # Disable experimental features in bulk
export OPENCODE_EXPERIMENTAL_EXA=0  # Old Exa flag (legacy)
export OPENCODE_AUTO_SHARE=0        # Disable auto-sharing

Environment variables are a separate system from configuration files and are easy to miss — in fact, on my main machine there was a period where only the configuration file side was applied and the rc side was unconfigured. For team distribution, we've added env | grep OPENCODE_ to the onboarding checklist to enable mechanical verification.

As a replacement for the disabled websearch, we distribute a skill that directly calls a search API with each person's own key, from a provider whose contractual terms confirm learning-non-use. The backend options are Gemini (Google AI Studio, with a free tier) and OpenAI (Responses API's web_search), and the selection criteria are not the search provider's features but "does the contract ensure queries won't be used for training." There's a difference: Gemini requires a key issued from a billing-enabled GCP project (free tier keys are subject to training use), while OpenAI's API defaults to learning-non-use. This is one part of the onboarding instructions we ask people not to skip.

Team-Shared RAG as the Entry Point for Business Use

Everything so far has been about the developer's entry point, but the value of a team AI environment isn't limited to coding. As an entry point for business use — searching across documents, summarizing, answering with citations — we've built a team-shared RAG based on NVIDIA RAG Blueprint on DGX Spark.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-rag-blueprint-mcp/

Details are in the article above (published 2026-08-07), but what I want to note in the context of this article is the connection form. The RAG is exposed as an MCP server, so developers can access team knowledge directly from opencode or Claude Code. For non-engineers, a Web UI serves as the entry point. We've validated that the search and generation pipeline can run locally end-to-end on DGX Spark, so the design maintains the "data doesn't leave" line even when internal documents are added.

The development entry point is opencode + routing, and the knowledge entry point is RAG + MCP — this two-pronged approach is the full picture of this environment.

Observability Runs on Two Systems

A team environment isn't "done once deployed" — improvement only cycles when you can observe how it's actually being used and what it costs. Current observability runs on two systems.

The first is NVIDIA's NeMo Relay. It records Claude Code and Codex CLI agent execution (model calls, tool executions, sub-agent branching) and sends it in OpenTelemetry format from OTel Collector to Tempo / Grafana. Setup and caveats are introduced in the following article (published 2026-08-06).

https://dev.classmethod.jp/articles/nvidia-nemo-relay-first-touch/

One thing to say upfront about Relay: by default, prompt body text reaches the observability infrastructure. For team deployment, we use the first-party pii-redaction plugin to drop body text and retain only the metadata needed for routing and cost analysis.

The second is Switchyard's own routing log. Per-request judgment, model, and token count are recorded to routing.jsonl locally on each machine (since it's stored in a Docker named volume, it survives configuration changes and key rotations), and a weekly snapshot script collects and aggregates team-wide data. In per-route breakdowns, in addition to the auto path's tier distribution, fixed-route usage is also separated out in forms like pinned:kimi-k3, making it possible to track "how much each of automatic and fixed was used." The billing reconciliation target is Fireworks dashboard model-level usage. Since strong and weak are different models, model-level usage directly becomes the tier distribution and cost breakdown.

Honest Current State of Observability Coverage

The observability target state is "unify all harnesses into Relay," but we haven't reached that point yet. Honestly stated, coverage by path is as follows.

The reason opencode isn't on Relay is Relay's support approach. The passive plugin method for opencode was dropped in June 2026, and NVIDIA has announced wrapped execution-based opencode support (NeMo Relay PR #73 close comment, 2026-06-03). Relay itself was officially released as the 0.7 series in August 2026, and native plugin integration with Switchyard is also in progress (NeMo Relay Epic #401, Switchyard-side PR #270), so the plan is to validate and deploy to the team as soon as support lands and complete the unification.

Until then, opencode is observed via routing.jsonl and the Fireworks dashboard as substitutes. While trace granularity is inferior to Relay, for actual cost figures this path is actually more accurate. routing.jsonl is primary data recorded by the router per request, and Fireworks' model-level usage is the billing itself. Even if observability looks uneven, the ability to answer "how much did it cost" has not degraded — that's the assessment of where we stand.

Summary

I introduced a team AI environment that connects opencode to Fireworks via NeMo Switchyard, automatically routing between two open-weight models — Kimi K3 and DeepSeek V4 Flash-0731. The pre-switch synthetic benchmark showed approximately 27% cost reduction (including classifier) without dropping task completion rate, while the re-run after switching strong to K3 had a run where pin distribution changed and reduction barely materialized. The operational takeaway is that reduction rate is not a fixed spec of the environment but "a variable that swings with configuration and traffic — something to keep observing." Rather than relying solely on routing, escape classifier-weak areas with per-mode fixed assignments, close data exit points with strict-privacy settings and AGENTS.md placement, and observe actual costs with routing.jsonl and the Fireworks dashboard. Packaging all of this into distribution from a git repository is the key point as a team environment.

I'll also note the honest limitations. The synthetic benchmark used for quality evaluation was at a difficulty level where weak alone achieves perfect scores, so it remains weak proof that "routing protected quality." Latency extends by the routing overhead, and even for the same task, whether it's pinned to strong or weak can vary by session, so reduction rate has variance. Observability also remains split across two systems, awaiting opencode's Relay support. I plan to update these as real traffic accumulates and upstream developments progress.

The team-shared RAG construction article is already published, so reading them together should connect the full picture. Next, I'm thinking of trying to put Hermes Agent in as a team assistant and create an entry point to pull knowledge from Slack using natural language.

Reference Links

• himorishige/switchyard-opencode-bundle — Router distribution bundle for this article (Apache-2.0)
• NVIDIA-NeMo/Switchyard
• NVIDIA/NeMo-Relay
• cm-dyoshikawa/opencode-with-strict-privacy — Recommended privacy configuration for opencode
• Fireworks AI — Serverless Pricing
• Fireworks AI — Data Handling (Zero Data Retention)
• Fireworks AI — DPA (4.3(f) is the contractual prohibition on training use)
• Kimi K3 on Fireworks
• DeepSeek-V4-Flash-0731 (Hugging Face)
• Trying NVIDIA's New LLM Routing Infrastructure NeMo Switchyard (published 2026-07-03)
• Building a Team-Shared RAG with NVIDIA RAG Blueprint × DGX Spark and Connecting via MCP (published 2026-08-07)
• NeMo Relay first-touch (published 2026-08-06)
• Running DeepSeek V4 Flash-0731 on a Single DGX Spark with llama.cpp (published 2026-08-02)
• Running DeepSeek V4 Flash-DSpark on Two DGX Spark Nodes