I thought about the ideal form of AI Gateway and router for using coding agents as a team

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

I think the token consumption of coding agents has been ballooning even further over the past six months and becoming a hot topic. An agent loops dozens of turns from a single instruction, re-reading context every turn, so it easily blows past budgets that were set with the chat era in mind. Overseas, there have been reports of major companies instructing teams to switch tools due to costs reaching $500–2,000 per person per month. No matter which agent you use, these figures are becoming impossible to ignore.

https://code.claude.com/docs/en/gateways

For individuals, it's just barely manageable to absorb a sizable amount of usage under each company's comprehensive subscription plan. But when it comes to teams or the whole organization, the story changes. The reality is that most people are juggling a mix of Team or Enterprise seats, Console API, and pay-as-you-go billing from Amazon Bedrock or Claude Platform on AWS. Throw in agents other than Claude Code, and the number of entry points grows even more. Codex CLI runs on a ChatGPT subscription, so in my case it's a Claude subscription, a ChatGPT subscription, and a pay-as-you-go window for the remaining tools—a multi-layered management situation. I've been wondering how to simplify this for a long time.

In my previous article, I organized how to position NeMo Relay and NeMo Switchyard as AI Gateways in terms of component responsibility separation (article as of 2026-08-29). The team AI environment itself is covered in the open-weight model series (article as of 2026-08-08).

https://dev.classmethod.jp/articles/nemo-relay-switchyard-ai-gateway/

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

In this article, I divide AI Gateways into five layers—billing path, entry point, routing, execution side, and observability—and envision the ideal setup for using coding agents in team development and business use as of August 2026. For actual measurements, I limited myself to a single verification: routing three types of agents through a common Gateway. I hope this resonates with those on the side of managing coding agents for a team or entire organization.

Thinking About AI Gateways in Five Layers

In my previous article, I noted that the term "AI Gateway" encompasses a wide range of functionality from authentication to auditing, and that having everything in one place is too heavy for everyday environments. This time, I want to think from one step further back. Until we decide who is paying, we can't decide where to place the entry point.

Broken down into five layers, it looks like this:

Layers 2 through 5 largely overlap with the "supplement missing functionality with dedicated components" table from my previous article. The difference is that I've placed the billing path at the top. As we'll see later, if you choose a subscription, Anthropic already holds the entry point on their side, and there's almost no room to insert your own. Conversely, if you choose pay-as-you-go, you need to own the entry point yourself, and that's where a common Gateway comes into play for the first time. Even with the same number of layers, the choice of billing path shifts where the remaining layers sit—that's the backbone of this article.

The Billing Path Determines Whether Each Layer Is Needed

The official Claude Code documentation lists six deployment options. Putting billing, authentication, and what you lose into a single table reveals their differing characteristics.

The official documentation says "Teams or Enterprise is best for most organizations." Indeed, with subscriptions, Anthropic holds the entry point, and you can set spend limits per organization or individual through admin settings. With Enterprise, you can also set per-group limits, and notifications fire at 75% and 90%. For Team, the Standard seat is $25/month (1.25x Pro) and the Premium seat is $125/month (6.25x); if you exceed the limit, you top up with usage credits at API unit pricing. As of August 2026, the current Enterprise plan has changed to charge $20 per seat plus all Claude, Claude Code, and Cowork usage at API unit pricing, with no per-seat usage cap.

There are two things worth keeping in mind here. First, subscription limits are determined by 5-hour and weekly time windows, not token counts. Even with the same seat, the amount you can use varies depending on how cache works and the model configuration. Second is the prompt cache TTL: subscriptions get 1 hour, while API keys, cloud providers, and periods when usage credits are being consumed get 5 minutes. Whether cache misses occur when you pause and return to a long session varies by path.

What directly connects to the discussion of bundling multiple agents is the authentication constraint. In February 2026, Anthropic revised its Consumer Terms to prohibit using OAuth authentication for Free, Pro, and Max from third-party harnesses. It was reported that actual blocking began on April 4th. The official legal page also clearly states that third-party developers may not embed Claude.ai login into their own applications, nor route requests on behalf of users using Pro / Max credentials. In other words, if you want to use Claude with tools like OpenCode, Pi, or Codex-family tools, you need an API key or cloud provider credentials.

This structure isn't limited to Anthropic. On the OpenAI side, Codex CLI runs on a ChatGPT subscription, sharing a 5-hour and weekly quota across web, CLI, and IDE, with overages requiring credits top-ups. For teams, there's the Business plan (formerly Team), and on August 10, 2026, a Premium seat was announced with 5x the quota of Standard and no 5-hour cap. At $125/month or $100/year (annual), it carries the same price tag as Claude's Team Premium. Both companies are converging on a two-tier approach: their own first-party agents on their own subscription, and third-party tools and other companies' models on API pay-as-you-go. The number of entry points grows by the number of companies whose agents you use, plus however many pay-as-you-go windows you have.

This is ultimately why I can't escape multi-layered management. Subscriptions are entry points each company has prepared exclusively for their own agents, so while they're cheaper to run, there's no way to consolidate multiple companies' subscriptions into a single entry point. If you want consolidation, you'd need to drop subscriptions, move all agents to API pay-as-you-go, and handle everything through your own entry point. But then the token consumption that was previously absorbed at a flat rate shows up directly on your bill. In Claude's case, the prompt cache TTL also shrinks from 1 hour to 5 minutes, so the same usage pattern leads to worse cache performance and the bill grows further. You'll want to look at the numbers in the next section before deciding which way to go.

Applying Billing Paths to Real Data from a Heavy User

The following is my Claude Code usage over the last 30 days (2026-07-31 to 08-29), aggregated with ccusage—please read this as an example of one person on the heavier end within a team. This table only includes Claude Code consumption. I also use Codex CLI on the ChatGPT Business (formerly Team) plan and, for some projects, Fireworks AI's pay-as-you-go for open-weight models, but those are separate entry point stories and are not included in these numbers.

98% of the total is cache reads. In a previous aggregation, I found an average of 21 agent loop turns per human instruction. Since the full context is re-read every turn, the total token count reaches this order of magnitude. The flat-rate plan absorbs this structure entirely, working out to about 1/40th of the API list price equivalent.

Placing this alongside publicly available averages gives a sense of proportion. Anthropic's cost management page cites an enterprise deployment average of $13 per working day per person, or $150–250/month, with 90% of users staying under $30 per working day. The report mentioned at the outset cited $500–2,000/month. My 30-day list-price equivalent of $7,879 is over 30 times the average, and about 4 times the upper end of the reported range.

Applying billing paths to these numbers, the cache hit rate determines the conclusion. If you switch to a pay-as-you-go path while maintaining close to the current ~98% hit rate, the monthly cost is dominated by the cache read unit price and stays relatively manageable. However, given a pricing structure where the cache read unit price is around 1/10th of the input price, my estimates show that if the hit rate drops to 90%, the pay-as-you-go equivalent is about 1.8x higher; at 50%, it jumps to about 4.9x. This is exactly where the difference between the subscription's 1-hour cache and the API's 5-minute cache makes itself felt—just changing the path for the same work could shift the bill by an order of magnitude.

Personally, I think this sensitivity is the true nature of "subscriptions being cheap." As we'll see in the measurement section later, even a single-turn Claude Code call that just returns "connected" writes 33,359 tokens to cache. The tool definitions and system prompt are that long, and subsequent turns re-read all of that. Keeping in mind that agent billing is determined not by the amount of text generated, but by the amount of context re-read, changes how you see the configuration patterns in the next section.

Where to Place the Entry Point: Seven Configuration Patterns

Here are seven patterns for the second layer entry point, combined with billing paths. The columns are billing, attribution unit, how to enforce budgets, usable agents, and what you lose.

The P3 Claude apps gateway is Anthropic's official self-hosted Gateway, published on June 29, 2026. It's bundled in the claude binary, starts with claude gateway --config gateway.yaml, authenticates users via an OIDC IdP, and bundles Bedrock, Claude Platform on AWS, Agent Platform, Foundry, and the Anthropic API as upstreams with failover. Spend limits can be set per user, group, and organization on daily/weekly/monthly bases via the Admin API; when exceeded, it returns a 429 billing_error, and Claude Code displays warnings at 75% and 95%. On the other hand, there are no service tokens for CI, WebSearch is disabled, and the 1-hour cache TTL is unavailable. And only Claude Code and Claude Desktop can be placed on it. This summary is based on my reading of the official documentation; I did not measure it in practice this time.

The common Gateways in P2 and P4 are represented by LiteLLM and Bifrost. LiteLLM lets you attach budgets, time periods, and model restrictions to virtual keys, and also offers budget fallback to drop to a cheaper model when the budget runs out. However, since there was a supply chain incident on March 24, 2026, where malicious 1.82.7 and 1.82.8 versions were published to PyPI, version pinning should be a prerequisite for adoption. Bifrost is a Go single binary that only depends on Postgres—its minimal dependencies are its distinguishing feature. P6 is a path for organizations already running an API Gateway like Envoy AI Gateway or Kong, adding an AI plugin to it; Envoy v0.4.0 added a translator that converts Anthropic Messages to Bedrock Converse.

My current situation is P7. Illustrated, it looks like entry points exist one per subscription, with only observability converging.

Looking at the table, what stands out is that no single row has both the "all agents supported" column and the "budget enforced in one place" column checked at the same time. P2 and P4 support everything but give up subscription-only Claude Code features; P1 and P3 enforce budgets in one place but only support Claude Code. P7 supports everything but budget management multiplies by the number of subscriptions. Thinking about the ideal form comes down to figuring out how to fill in that one column we've had to give up.

Connecting Three Types of Agents to a Common Gateway

Here's the main topic. I actually had a single machine handle the "supports everything" side of P2 and P4. I set up LiteLLM v1.98.0 and Postgres on my everyday machine, issued 4 virtual keys per user, and connected 3 types of agents. The upstream is Bedrock Sonnet 5 for Claude Code, OpenAI gpt-5.6-terra for Codex CLI, and Fireworks AI's DeepSeek V4 Flash for OpenCode. I did not use an Anthropic Console key or claude.ai login. Codex CLI normally also runs on a ChatGPT subscription, but here the goal was to get it onto the common entry point, so I'm using the API path.

The verification results are as follows:

Connectivity, attribution, budget, and model restrictions all worked as expected. Budget exceeded returning 429 is the same status code as with the Claude apps gateway, so from the agent's retry perspective, it's treated the same way. However, without a Retry-After, the agent has no way of knowing when it will be lifted.

What caught my eye was the header handling. Claude Code's gateway protocol requires anthropic-beta to be forwarded verbatim as an open list, but LiteLLM reconstructs the beta header from the body's contents. For this Bedrock-bound request, effort-2025-11-24,interleaved-thinking-2025-05-14 was attached. Since experimental beta was disabled on the Claude Code side there was no actual harm, but this means that when Claude Code starts sending a new capability via header, values the Gateway doesn't know about will silently be dropped. Since cache_control and metadata.user_id were delivered, cache and attribution are preserved.

There was also a discrepancy in cost figures. For the same 1-turn call, Claude Code estimated $0.0887 while the LiteLLM spend log recorded $0.0976. Both are estimates based on list price tables, with the authoritative bill coming from the Bedrock side. Inserting a Gateway gives you "who spent how much," but at the cost of having multiple sources of estimates—worth keeping in mind.

Routing and Execution Side Live Outside the Gateway

Layer 3 routing and Layer 4 execution side are the layers I assigned to Switchyard and Relay in the previous article. Overlaying the billing path discussion makes it clear where they can be placed.

Switchyard's job is model selection. However, on the subscription path, there is no room to swap models inside Claude Code. The OAuth credential cannot be taken outside Claude Code, and pointing only ANTHROPIC_BASE_URL to a Gateway still applies subscription billing and usage limits. So routing only takes effect on pay-as-you-go paths and on the side of agents other than Claude Code. On August 11, 2026, NVIDIA repositioned Switchyard in a blog post as an open source model routing library, and as of August 29, the latest GitHub release remains v0.2.0. Relay 0.8 removed integration with the Switchyard server; it is scheduled to be distributed as a dynamic plugin in Switchyard 0.3.0. During this transitional period, it seems more practical to keep the routing layer outside the Gateway as a swappable component rather than fixing it inside.

Relay's job is logging and controlling agent execution. Placed on each developer's machine, it records tool executions and LLM calls as a single unit of work and sends them via OTLP to a central Collector. As seen in the previous measurements, Relay's PII redaction targets observation data and does not modify the body sent to the provider. If you want to scrub information before sending, you'd insert DLP into Relay's request middleware. What the Gateway sees at the entry point is "who used which model for how much," while "which files were read and what was executed" is only visible to Relay on the machine side.

If asked "does it seem like Switchyard and Relay have a role to play?"—the answer is yes. However, the main stage is not the entry point, but around it. Running Claude Code on a subscription while having Switchyard route other agents on pay-as-you-go paths, with Relay on each machine consolidating all agents' executions and sending them to a central location. Even taking P7's double management as a given, this placement makes both routing and the execution side effective across both paths.

Converging Observations and Financial Attribution in One Place

Layer 5 observations are the only converging layer in P7. Here is a breakdown of what can be obtained, in what units, and from where.

Claude Code OpenTelemetry can be enforced with priority over developer-side settings by distributing CLAUDE_CODE_ENABLE_TELEMETRY=1 and the OTLP endpoint via env in managed settings. Adding team.id or cost_center to OTEL_RESOURCE_ATTRIBUTES enables aggregation along financial dimensions. The official documentation also states that OpenTelemetry is the only way to stream per-user token and cost data to your own observability infrastructure in near real time.

On the Bedrock side, as of April 9, 2026, per-IAM-principal cost attribution became available in Cost Explorer and CUR 2.0. A method for splitting costs by department using application inference profiles and cost allocation tags was also introduced in the AWS Architecture blog on August 13. AWS Budgets' Budget Actions can automatically apply a Deny policy when a limit is reached, but Budget Actions are only available for monthly or longer budgets and cannot be configured on a daily basis. Day-level enforcement must be handled on the Gateway side.

The Ideal Form for August 2026 by Scale

Here are the components organized by scale. While described as an ideal form, it is drawn from parts that can be assembled as of August 29.

The company-wide form looks like the following diagram.

To summarize the ideal form in one sentence: a single gateway, replaceable routing and execution components, and converged observations. In the company-wide row, Claude Code is also placed on the pay-as-you-go route rather than a subscription, unifying the gateway. This closes the "budget centralization" gap in P7, but at the cost of subscription-only features and 1-hour cache. In terms of the numbers in the heavy user section, whether you can maintain a high cache hit rate in operation will determine the profit and loss of this choice.

Another consideration: the more you consolidate to a single gateway, the more that gateway itself becomes a single point of failure. A configuration that distributes ANTHROPIC_BASE_URL to everyone via managed settings is also one where everyone's work stops the moment the Gateway goes down. Making the Gateway itself stateless and placing multiple instances behind a load balancer, with HA for the Postgres holding spend data, is the foundation—but there are still three decisions to make. First, do not break the stream. Since Claude Code cuts streams after 300 seconds of silence, verify the idle timeout of your LB and proxy and keep-alive ping forwarding in advance. Second, the choice between fail-open and fail-closed for budget checks: Claude apps gateway defaults to fail-open, meaning it does not stop inference during DB failures but allows unmeasured spending. Third, whether to leave an escape route to bypass the Gateway and connect directly in emergencies. This is less about availability and more an organizational decision about how much to relax key control.

As an alternative to avoid a single point of failure, you could distribute individually configured Gateways to each user's environment rather than sharing one. Currently, we have settled on distributing a configuration under verification to each user's local Docker, with placement determined by state and keys. Local distribution fits well for routing, execution, and sending out observations. However, the gateway layer alone cannot be enforced if placed locally, because budget and model restrictions can be bypassed by editing one's own config files. To enforce the gateway with local distribution, you would issue per-user keys and move enforcement to the provider side—Console workspace spend limits for Console, and IAM principals plus Budgets for Bedrock. It seems best to view this as a trade-off between a shared gateway carrying a single point of failure versus shifting gateway enforcement to the provider and lightening the Gateway.

Personally, I think the 5–20 person row is the most practical, and P7 will likely remain the state for a while. Subscriptions from each vendor—one hour of cache for Claude, time-window limits for both—absorb heavy user demand well. There is little reason to break that to unify the gateway until audit requirements emerge. On the other hand, once everyone starts using agents and models not covered by any subscription, a shared Gateway becomes unavoidable, so it is worth setting that up in advance with observations converged first.

Summary

We explored an AI Gateway for teams using coding agents, divided into five layers: billing route, gateway, routing, execution, and observation.

The main conclusion is that the billing route chosen first determines everything else. Choose a subscription, and each vendor handles the gateway and budget management, absorbing heavy usage at a flat rate—but only their own agent can be onboarded. Choose pay-as-you-go, and any agent can be placed behind a single gateway—but you must operate the gateway yourself, cache lifetimes shrink, and bills tend to grow. Whether to sacrifice cost efficiency and multiply subscriptions per vendor, or consolidate into pay-as-you-go for ease of bundling—the reality of multiple management situations was the repetition of this binary choice for each vendor.

That the pay-as-you-go gateway actually works was confirmed with a single LiteLLM instance. It handles Claude Code, Codex CLI, and OpenCode with per-user keys, attribution of who spent how much, 429s on budget overrun, 403s on model restrictions, and maintaining 1-hour cache. At the same time, the cost of inserting an intermediary became visible—the Gateway rewriting headers, or cost estimates diverging from Claude Code.

The ideal form above that is: a single gateway, routing and execution as replaceable external components, and observations at a single convergence point. That said, I personally intend to continue a hybrid approach with first-party agents remaining on each vendor's subscription for the time being. Whether to truly consolidate to a single gateway will depend on whether cache hit rates can be maintained on pay-as-you-go, and how much per-request auditing is required.

I have not yet tested Claude apps gateway itself, or the behavior when using Anthropic API as the upstream. Once Switchyard 0.3.0's dynamic plugin is released, I would like to add the routing layer as a downstream component in this configuration and try it out.

Reference Links

• Run Claude Code through a gateway — Choosing between Claude apps gateway and third-party Gateways
• Gateway protocol reference — Headers and body the Gateway must forward, list of features that break
• Claude apps gateway / spend limits
• Manage costs effectively — Average cost per person and organization-level cost management
• Monitoring — OpenTelemetry metrics and attributes
• Enterprise deployment overview — Comparison table of 6 deployment routes
• Legal and compliance — OAuth and third-party tool handling
• What is the Team plan? / What is the Enterprise plan?
• Manage usage credits for Team and seat-based Enterprise plans
• Codex pricing — Codex included in ChatGPT plans and 5-hour/weekly limits
• Premium seats are coming to ChatGPT Business (announced 2026-08-10)
• Amazon Bedrock now supports cost allocation by IAM user and role (2026-04-09)
• Track generative AI costs with Amazon Bedrock inference profiles (2026-08-13)
• Introducing Claude apps gateway for AWS (2026-07-08)
• Claude Code Quickstart | LiteLLM / Security Update: Suspected Supply Chain Incident
• Microsoft cancels Claude Code licenses, shifting developers to GitHub Copilot CLI (Windows Central)
• NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI (2026-08-11)
• NeMo Relay Release Notes — Removal of Switchyard integration in 0.8 and preview of 0.3.0