Thinking About the Ideal AI Gateway and Router for Using Coding Agents in a Team

Introduction

Hi, I'm Mori from the Manufacturing Business Technology Department at Classmethod.

Token consumption by coding agents has ballooned over the past six months, and it's become a hot topic. An agent loops dozens of turns from a single instruction and rereads its context every turn, so it easily blows past budgets designed with chat-era intuition. Overseas, there was a report that a large company instructed its staff to switch tools because costs had reached $500–2,000 per person per month. Whichever agent you use, these amounts are no longer someone else's problem.

https://code.claude.com/docs/en/gateways

Individuals can somehow absorb this with the comprehensive subscription plans each vendor offers, which wrap a large amount of usage in a flat fee. But for a team or a whole company, the story changes. In practice, you probably juggle Team or Enterprise seats, the Console API, and pay-as-you-go billing through Amazon Bedrock or Claude Platform on AWS. Add agents other than Claude Code and the number of billing entry points grows further. Codex CLI runs on a ChatGPT subscription, so in my case I'm managing a Claude subscription, a ChatGPT subscription, and a pay-as-you-go entry point for the remaining tools. How to shrink this has been on my mind for a long time.

Last time, I organized how to place NeMo Relay and NeMo Switchyard as an AI Gateway, as a breakdown of product responsibilities (the article is current as of 2026-08-29). The team AI environment itself is covered in my open-weight model series (current as of 2026-08-08).

https://dev.classmethod.jp/articles/nemo-relay-switchyard-ai-gateway/

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

In this article, I split the AI Gateway into five layers (billing route, entry point, routing, execution side, and observability) and sketch the ideal setup for using coding agents in team development and business use as of August 2026. The only measurement is a single check that passes three kinds of agents through one shared Gateway. I hope it resonates with those who manage coding agents for a team or a whole company.

Thinking of the AI Gateway in Five Layers

In the previous article I wrote that the term AI Gateway covers a wide range of functions from authentication to auditing, and that preparing everything at once is too heavy for a normal environment. This time I start from one step earlier. Unless it's decided whose wallet pays, it's also not decided where to put the entry point.

Split into five layers, it looks like this.

Layers 2 through 5 largely overlap with the "fill missing features with dedicated components" table in the previous article. The difference is that I put the billing route at the top. As we'll see below, if you choose a subscription, the layer-2 entry point already exists on Anthropic's side and there's almost no room to insert your own. Conversely, if you choose pay-as-you-go, you need to own the entry point, and that's where a shared Gateway first comes into play. The number of layers is the same, but the choice of billing route moves where the other layers sit. That's the backbone of this article.

The Billing Route Decides Which Layers Are Needed

The official Claude Code documentation lists six deployment options. Putting billing, authentication, and what becomes unavailable in one table shows the differences in character.

The official docs say "Teams or Enterprise is best for most organizations." Indeed, with a subscription Anthropic owns the entry point, and admin settings let you set spend limits per organization or individual. Enterprise can also set per-group caps, with notifications at 75% and 90%. For Team, the Standard seat is $25/month, 1.25x Pro, and the Premium seat is $125/month, 6.25x; once you exceed the cap you buy extra usage credits at API rates. As of August 2026, the current Enterprise plan has changed to a $20 seat plus all Claude, Claude Code, and Cowork usage billed at API rates, with no per-seat usage cap.

Two points to note here. First, subscription limits are determined not by tokens but by 5-hour and weekly time windows. Even with the same seat, the usable amount changes depending on cache effectiveness and model mix. Second, the prompt cache lifetime: 1 hour for subscriptions, and 5 minutes for API keys, cloud providers, and while consuming usage credits. The likelihood of a cache miss when you return to a long interrupted session varies by route.

The authentication restriction connects directly to bundling multiple agents. In February 2026 Anthropic revised its Consumer Terms to prohibit using Free, Pro, and Max OAuth authentication from third-party harnesses. It was reported that actual blocking began on April 4. The official legal page also states plainly that third-party developers may not embed Claude.ai login in their applications or route requests on behalf of users with Pro / Max credentials. In other words, to use Claude from OpenCode, Pi, or Codex-family tools, you need an API key or cloud provider credentials.

This picture isn't limited to Anthropic. On the OpenAI side, Codex CLI runs on a ChatGPT subscription, with web, CLI, and IDE sharing 5-hour and weekly windows, and overage requires buying credits. For teams there's the Business plan (formerly Team), and on August 10, 2026 a Premium seat was announced with 5x the Standard allowance and no 5-hour cap. It's $125 monthly, $100 annually, the same price tag as Claude's Team Premium. Each vendor is converging on a two-tier approach: its own first-party agent rides on its own subscription, while third-party tools and other vendors' models are paid via API pay-as-you-go. The number of entry points grows by the number of vendors whose agents you use, plus one shared pay-as-you-go.

This is ultimately why I can't escape multi-management. A subscription is an entry point each vendor prepared for its own agent, so while it runs cheaply, there's no way to consolidate multiple vendors' subscriptions into one entry point. To consolidate, you'd cancel the subscriptions, move all agents onto API pay-as-you-go, and receive them at your own entry point. But then the token consumption that the flat fee had been absorbing lines up directly on the invoice. In Claude's case the prompt cache lifetime also shrinks from 1 hour to 5 minutes, so the cache works worse for the same usage and the bill swells further. I'd like to decide which to choose after seeing the numbers in the next chapter.

Applying Billing Routes to a Heavy User's Real Data

This is my Claude Code usage for the last 30 days (2026-07-31 to 08-29) aggregated with ccusage; please read it as an example of one heavy user in the team. The table includes only Claude Code consumption. I also use Codex CLI on a ChatGPT Business (formerly Team) plan and, depending on the project, open-weight models on Fireworks AI pay-as-you-go, but those are separate entry points and are not in these numbers.

98% of the total is cache read. In an earlier tally, a single human instruction spun an agent loop of 21 turns on average. Every turn rereads the whole context, so the total token count reaches this order of magnitude. The flat plan absorbs this structure as is, working out to 1/40 of the API list price.

Placing it next to published averages shows where I stand. Anthropic's cost management page puts the enterprise deployment average at $13 per person per working day and $150–250 per month, with 90% of users under $30 per working day. The report I mentioned at the start was $500–2,000 per month. My 30 days is $7,879 at list price, over 30x the average and about 4x the upper end of the report.

Applying billing routes to these numbers, the cache hit rate decides the conclusion. If moving to a pay-as-you-go route keeps the current nearly-98% hit rate, the monthly cost is dominated by the cache read price and stays relatively mild. But under a pricing scheme where cache read is around a tenth of the input price, by my estimate the pay-as-you-go equivalent jumps to about 1.8x at a 90% hit rate and about 4.9x at 50%. This is where the difference between the subscription's 1-hour cache and the API's 5-minute cache matters: for the same work, just changing the route can shift the bill by an order of magnitude.

Personally, I think this sensitivity is what "subscriptions are cheap" really means. As we'll see in the measurement chapter later, Claude Code writes 33,359 tokens to the cache even for a call that just returns the single word "connected." The tool definitions and system prompt are that long, and subsequent turns reread them. Keeping in mind that an agent's bill is determined not by the amount of text generated but by the amount of context reread changes how the configuration patterns in the next chapter look.

Where to Put the Entry Point: Seven Configuration Patterns

I list seven combinations of the layer-2 entry point with billing routes. The columns are billing, unit of attribution, how budgets are enforced, usable agents, and what you lose.

The Claude apps gateway in P3 is Anthropic's official self-hosted Gateway, released on June 29, 2026. It's bundled in the claude binary, starts with claude gateway --config gateway.yaml, signs users in via an OIDC IdP, and bundles Bedrock, Claude Platform on AWS, Agent Platform, Foundry, and the Anthropic API as upstreams with failover. Spend limits can be set via the Admin API per user, group, or organization on a daily, weekly, or monthly basis; when exceeded it returns a 429 billing_error, and Claude Code shows warnings at 75% and 95%. On the other hand, there are no service tokens for CI, WebSearch is disabled, and the 1-hour cache TTL is unavailable. And only Claude Code and Claude Desktop can use it. This is my summary from reading the official docs; I haven't measured it this time.

The shared Gateways in P2 and P4 are typified by LiteLLM and Bifrost. LiteLLM lets a virtual key carry a budget, period, and model restrictions, and it also has a budget fallback that drops to a cheaper model when the budget runs out. However, on March 24, 2026 there was a supply chain incident in which malicious versions 1.82.7 and 1.82.8 were published to PyPI, so if you adopt it, pin the version. Bifrost is a single Go binary depending only on Postgres, with few dependencies as its strength. P6 is the path for organizations already running an API Gateway to add AI plugins, as with Envoy AI Gateway or Kong; Envoy added in v0.4.0 a translator that converts Anthropic Messages to Bedrock Converse.

My current setup is P7. As a diagram, there are as many entry points as subscriptions, with only observability merging.

Looking at the table, you notice that no row has both the "all agents can use it" column and the "budget works in one place" column. P2 and P4 take all agents but give up Claude Code's subscription features; P1 and P3 have a budget that works in one place but only Claude Code can use them. P7 takes all agents, but budget management multiplies with the number of subscriptions. Thinking about the ideal form seems to come down to how to fill in that one given-up column.

Connecting Three Kinds of Agents to a Shared Gateway

Now the main part. I actually received the "all agents" side of P2 and P4 on a single machine. On my usual machine I stood up LiteLLM v1.98.0 and Postgres, issued four virtual keys, one per user, and connected three kinds of agents. Upstreams are Bedrock Sonnet 5 for Claude Code, OpenAI gpt-5.6-terra for Codex CLI, and Fireworks AI DeepSeek V4 Flash for OpenCode. I did not use an Anthropic Console key or a claude.ai login. Codex CLI normally also runs on a ChatGPT subscription, but here the goal was to put it on a shared entry point, so I received it via the API route.

The results of the checks are as follows.

Connectivity, attribution, budget, and model restriction worked as expected. A budget overrun returning 429 is the same status as Claude apps gateway, so it is handled the same way from the agent's retry perspective. However, there's no Retry-After, so the agent can't know when the block will be lifted.

What stood out was header handling. Claude Code's gateway protocol asks that anthropic-beta be forwarded verbatim as an open list, but LiteLLM rebuilds the beta header from the body content. For this Bedrock call, effort-2025-11-24,interleaved-thinking-2025-05-14 was attached. Since I had disabled experimental betas on the Claude Code side there was no actual harm, but when Claude Code starts sending a new capability via a header, values the Gateway doesn't know are silently dropped. cache_control and metadata.user_id arrived, so cache and attribution are preserved.

There's one more point: two cost figures appeared. For the same 1-turn call, Claude Code estimated $0.0887 while LiteLLM's spend log recorded $0.0976. Both are estimates from a price list, and the source of truth for billing is Bedrock. Adding a Gateway gives you "who spent how much," but it also adds sources of estimates, which is worth remembering.

Put Routing and the Execution Side Outside the Gateway

Layer 3 (routing) and layer 4 (execution side) are the layers I assigned to Switchyard and Relay in the previous article. Overlaying this article's billing route discussion makes the places they can go clear.

Switchyard's job is model selection. But on the subscription route, there's no room to swap models inside Claude Code. The OAuth credential can't leave Claude Code, and even if you point only ANTHROPIC_BASE_URL at a Gateway, subscription billing and usage limits still apply. So routing is effective on the pay-as-you-go route and on the side of agents other than Claude Code. In its August 11, 2026 blog, NVIDIA repositioned Switchyard as an open source model routing library, and on GitHub the latest release as of August 29 is still v0.2.0. Relay 0.8 removes the Switchyard server integration, and it is planned to be distributed as a dynamic plugin in Switchyard 0.3.0. Rather than fixing the routing layer inside the Gateway, keeping it outside as a replaceable component seems more convenient in this transition period.

Relay's job is recording and controlling agent execution. Placed on each developer's machine, it records tool execution and LLM calls as one unit of work and sends them to a central Collector via OTLP. As seen in the previous measurement, Relay's PII redaction targets observability data and does not modify the body sent to the provider. If you want to remove information before sending, you'd insert DLP into Relay's request middleware. What the Gateway sees at the entry point is "who spent how much on which model"; "which files were read and what was executed" is visible only to Relay on the machine.

If asked "is there a place for Switchyard and Relay to shine?", I can say yes. But the main stage isn't the entry point; it's before and after it. Run Claude Code on the subscription, let Switchyard route on the other agents' routes, and have Relay on the machine align all agents' execution and send it centrally. Even assuming P7's dual management, routing and the execution side work on both routes with this placement.

Merge Observability and Financial Attribution in One Place

Layer 5, observability, is the only layer that merges in P7. Here's what can be obtained, in which unit, and from where.

Claude Code's OpenTelemetry can be enforced, taking priority over developer-side settings, by distributing CLAUDE_CODE_ENABLE_TELEMETRY=1 and an OTLP endpoint via env in managed settings. Adding team.id or cost_center to OTEL_RESOURCE_ATTRIBUTES lets you aggregate along finance axes. The official docs also say OpenTelemetry is the only way to stream per-user token and cost into your own observability stack in near real time.

On the Bedrock side, since April 9, 2026, cost allocation per IAM principal has been available in Cost Explorer and CUR 2.0. A method of splitting by department using application inference profiles and cost allocation tags was also described in the August 13 AWS Architecture blog. There's also the use of AWS Budgets Budget Actions to automatically apply a Deny policy when a limit is reached, but Budget Actions work only for monthly or longer budgets, not daily ones. Daily cutoffs have to be handled on the Gateway side.

The Ideal Setup for August 2026, by Scale

I lay out the materials so far by scale. Even as an ideal, it's a form drawn from components available on August 29.

The company-wide form looks like this as a diagram.

In one sentence, the ideal is: one entry point, replaceable routing and execution side, merged observability. In the company-wide row, Claude Code also moves onto the pay-as-you-go route instead of the subscription, giving a single entry point. This fills P7's "unified budget" hole, but in exchange you give up subscription-only features and the 1-hour cache. In terms of the heavy user chapter's numbers, whether you can operate in a way that keeps the cache hit rate high decides the profit and loss of this choice.

Another point: the more you gather entry points into one, the more the entry point itself becomes a single point of failure. A setup that distributes ANTHROPIC_BASE_URL to everyone via managed settings is also one where everyone's hands stop the moment the Gateway goes down. The foundation is to make the Gateway itself stateless, run several behind a load balancer, and make the Postgres holding spend HA, but there are still three decisions to make. First, don't break streams: Claude Code cuts a stream after 300 seconds of silence, so check the LB and proxy idle timeouts and keep-alive ping forwarding first. Second, fail-open versus fail-closed for budget checks; Claude apps gateway defaults to fail-open, meaning inference isn't stopped during a DB outage, at the price of allowing unmetered spend. Third, whether to leave an escape route that bypasses the Gateway to connect directly in an emergency. This is less about availability than an organizational decision about how far to loosen key control.

As a way to avoid a single point of failure, one could also distribute a consistently configured setup to each user's environment instead of sharing a Gateway. I'm currently testing distributing the environment to each person's local Docker, and I've settled on deciding where to put things by state and keys. Local distribution fits well for routing, the execution side, and sending out observability. The entry point alone, however, isn't enforced if it's local, because budgets and model restrictions can be removed by rewriting your own config file. To make the entry point work with local distribution, you'd issue keys per user and move enforcement to the provider side: workspace spend limits for Console, IAM principals and Budgets for Bedrock. It seems best to view it as a trade-off between a shared entry point carrying a single point of failure and shifting enforcement to the provider to keep the Gateway light.

Personally, I think the 5–20 person row is the most realistic and that we'll probably stay with P7 for a while. Each vendor's subscription absorbs the heavy user's share well, with Claude's 1-hour cache and time-window caps on both. The reason to break that to unify the entry point is thin until audit requirements appear. On the other hand, once everyone starts using agents or models that fit no subscription, a shared Gateway is unavoidable, so I think the order is to stand that up first and merge just the observability ahead of time.

Summary

I sketched an AI Gateway for using coding agents in a team, split into five layers: billing route, entry point, routing, execution side, and observability.

The main conclusion is that the billing route chosen first decides everything else. If you choose a subscription, each vendor owns the entry point and budget management, and the flat fee absorbs heavy usage. In exchange, only that company's agent fits. If you choose pay-as-you-go, any agent can go through one entry point. In exchange, you operate the entry point yourself, and the cache lifetime shortens, so the bill tends to swell. Increase subscriptions per vendor for cheapness, or move to pay-as-you-go for bundling ease? The reality of multi-management was repeating this two-way choice for each vendor.

That a pay-as-you-go entry point actually works was confirmed with a single LiteLLM. Taking Claude Code, Codex CLI, and OpenCode with per-user keys, it handles attribution of who spent how much, 429 on budget overrun, 403 on model restriction, and even 1-hour cache retention. On the other hand, I also saw the costs of inserting one more layer in between, such as the Gateway rebuilding headers and its cost estimates disagreeing with Claude Code's.

The ideal form on that basis is one entry point, routing and execution side outside as replaceable components, and observability at a single merge point. Still, for the time being I intend to continue the hybrid, keeping first-party agents on each vendor's subscription. Whether to really consolidate to one entry point depends on whether we can keep the cache hit rate under pay-as-you-go, and how much per-request auditing is needed.

I haven't yet tried Claude apps gateway itself, nor behavior with the Anthropic API as upstream. Once Switchyard 0.3.0's dynamic plugin is out, I'd like to add the routing layer behind this setup and try it.

Reference Links

• Run Claude Code through a gateway — How to choose between Claude apps gateway and other Gateways
• Gateway protocol reference — Headers and body a Gateway must forward, and list of features that break
• Claude apps gateway / spend limits
• Manage costs effectively — Average cost per person and organizational cost management
• Monitoring — OpenTelemetry metrics and attributes
• Enterprise deployment overview — Comparison table of the six deployment routes
• Legal and compliance — Handling of OAuth and third-party tools
• What is the Team plan? / What is the Enterprise plan?
• Manage usage credits for Team and seat-based Enterprise plans
• Codex pricing — Codex included in ChatGPT plans and the 5-hour / weekly windows
• Premium seats are coming to ChatGPT Business (announced 2026-08-10)
• Amazon Bedrock now supports cost allocation by IAM user and role (2026-04-09)
• Track generative AI costs with Amazon Bedrock inference profiles (2026-08-13)
• Introducing Claude apps gateway for AWS (2026-07-08)
• Claude Code Quickstart | LiteLLM / Security Update: Suspected Supply Chain Incident
• Microsoft cancels Claude Code licenses, shifting developers to GitHub Copilot CLI (Windows Central)
• NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI (2026-08-11)
• NeMo Relay Release Notes — Removal of Switchyard integration in 0.8 and the 0.3.0 announcement