I tried deploying Nemotron 9B-v2-Japanese to SageMaker Tokyo Region with a VPC closed network configuration

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article, I ran NVIDIA's official Japanese-enhanced LLM Nemotron 9B-v2-Japanese on DGX Spark and tested various use cases including Tool Calling and RAG.

Local verification on DGX Spark is convenient, but when it comes to "how to run it in production," things get more complex. In cases involving personal information or confidential data, closed network configurations that keep data within Japan are often required.

In this article, I tried deploying Nemotron 9B-v2-Japanese to Amazon SageMaker in the Tokyo region and getting it running in a closed VPC configuration. Since I'm comparing with the same benchmark (JCommonsenseQA) as last time, you'll be able to see the differences between local and cloud environments.

Before Running on SageMaker

Using vLLM as the Inference Engine

In my previous article, I confirmed that the NGC container version of vLLM (v0.13.0) works on DGX Spark, but on SageMaker, vLLM is used as a backend via the LMI container.

I specify vLLM as the backend via SageMaker's LMI (Large Model Inference) container. Since LMI containers V17 (September 2025) and later, async mode integrating vLLM's AsyncLLMEngine has become the default. This time, I'm starting in async mode with OPTION_ROLLING_BATCH=disable and specifying a chat template via OPTION_CHAT_TEMPLATE.

JumpStart / Marketplace Status

As of February 2026, Nemotron 9B-v2-Japanese is not registered in SageMaker JumpStart. The English version Nemotron nano 9b v2 is registered on AWS Marketplace, but the Japanese version is not. I'll configure and deploy the LMI container myself.

Deployment Configuration

Solid lines represent inference request flow, dashed lines represent model loading at deploy time. All communication is contained within the VPC.

• The model is downloaded from HuggingFace Hub to S3 in advance. No internet access from the endpoint is required
• Calls to SageMaker Runtime go through VPC endpoints. Data does not leave the VPC
• The inference engine is vLLM inside the LMI container. It fits within A10G (24GB) at full BF16 precision

The reason I chose ml.g5.2xlarge is that BF16 (17.8GB) + vLLM overhead (~1GB) + KV cache (~2-3GB for 4 Attention layers) totals approximately 21GB, which fits within the A10G's 24GB. With the Mamba-2 hybrid architecture, only 4 of 56 layers are Attention layers, so the KV cache is approximately 1/14 of what full-Attention would require.

Deployment Steps

Placing the Model in S3

In a closed network configuration, the endpoint cannot access HuggingFace Hub. Download the model in advance and upload it to S3.

This is the only step that requires an internet connection. The execution environment needs AWS CLI (v2.32.0 or later) and HuggingFace CLI (hf). hf can be installed with brew install huggingface-cli, or if you have uv, you can use uvx hf without installation. Please ensure at least 20GB of free disk space (model size is approximately 18GB). This can be run from anywhere—your local PC or an EC2 instance—as long as the requirements are met.

For AWS CLI authentication, aws login is convenient. A browser opens and you can sign in the same way as the management console, with temporary credentials automatically rotated. No login is required for hf since it's downloading a public model.

# Download model from HuggingFace (approximately 18GB)
hf download nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
  --local-dir /tmp/nemotron-9b-japanese

# Upload to S3 (bucket name is arbitrary, align with subsequent steps)
aws s3 cp /tmp/nemotron-9b-japanese \
  s3://<your-bucket>/models/nemotron-9b-v2-japanese/ \
  --recursive

The S3 bucket name is arbitrary, but using SageMaker's default bucket (sagemaker-{region}-{account-id}) makes it easier to align with subsequent steps. Calling sagemaker.Session().default_bucket() will auto-create it, so you don't need to create the bucket in advance.

from huggingface_hub import snapshot_download
from sagemaker.s3 import S3Uploader
import sagemaker

sess = sagemaker.Session()
bucket = sess.default_bucket()

model_dir = snapshot_download(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    local_dir="/tmp/nemotron-9b-japanese"
)

s3_model_uri = S3Uploader.upload(
    local_path=model_dir,
    desired_s3_uri=f"s3://{bucket}/models/nemotron-9b-v2-japanese/"
)
print(f"Model uploaded to: {s3_model_uri}")

All subsequent steps are contained within the VPC.

Creating VPC Endpoints

To run SageMaker endpoints in a closed network configuration, create interface endpoints within the VPC to confine communication to AWS services within the VPC.

Six VPC endpoints are required.

The top 5 are required for deployment, and adding CloudWatch Logs gives peace of mind for debugging. In a fully closed VPC, container logs won't reach CloudWatch, making it impossible to trace the cause of deployment failures. S3 is Gateway type (free), others are Interface type (ENI billing applies).

For Interface type endpoints, allow HTTPS (443) from the VPC CIDR in the security group inbound rules, and enable private DNS. With private DNS enabled, you can communicate via the closed network without changing SDK code.

Creating a SageMaker Model

From the SageMaker console in AWS, create a model. Specify the vLLM backend for the LMI container and pass Nemotron-specific settings as environment variables.

There are two easily overlooked environment variables.

OPTION_TRUST_REMOTE_CODE=true is required to load the custom model code for the Mamba-2 hybrid architecture—without it, model loading itself will fail.

OPTION_MAMBA_SSM_CACHE_DTYPE=float32 specifies the precision of the Mamba-2 layer state cache. If omitted, inference result precision degrades. Since it doesn't cause an error, it's easy to miss—make sure to set it.

The environment variables are summarized below.

OPTION_ROLLING_BATCH=disable disables LMI's rolling batch mode and starts in async mode (default since V17). Async mode uses vLLM's AsyncLLMEngine directly, offering improved performance and stability over rolling batch mode. OPTION_CHAT_TEMPLATE is the path to the chat template bundled with the model, which is automatically applied to prompts during inference requests.

import sagemaker
import boto3

role = sagemaker.get_execution_role()
sess = sagemaker.Session()
region = "ap-northeast-1"

# LMI container URI (latest as of 2026-02: LMI V20 / DJL 0.36.0)
container_uri = sagemaker.image_uris.retrieve(
    framework="djl-lmi",
    version="0.36.0",
    region=region
)

env = {
    "HF_MODEL_ID": "/opt/ml/model",
    "OPTION_ROLLING_BATCH": "disable",
    "OPTION_DTYPE": "bf16",
    "OPTION_TRUST_REMOTE_CODE": "true",
    "OPTION_MAMBA_SSM_CACHE_DTYPE": "float32",
    "OPTION_TENSOR_PARALLEL_DEGREE": "1",
    "OPTION_MAX_MODEL_LEN": "4096",
    "OPTION_CHAT_TEMPLATE": "/opt/ml/model/chat_template.jinja",
}

# VPC information (example using default VPC)
ec2 = boto3.client("ec2", region_name=region)
vpcs = ec2.describe_vpcs(Filters=[{"Name": "isDefault", "Values": ["true"]}])
vpc_id = vpcs["Vpcs"][0]["VpcId"]
subnets = ec2.describe_subnets(Filters=[{"Name": "vpc-id", "Values": [vpc_id]}])
subnet_ids = [s["SubnetId"] for s in subnets["Subnets"]]
sgs = ec2.describe_security_groups(
    Filters=[{"Name": "vpc-id", "Values": [vpc_id]}, {"Name": "group-name", "Values": ["default"]}]
)
sg_id = sgs["SecurityGroups"][0]["GroupId"]

model = sagemaker.Model(
    image_uri=container_uri,
    model_data=f"s3://{sess.default_bucket()}/models/nemotron-9b-v2-japanese/",
    env=env,
    role=role,
    vpc_config={"Subnets": subnet_ids, "SecurityGroupIds": [sg_id]},
)

predictor = model.deploy(
    instance_type="ml.g5.2xlarge",
    initial_instance_count=1,
    container_startup_health_check_timeout=900,
    model_data_download_timeout=1800,
)

Creating an Endpoint

Once the model is created, create an endpoint configuration and deploy the endpoint.

I recommend changing two timeout settings. Extend container_startup_health_check_timeout to 900 seconds (default: 300 seconds). Since loading the 18GB model and initializing vLLM takes time, the default may time out. Similarly, extending model_data_download_timeout to 1800 seconds is a safe practice.

Confirming Deployment Completion

When the endpoint status becomes InService, deployment is complete. In my environment, it took approximately 11 minutes (677 seconds) including model download from S3 and vLLM initialization.

As an operation check, let's send a simple prompt.

import json
import boto3

sm_runtime = boto3.client("sagemaker-runtime", region_name="ap-northeast-1")

payload = {
    "inputs": "What is the height of Tokyo Tower?",
    "parameters": {
        "max_new_tokens": 128,
        "temperature": 0,
    }
}

response = sm_runtime.invoke_endpoint(
    EndpointName="nemotron-9b-v2-japanese",
    ContentType="application/json",
    Body=json.dumps(payload),
)
result = json.loads(response["Body"].read().decode("utf-8"))
print(result["generated_text"])

If a response is returned normally, deployment in the closed network configuration is complete.

Pitfalls Encountered

There were several pitfalls during the deployment and operation verification process. Among them, the chat template special token issue was an important discovery directly affecting benchmark accuracy.

Chat Template Special Token Issue

This was the biggest discovery in this verification.

Nemotron 9B-v2-Japanese's chat template uses special tokens for role separators between System / User / Assistant. You often see the notation <extra_id_0> <extra_id_1> <extra_id_2> in README files and blog posts on HuggingFace Hub, but these are actually not the correct special tokens.

Checking the model's tokenizer_config.json, the actual role separator tokens are <SPECIAL_10> (ID: 10), <SPECIAL_11> (ID: 11), and <SPECIAL_12> (ID: 12). When the string <extra_id_0> is passed to the tokenizer, it is not recognized as a single special token and is split into 6 subwords: <, extra, _id, _, 0, >.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    trust_remote_code=True
)

# Wrong: split into 6 subwords excluding BOS
tokenizer.encode("<extra_id_0>")
# → [1, 523, 15683, 1620, 290, 29900, 1572]  (BOS + 6 subwords)

# Correct: single special token
tokenizer.encode("<SPECIAL_10>")
# → [1, 10]  (BOS + special token)

Looking at the entire template, the <extra_id_X> method introduces 18 extra unnecessary tokens just for role separators. Since these are meaningless noise to the model, they affect inference accuracy.

When I actually compared them in benchmarks, the <extra_id_X> method achieved an accuracy of 83.3%, while using the correct <SPECIAL_X> tokens improved it to 84.8%. The 1.5pp difference may seem small, but for specific categories (questions where the answer is A), there was a significant improvement from 66.2% → 75.0%.

The reliable countermeasure is to apply the template locally using HuggingFace's tokenizer.apply_chat_template() and send the rendered text as inputs.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    trust_remote_code=True
)

messages = [{"role": "user", "content": "What is the height of Tokyo Tower?"}]
formatted = tokenizer.apply_chat_template(
    messages, tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

# formatted correctly contains <SPECIAL_10>, <SPECIAL_11>, etc.
payload = {
    "inputs": formatted,
    "parameters": {"max_new_tokens": 128, "temperature": 0},
}

LMI Messages API Limitations

The LMI container's async mode also supports the OpenAI-compatible messages format, but it didn't work correctly with Nemotron 9B-v2-Japanese. All requests show prompt_tokens: 8, and the same response is returned regardless of the prompt content.

It's likely that the <SPECIAL_X> tokens in the chat template are not being correctly interpreted by the LMI-side tokenizer processing. As mentioned above, applying the template locally and sending in inputs format works without issues.

S3 Model Specification Method

When using the sagemaker-core v3 SDK (sagemaker.core.resources), be careful with S3ModelDataSource settings. Specify "S3Prefix" for s3_data_type and explicitly set compression_type to "None". Omitting compression_type defaults to "Gzip", which causes an error when it tries to decompress uncompressed model files.

Benchmark Comparison

JCommonsenseQA

Measured under the same conditions as the previous DGX Spark verification.

Between DGX Spark (NGC container) and SageMaker (LMI container), even with different vLLM versions and GPUs, the accuracy difference is contained within 1.3pp. This shows that as long as the same inference engine is used, accuracy remains nearly consistent even when hardware changes.

Cost Comparison

DGX Spark, once purchased, can continue to be used with only electricity costs, making it overwhelmingly cost-effective during development and experimentation phases. On the other hand, SageMaker is suited for production environments with low usage frequency or batch processing use cases, since billing stops when you delete the endpoint when not in use.

Don't forget to also factor in VPC endpoint (Interface type) costs. In the Tokyo region, each ENI costs $0.014/h, which accumulates based on the number of AZs × number of endpoints. Deleting VPC endpoints when not in use helps save costs.

Cleanup

When verification is complete, don't forget to delete the endpoint. At $2.197/h for ml.g5.2xlarge, leaving it running for a day costs approximately $53.

There are 3 items to delete. Delete them in reverse dependency order: endpoint → endpoint configuration → model.

import boto3

sm = boto3.client("sagemaker", region_name="ap-northeast-1")

# 1. Delete endpoint
sm.delete_endpoint(EndpointName="nemotron-9b-v2-japanese")

# 2. Delete endpoint configuration
sm.delete_endpoint_config(EndpointConfigName="nemotron-9b-v2-japanese")

# 3. Delete model
sm.delete_model(ModelName="nemotron-9b-v2-japanese")

If VPC endpoints are no longer needed, delete them as well. You can select the 6 created this time from the endpoints list in the VPC console and delete them.

Summary

Deploying Nemotron 9B-v2-Japanese to SageMaker's Tokyo region in a closed network configuration itself went smoothly. Once the 6 VPC endpoints are configured, you can achieve a setup where data never leaves the VPC within Japan. Deployment time is approximately 11 minutes, and inference latency at 0.30 seconds per question is at a practical level.

The biggest takeaway from this verification is that precision remains nearly consistent with vLLM even when hardware changes. Between DGX Spark (NGC vLLM, 83.5%) and SageMaker (LMI vLLM, 84.8%), the difference was only 1.3pp despite different versions and GPUs. Knowing that accuracy confirmed locally can be reproduced in the cloud when using the same inference engine is reassuring when selecting a deployment target.

Reference Links

• NVIDIA-Nemotron-Nano-9B-v2-Japanese (HuggingFace)
• Previous article: Trying Nemotron 9B-v2-Japanese in Various Use Cases
• SageMaker LMI Deployment Guide (DJL)
• SageMaker VPC Endpoints (AWS Docs)
• LMI Container Release Notes (DJL)
• vLLM Engine User Guide (DJL)
• LMI Async Mode Migration Guide (DJL)