I tried deploying Nemotron 9B-v2-Japanese to SageMaker Tokyo Region with a VPC closed network configuration

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article, I ran NVIDIA's officially enhanced Japanese LLM Nemotron 9B-v2-Japanese on DGX Spark and tested it in various scenarios including Tool Calling and RAG.

Local verification on DGX Spark is convenient, but things change when it comes to "how to run it in a production environment." In cases involving personal information or confidential data, a closed-network configuration where data never leaves Japan is often required.

In this article, I tried deploying Nemotron 9B-v2-Japanese to Amazon SageMaker in the Tokyo region and getting it running in a closed-network configuration within a VPC. Since I'm comparing with the same benchmark (JCommonsenseQA) as last time, you should be able to see the differences between local and cloud environments.

Before Running on SageMaker

Using vLLM as the Inference Engine

In my previous article, I confirmed that the NGC container version of vLLM (v0.13.0) works on DGX Spark as well, but on SageMaker, vLLM is used as a backend via the LMI container.

You specify vLLM as the backend through SageMaker's LMI (Large Model Inference) container. Since LMI containers V17 (September 2025) and later, async mode integrating vLLM's AsyncLLMEngine has become the default. This time, I'm launching in async mode with a configuration that specifies the chat template using OPTION_ROLLING_BATCH=disable and OPTION_CHAT_TEMPLATE.

JumpStart / Marketplace Status

As of February 2026, Nemotron 9B-v2-Japanese is not registered in SageMaker JumpStart. The English version Nemotron nano 9b v2 is registered on AWS Marketplace, but the Japanese version is not. You'll need to configure and deploy using the LMI container yourself.

Deployment Configuration

Solid lines represent the inference request flow, and dashed lines represent model loading at deployment time. All communications are completed within the VPC.

• The model is downloaded from HuggingFace Hub to S3 in advance. No internet access from the endpoint is required.
• Calls to SageMaker Runtime go through VPC endpoints. Data never leaves the VPC.
• The inference engine is vLLM inside the LMI container. It fits within the A10G (24GB) at BF16 full precision.

The reason I chose ml.g5.2xlarge is that the total of BF16 (17.8GB) + vLLM overhead (~1GB) + KV cache (~2-3GB for 4 Attention layers) comes to approximately 21GB, which fits within the A10G's 24GB. With the Mamba-2 hybrid architecture, only 4 out of 56 layers are Attention layers, so the KV cache is about 1/14 of what it would be with all-Attention layers.

Deployment Steps

Placing the Model in S3

In a closed-network configuration, the endpoint cannot access HuggingFace Hub. You need to download the model in advance and upload it to S3.

Only this step requires an internet connection. The execution environment requires AWS CLI (v2.32.0 or later) and HuggingFace CLI (hf). hf can be installed with brew install huggingface-cli, or if you have uv, you can use it without installation via uvx hf. Make sure you have at least 20GB of free disk space (model size is approximately 18GB). This can be run from anywhere that meets the conditions, whether your local PC or EC2.

For AWS CLI authentication, aws login is convenient. It opens a browser where you can sign in the same way as the management console, and temporary credentials are automatically rotated. No login is required for hf since the model is public.

# Download model from HuggingFace (approximately 18GB)
hf download nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
  --local-dir /tmp/nemotron-9b-japanese

# Upload to S3 (bucket name is arbitrary, match it with subsequent steps)
aws s3 cp /tmp/nemotron-9b-japanese \
  s3://<your-bucket>/models/nemotron-9b-v2-japanese/ \
  --recursive

The S3 bucket name is arbitrary, but using SageMaker's default bucket (sagemaker-{region}-{account-id}) makes it easier to align with subsequent steps. Calling sagemaker.Session().default_bucket() will automatically create it, so you don't need to create the bucket in advance.

from huggingface_hub import snapshot_download
from sagemaker.s3 import S3Uploader
import sagemaker

sess = sagemaker.Session()
bucket = sess.default_bucket()

model_dir = snapshot_download(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    local_dir="/tmp/nemotron-9b-japanese"
)

s3_model_uri = S3Uploader.upload(
    local_path=model_dir,
    desired_s3_uri=f"s3://{bucket}/models/nemotron-9b-v2-japanese/"
)
print(f"Model uploaded to: {s3_model_uri}")

All subsequent steps are completed within the VPC.

Creating VPC Endpoints

To run the SageMaker endpoint in a closed-network configuration, you create interface endpoints within the VPC to confine communications to AWS services within the VPC.

Six VPC endpoints are required.

The top 5 are mandatory for deployment, and adding CloudWatch Logs for debugging gives peace of mind. In a completely closed VPC, container logs won't reach CloudWatch, making it impossible to trace the cause of deployment failures. S3 is a Gateway type (free), while the others are Interface types (ENI charges apply).

For Interface type endpoints, allow HTTPS (443) from the VPC CIDR in the security group inbound rules and enable private DNS. With private DNS enabled, you can communicate through the closed network without changing the SDK code.

Creating a SageMaker Model

From the SageMaker section of the AWS console, create a model. Specify the vLLM backend for the LMI container and pass Nemotron-specific settings via environment variables.

There are two environment variables that are easy to overlook.

OPTION_TRUST_REMOTE_CODE=true is required to load the custom model code for the Mamba-2 hybrid architecture. Without it, model loading itself will fail.

OPTION_MAMBA_SSM_CACHE_DTYPE=float32 specifies the precision of the Mamba-2 layer's state cache. Omitting it will degrade inference result precision. Since it won't cause an error, it's easy to miss, so be sure to set it.

The environment variables are summarized below.

OPTION_ROLLING_BATCH=disable disables LMI's rolling batch mode, launching in async mode (default since V17). Async mode uses vLLM's AsyncLLMEngine directly, improving performance and stability over rolling batch mode. OPTION_CHAT_TEMPLATE is the path to the chat template bundled with the model, which is automatically applied to prompts during inference requests.

import sagemaker
import boto3

role = sagemaker.get_execution_role()
sess = sagemaker.Session()
region = "ap-northeast-1"

# LMI container URI (latest as of 2026-02: LMI V20 / DJL 0.36.0)
container_uri = sagemaker.image_uris.retrieve(
    framework="djl-lmi",
    version="0.36.0",
    region=region
)

env = {
    "HF_MODEL_ID": "/opt/ml/model",
    "OPTION_ROLLING_BATCH": "disable",
    "OPTION_DTYPE": "bf16",
    "OPTION_TRUST_REMOTE_CODE": "true",
    "OPTION_MAMBA_SSM_CACHE_DTYPE": "float32",
    "OPTION_TENSOR_PARALLEL_DEGREE": "1",
    "OPTION_MAX_MODEL_LEN": "4096",
    "OPTION_CHAT_TEMPLATE": "/opt/ml/model/chat_template.jinja",
}

# VPC information (example using default VPC)
ec2 = boto3.client("ec2", region_name=region)
vpcs = ec2.describe_vpcs(Filters=[{"Name": "isDefault", "Values": ["true"]}])
vpc_id = vpcs["Vpcs"][0]["VpcId"]
subnets = ec2.describe_subnets(Filters=[{"Name": "vpc-id", "Values": [vpc_id]}])
subnet_ids = [s["SubnetId"] for s in subnets["Subnets"]]
sgs = ec2.describe_security_groups(
    Filters=[{"Name": "vpc-id", "Values": [vpc_id]}, {"Name": "group-name", "Values": ["default"]}]
)
sg_id = sgs["SecurityGroups"][0]["GroupId"]

model = sagemaker.Model(
    image_uri=container_uri,
    model_data=f"s3://{sess.default_bucket()}/models/nemotron-9b-v2-japanese/",
    env=env,
    role=role,
    vpc_config={"Subnets": subnet_ids, "SecurityGroupIds": [sg_id]},
)

predictor = model.deploy(
    instance_type="ml.g5.2xlarge",
    initial_instance_count=1,
    container_startup_health_check_timeout=900,
    model_data_download_timeout=1800,
)

Creating an Endpoint

Once the model is created, create an endpoint configuration and deploy the endpoint.

It's recommended to change two timeout settings. Extend container_startup_health_check_timeout to 900 seconds (default: 300 seconds). Loading the 18GB model and initializing vLLM takes time, so the default may cause a timeout. Similarly, extend model_data_download_timeout to 1800 seconds for peace of mind.

Confirming Deployment Completion

When the endpoint status becomes InService, deployment is complete. In my environment, it took approximately 11 minutes (677 seconds) including model download from S3 and vLLM initialization.

As a sanity check, try sending a simple prompt.

import json
import boto3

sm_runtime = boto3.client("sagemaker-runtime", region_name="ap-northeast-1")

payload = {
    "inputs": "What is the height of Tokyo Tower?",
    "parameters": {
        "max_new_tokens": 128,
        "temperature": 0,
    }
}

response = sm_runtime.invoke_endpoint(
    EndpointName="nemotron-9b-v2-japanese",
    ContentType="application/json",
    Body=json.dumps(payload),
)
result = json.loads(response["Body"].read().decode("utf-8"))
print(result["generated_text"])

If a response is returned normally, the closed-network deployment is complete.

Pain Points Encountered

During the process from deployment to operation verification, I ran into several issues. Among them, the special token problem with the chat template was a particularly important discovery that directly impacts benchmark accuracy.

Chat Template Special Token Issue

This was the biggest discovery in this verification.

Nemotron 9B-v2-Japanese's chat template uses special tokens as role separators for System / User / Assistant. In README files and blog articles on HuggingFace Hub, you often see the notation <extra_id_0> <extra_id_1> <extra_id_2>, but these are actually not the correct special tokens.

Looking at the model's tokenizer_config.json, the actual role separator tokens are <SPECIAL_10> (ID: 10), <SPECIAL_11> (ID: 11), and <SPECIAL_12> (ID: 12). When the string <extra_id_0> is passed to the tokenizer, it's not recognized as a single special token and is split into 6 subwords: <, extra, _id, _, 0, >.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    trust_remote_code=True
)

# Wrong: split into 6 subwords excluding BOS
tokenizer.encode("<extra_id_0>")
# → [1, 523, 15683, 1620, 290, 29900, 1572]  (BOS + 6 subwords)

# Correct: single special token
tokenizer.encode("<SPECIAL_10>")
# → [1, 10]  (BOS + special token)

Looking at the entire template, using the <extra_id_X> method results in 18 extra tokens being injected into the prompt just for role separators. Since these are meaningless noise to the model, they affect inference accuracy.

When actually compared in benchmarks, the <extra_id_X> method achieved an accuracy of 83.3%, while using the correct <SPECIAL_X> tokens improved it to 84.8%. A difference of 1.5pp may seem small, but in certain categories (questions with answer A), there was a significant improvement from 66.2% → 75.0%.

The reliable solution is to apply the template locally using HuggingFace's tokenizer.apply_chat_template() and send the rendered text as inputs.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    trust_remote_code=True
)

messages = [{"role": "user", "content": "What is the height of Tokyo Tower?"}]
formatted = tokenizer.apply_chat_template(
    messages, tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

# formatted correctly contains <SPECIAL_10>, <SPECIAL_11>, etc.
payload = {
    "inputs": formatted,
    "parameters": {"max_new_tokens": 128, "temperature": 0},
}

LMI Messages API Limitations

The LMI container's async mode also supports the OpenAI-compatible messages format, but it didn't work well with Nemotron 9B-v2-Japanese. All requests showed prompt_tokens: 8, and the same response was returned regardless of the prompt content.

There's a high probability that the <SPECIAL_X> tokens in the chat template are not being correctly interpreted by the LMI-side tokenizer processing. As mentioned above, applying the template locally and sending in inputs format works without issues.

S3 Model Specification Method

When using the sagemaker-core v3 SDK (sagemaker.core.resources), care is needed with the S3ModelDataSource settings. Specify "S3Prefix" for s3_data_type and explicitly set compression_type to "None". If compression_type is omitted, "Gzip" becomes the default, and an error will occur when it tries to decompress uncompressed model files.

Benchmark Comparison

JCommonsenseQA

Measured under the same conditions as the DGX Spark verification from last time.

The vLLM versions and GPUs differ between DGX Spark (NGC container) and SageMaker (LMI container), but the difference in accuracy is within 1.3pp. The result shows that with the same inference engine, accuracy remains nearly identical even when the hardware changes.

Cost Comparison

DGX Spark only requires electricity costs once purchased, making it overwhelmingly cost-effective during development and experimentation phases. On the other hand, SageMaker stops charging when you delete the endpoint when not in use, making it suitable for production environments with low usage frequency or batch processing-style usage.

Don't forget to factor in the cost of VPC endpoints (Interface type). In the Tokyo region, it's $0.014/h per ENI, which accumulates by the number of AZs × number of endpoints. Deleting VPC endpoints when not in use will save costs.

Cleanup

When verification is complete, don't forget to delete the endpoint. ml.g5.2xlarge costs $2.197/h, so leaving it running for a day results in approximately $53 in charges.

There are three items to delete. Delete them in reverse dependency order: endpoint → endpoint configuration → model.

import boto3

sm = boto3.client("sagemaker", region_name="ap-northeast-1")

# 1. Delete endpoint
sm.delete_endpoint(EndpointName="nemotron-9b-v2-japanese")

# 2. Delete endpoint configuration
sm.delete_endpoint_config(EndpointConfigName="nemotron-9b-v2-japanese")

# 3. Delete model
sm.delete_model(ModelName="nemotron-9b-v2-japanese")

If VPC endpoints are no longer needed, delete them as well. You can select the 6 endpoints created this time from the endpoint list in the VPC console and delete them.

Summary

Deploying Nemotron 9B-v2-Japanese in a closed-network configuration to the SageMaker Tokyo region itself worked without issues. Once you've configured the 6 VPC endpoints, you can achieve a configuration where data never leaves Japan's VPC. Deployment time is approximately 11 minutes, and inference latency is also practical at 0.30 seconds/question.

The biggest takeaway from this verification is that with vLLM, accuracy remains nearly identical even when the hardware changes. Despite differences in both version and GPU between DGX Spark (NGC vLLM, 83.5%) and SageMaker (LMI vLLM, 84.8%), the difference was only 1.3pp. The fact that accuracy confirmed locally can be reproduced in the cloud as long as the same inference engine is used is reassuring when selecting deployment targets.

Reference Links

• NVIDIA-Nemotron-Nano-9B-v2-Japanese (HuggingFace)
• Previous article: Trying Nemotron 9B-v2-Japanese in Various Scenarios
• SageMaker LMI Deployment Guide (DJL)
• SageMaker VPC Endpoints (AWS Docs)
• LMI Container Release Notes (DJL)
• vLLM Engine User Guide (DJL)
• LMI Async Mode Migration Guide (DJL)