I thought about a configuration for delivering reasoning models to the field on NVIDIA Jetson

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Reading equipment logs, referencing maintenance manuals, and proposing the next action items to workers — when you want to run this kind of support on terminals at a job site, simply having the model start up is not enough. You want it to avoid adding facts not found in the logs, return well-grounded proposals, and do so within a time that doesn't interrupt the work.

A technical article published by NVIDIA on September 4, 2026, introduces a configuration for running Nemotron 3.5 Lightning and Qwen3.8-27B on Jetson, along with methods for accelerating generation.

https://developer.nvidia.com/blog/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models-on-nvidia-jetson/

The reasoning models referred to here are models that construct answers by performing multi-step reasoning. On the other hand, the execution process of passing input to a model and obtaining output is also called inference. In this article, we consider how to adjust the memory and latency of the latter in order to use the former on-site.

Local processing can reduce dependence on networks and transmission of data to external parties. However, if responses are slow, it becomes difficult to use on-site. The selection of Thor, AGX Orin, and Orin Nano is also considered together with the tasks they are assigned.

This article introduces the mechanisms of quantization and speculative decoding, as well as considerations for choosing models and deployment targets. It is intended for those who are familiar with the overview of LLMs and RAG and are considering a PoC using Jetson.

Public materials have been verified as of September 5, 2026. The performance figures presented are values reported by NVIDIA and are not measurements I have re-verified on actual Jetson hardware. The equipment maintenance and video recognition scenarios are also used as fictional design examples for explanatory purposes, not as systems already deployed.

Reading NVIDIA's Reported Acceleration Results

What the original article covers is NVFP4 for quantizing models, and speculative decoding for changing how answer generation proceeds. First, let's confirm what exactly was made faster.

NVIDIA's Figure 2 compares the decode throughput multiplier relative to a baseline configuration where each model runs in 16-bit BF16 format. Decoding is the part that generates the answer after processing the input.

The figures are NVIDIA's published values organized into a table. For Nemotron, DSpark, and for Qwen, DFlash2, were used as the fastest configurations among those compared by NVIDIA.

What is notable in this table is that speed is extended not by stopping at quantization alone, but by additionally changing the generation procedure. However, the 1x baseline for both models does not represent the same speed. The fact that Qwen's multiplier is larger does not mean this table shows that Qwen is faster in absolute terms or higher in answer quality than Nemotron.

From the body text and figure captions reviewed, it was not possible to identify all conditions corresponding to these figures, such as the specific device, power mode, and input/output lengths. These are not results showing that the same multiplier is achieved across all Jetson models, nor results showing the performance difference between Thor and Orin.

In equipment maintenance, comparisons are made over the combined time of log acquisition, manual search, input processing, and answer generation. If search or tool waiting time is long, speeding up generation will result in only a small reduction in overall time. For processes that repeatedly generate long answers, the improvement is likely to be more effective.

How to Choose Between Thor, AGX Orin, and Orin Nano

Even within Jetson, onboard memory and GPU architecture are not uniform. Here we use the AGX Thor with Jetson T5000 in a 128 GB configuration, the AGX Orin 64GB, and the Orin Nano Super Developer Kit in an 8 GB configuration as representative examples. The target is the Orin-generation Nano, not the older Jetson Nano.

Hardware specifications are based on NVIDIA's product pages for Thor, Orin, and Orin Nano Super. The last column is not a specification but a consideration of the scope of responsibility in the equipment maintenance example.

The power range does not represent measured power consumption for the entire device including storage and peripherals. Also, because comparing FP4 compute performance and INT8 TOPS as parallel performance metrics does not align conditions, they are not included in this table.

Thor is not exclusively for verification. If you need a model that delivers the required quality, or if you need memory to accommodate multiple processes, choosing a production device with Thor is also an option. Conversely, if the use case is narrow and the deployment constraints are well understood, it is fine to start verification directly with a Nano. There is no need to begin every project by procuring a Thor.

It also seems premature to assume that distillation is required to move to AGX Orin. The Nemotron 3.5 Lightning model page explicitly lists AGX Orin 64GB as a supported platform. If there is a candidate that can use the same model, it is worth first verifying business quality and latency with that configuration.

On the other hand, the 8 GB on the Nano is memory shared between the CPU and GPU, and it is not capacity that can be used solely for model weights. If log acquisition and search are also run on the same terminal, that overhead must be accounted for. If miniaturization leads to unstable operation, options such as using AGX Orin or offloading processing to a hub-side device are compared. If intermediate capacity is needed, Orin NX is also a candidate.

Development kits and production hardware should be distinguished. NVIDIA's FAQ states that development kits are for development and evaluation, while production modules are for production deployment. For production, in addition to the module and a production carrier board, an enclosure and cooling must also be selected.

Depending on requirements for safety functions, security, and long-term support, systems equipped with NVIDIA IGX are also candidates. IGX is not universally required, nor does its adoption alone guarantee the safety of the entire device.

Separating Model Scale from Required Memory

Nemotron 3.5 Lightning is a MoE model with a total of 30B parameters, where 3B parameters are active per token. MoE stands for Mixture of Experts, where a subset of Experts is selected and used for each token processed.

In contrast, Qwen3.8-27B is a Dense model with 27B parameters. There is no Expert switching as in MoE; all 27B parameters are used for each token processed. NVIDIA's article suggests viewing Nemotron as suited for agents that make repeated responses, and Qwen for use cases requiring difficult judgments in fewer turns. However, this is just a starting point for selection. Whether the model can prevent equipment mix-ups or correctly cite the basis from maintenance manuals is not determined by model architecture alone.

One thing to be particularly careful about is not reading "3B active parameters" as meaning the required memory is equivalent to 3B parameters. Even if only a subset of Experts is used, the weights of the others are not eliminated.

For illustration, let's assume all 30B weights are stored in 4-bit format. 30,000,000,000 × 4 ÷ 8 gives 15,000,000,000 bytes, approximately 15 GB in decimal notation. This is a simple calculation and not a measured value of actual checkpoint size or runtime memory.

In practice, quantization scale information, runtime state such as the KV cache that reuses past computations, auxiliary models, and the inference engine's working area also consume memory. If inputs are made longer or concurrent processing is increased, differences will arise that cannot be explained by weight capacity alone. Whether the model file has become smaller and whether it fits on the terminal together with the application are separate things to verify.

For the Nano, it is also worth trying a replacement with an existing small model at an early stage. The official TensorRT Edge-LLM tutorial includes an example of quantizing the weights of Qwen3-4B-Instruct-2507 to 4-bit with INT4 AWQ and using it on an Orin Nano 8 GB. This is not a distilled version of Qwen3.8-27B, but an example of choosing a different model.

Speeding Up Generation with Quantization and Speculative Decoding

Input Processing and Answer Generation

LLM inference has two phases: Prefill, which reads the input, and Decode, which generates the answer. This section mainly covers Decode. Latency immediately after input and generation speed do not necessarily improve with the same adjustments.

NVFP4 Reduces Numerical Representation Size

Quantization is a technique for representing values such as weights with a lower number of bits. NVIDIA's explanation of NVFP4 combines 4-bit values with block-level and global scales. It is a format that not only coarsens the numerical representation but also uses scales calibrated to the magnitude of the values.

Thinking of it as reducing the amount of data handled each time may help in grasping the intent. If the amount of data when the GPU reads out weights and performs computation can be reduced, this affects not only memory capacity but also the time required for generation.

However, 4-bit conversion does not reduce the number of parameters. Not all memory regions shrink at the same ratio, and whether the precision degradation is acceptable for business purposes is a separate matter to verify. Required memory, the operations used, speed, and answer quality must all be confirmed with the chosen model and implementation.

Speculative Decoding Verifies Drafts in Batches

Normal autoregressive generation runs the main model to determine the next token, then uses that result to proceed to the next. The Jetson AI Lab tutorial explains that in environments handling a small number of requests, the memory bandwidth for reading weights per token tends to become a bottleneck.

In speculative decoding, a lightweight predictor drafts multiple candidates, and the main model verifies them all at once. Candidates are accepted from the beginning, and drafts after the first rejected position are discarded. Since generation can advance by the number of accepted candidates, more output can be obtained from a single pass of reading the main model's weights. The key point is that drafts are not used directly as the answer; acceptance is determined by passing through the main model.

Quantization changes the numerical representation; speculative decoding changes the generation procedure. This is a schematic diagram and does not represent processing time, acceptance rate, or speedup multipliers. Drawn with reference to Jetson AI Lab and NVIDIA's NVFP4 materials.

There are also differences in approach. MTP uses prediction heads tailored to the model, DFlash drafts blocks in parallel, and DSpark adds correction and confidence mechanisms to parallel drafting. All of these require confirming compatible models and execution paths, and are not simple drop-in replacements for drafts intended for a different model size.

More candidates does not necessarily mean faster. Generating and verifying rejected candidates also has a cost. For inputs that include equipment names or unique error codes, acceptance rates may differ from general text, so comparisons should be made with inputs representative of the actual business tasks.

Correctly implemented standard speculative decoding is designed to preserve the output distribution of the main model. However, this does not mean it restores quality to what it was before quantization, nor is it an explanation that guarantees bit-level agreement across different implementations. It should not be used as a reason to skip quality evaluation of quantization.

Loading a Model Does Not Guarantee Fast Execution

Even if an NVFP4-format model can be loaded, it does not necessarily mean it will be processed using the GPU's FP4 arithmetic capabilities. The actual computation method varies depending on the device and the inference engine.

For example, TensorRT Edge-LLM's NVFP4 support for Jetson targets Thor. On the other hand, NVIDIA also introduces a configuration where an NVFP4 model is loaded onto AGX Orin using vLLM.

However, how that Orin configuration computes internally was not confirmed this time. Rather than judging speed solely by the model's storage format, speed should be verified for the specific combination of deployment device and inference engine.

Understanding the Official Deployment Configuration

NVIDIA's startup example assumes JetPack 7.2, Docker with NVIDIA Container Runtime, and vllm/vllm-openai:v0.28.0. Rather than reproducing the full commands here, we will examine what is being combined.

The Nemotron example specifies nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 as the main model and a dedicated draft with -DSpark appended to the same name. The method is dspark with 5 candidate tokens. Another speculative decoding tutorial uses 4 candidates, so rather than mixing configurations from different sources, start with one configuration as the baseline.

The key groups of settings to check are as follows. The meaning of options can also be referenced in vLLM's serve CLI, but be careful not to apply default values from the updated stable documentation to older versions.

The Qwen side uses the combination of Inferact/Qwen3.8-27B-NVFP4 and incoai/Qwen3.8-27B-DFlash2, with the method specified as dflash and 7 candidates. Be careful not to confuse the checkpoint name DFlash2 with the CLI specification name.

When changing the runtime, also verify the required APIs. For example, the Nemotron model page explicitly states that the experimental sample in TensorRT Edge-LLM 0.10.0 does not support OpenAI tools requests. The model's ability to select tools and the server's ability to handle tool requests are separate matters.

The configurations presented so far are officially published examples and are not procedures I have personally verified as working. When verifying, record the model number, JetPack version, container tag and digest, model and draft revisions, key settings, and power mode. Do not expose development startup examples directly to a production network; also determine authentication, access scope, distribution source, and tool permissions separately.

Deployment to the Field Is More Than Just Miniaturization

From here, we consider how to proceed with deployment based on the technologies introduced. We use a support agent that uses equipment logs and maintenance manuals as an example, but this does not include automated execution of equipment operations.

First, Build a Business Flow That Enables Comparison

First, build a business flow that allows comparison when switching models. Standardize inputs to include equipment ID, logs, measurements, and relevant sections of the maintenance manual, and request that answers include facts, grounds, next action items, and matters for human confirmation. Leave aggregation and threshold judgments to code.

In parallel with verifying required quality, also run the system on candidate deployment targets early. Evaluation ground truth should not be taken directly from model outputs but prepared through manual review and confirmation by business staff.

First Compare Existing Models and Changes in Scope of Responsibility

Compare configuration adjustments to the same model, replacement with existing small models, redistribution of processing, and additional training or distillation if needed. This is not a sequence of steps to be carried out one after another. For example, temperature rise detection could be handed off to fixed rules, with the model responsible only for explaining the relevant procedure.

A design proposal that compares model adjustment, replacement, distillation, and processing redistribution on the same evaluation set, then evaluates at the deployment target. Distillation is not a mandatory step, and the training and conversion environment is considered separately from the deployment target.

Distillation Is an Option When a Shortfall Is Identified

When a small model lacks sufficient quality, consider distillation narrowed to the use case. NVIDIA's explanation of knowledge distillation distinguishes between learning from teacher-generated data and learning to approximate the output distribution.

The target is not all general capabilities, but the specific task of returning well-grounded action items in a defined format. Since teacher errors can also be learned, review the generated data, and separate evaluation sets by equipment event from training data.

The environment where inference runs does not necessarily support training as well. Separate the training and conversion environment from the deployment target, and verify the terms for commercial use and distillation of teacher and student models and data.

Quantization-Aware Distillation (QAD) introduced by NVIDIA uses a teacher to help recover precision lost through quantization. This is different from the goal of replacing a model with a smaller student.

The draft in speculative decoding plays a different role from a student model that completes business tasks on its own.

Dividing Work Between the Equipment Side and the Hub Side

Another option is to assign only limited classification tasks and short descriptions to the Nano, and pass investigation of multiple pieces of equipment to the AGX Orin or Thor on the hub side. The scope and number of units are determined based on evaluation at the deployment target.

When communication is unavailable, the terminal should not make judgments beyond its scope; instead, it should escalate to a human. Conditions for escalation should not rely solely on the model's self-report; insufficient grounds, missing inputs, tool failures, and out-of-spec equipment states should also be detected on the application side.

Separating Models That Explore from Models That Run Continuously for Video

Building on the SAM 3.1 verification article as of April 1, 2026, this time we consider the division of training and deployment.

VSS is a Blueprint that combines video search, summarization, and other functions; it is not a single trained model. The proposed approach is: use VSS or a VLM that handles images and language to search for target scenes, use SAM variants to generate annotation candidates such as masks showing object regions, have humans verify classes and missed items, use that data to train a YOLO-type model suited to the use case, and evaluate it on Jetson.

This is not about converting SAM weights into YOLO. Supervised learning using pseudo-labels and distillation that aligns feature maps or output distributions are treated as distinct. Exporting to TensorRT format after training is also a separate step.

For counting conveyed objects, detection, tracking, and passage determination are combined; if contours are needed, a segmentation-capable model is selected. First compare whether existing lightweight models are sufficient, and do not assume that interpretation of work order or explanation of reasons can be transferred to the detector.

To prevent adjacent frames from the same video from mixing between training and evaluation sets, separate by video or shooting date, and also verify under different camera conditions. If a VLM is only called when a lightweight model detects something, missed detections will never reach the downstream stage. Spot checks of ordinary footage are also necessary.

This configuration is unverified. Do not apply the LLM speedup multipliers to video, and determine where to allocate compute resources among exploration, annotation generation, and continuous recognition based on quality and response time.

Judging Deployment Based on Quality, Latency, and Operational Conditions

Even with the same optimization, results vary depending on the type of input. NVIDIA's Figure 3 compares each model and category against the NVFP4 configuration as a 1x baseline. This differs from the BF16 baseline used earlier.

Across the four categories shown — Writing, Reasoning, Summarization, and RAG — both models show a larger improvement for RAG and a smaller improvement for Summarization. This is not a measurement of overall search time in a RAG system but a comparison of decoding for inputs in that category. Rather than assuming the same multiplier will apply to equipment maintenance RAG, take this as motivation to prepare inputs close to actual business tasks.

For deployment decisions, the following items should be verified in addition to speed.

The Jetson AI Lab benchmarking guide is a useful reference for definitions of speed metrics. TTFT is time to first token, and ITL is inter-token latency during generation. On the other hand, Output token throughput from vLLM's serving benchmark is the number of output tokens relative to the total measurement time and is not necessarily the speed of pure Decode alone.

With reasoning models, even when the thinking portion of the output begins, the answer that the worker can actually use may not yet have arrived. In addition to TTFT, it is worth recording the time until action items become visible and the time until necessary references are complete. Long latencies that averages do not reveal, as well as failed and timed-out requests, should also be retained.

When making comparisons, align business inputs, search results, tool response conditions, output length limits, concurrency, and cache conditions. For before-and-after comparisons of the same model's optimization, also fix sampling settings, and avoid simply extending speed by changing to settings that make candidates more likely to be accepted.

Configurations that do not pass quality or exception handling are excluded from deployment candidates, regardless of speed. After that, the order is to compare latency, terminal configuration, and operational burden among the passing candidates. Acceptance rates and time figures should be agreed upon with the business side in accordance with the impact of equipment downtime and the time workers can afford to wait.

Summary

When considering reasoning models on Jetson, the fact that a model starts up, that answer generation is fast, and that it is useful on-site are each separate things to verify. NVIDIA's article presents a configuration that extends generation speed by combining quantization and speculative decoding. However, the multiplier alone is not enough to determine the deployment target or business quality.

My own view is that it is easier to proceed by first building a comparable business flow, trying existing small models and reviewing the scope of responsibility, and then considering distillation if a shortfall is identified. This is because, depending on evaluation results, the option of continuing to use Thor, or distributing processing to AGX Orin or Nano, can both remain open.

At this stage, the configuration has been organized based on publicly available information, and quality and continuous operation on actual Jetson hardware have not been verified. Rather than cramming everything into the smallest terminal, the idea is to allocate processing according to required quality and latency. Preparing the business inputs and evaluation criteria to inform that judgment seems to be the starting point for deployment.

Reference Links

• Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson
• NVIDIA Jetson Thor
• NVIDIA Jetson Orin
• NVIDIA Jetson Orin Nano Super Developer Kit
• Jetson FAQ — NVIDIA
• NVIDIA IGX
• Nemotron 3.5 Lightning — Jetson AI Lab
• Qwen3.8 27B — Jetson AI Lab
• Speculative Decoding on Jetson — Jetson AI Lab
• TensorRT Edge-LLM on Jetson — Jetson AI Lab
• GenAI Benchmarking — Jetson AI Lab
• QAT and QAD — NVIDIA Model Optimizer
• Data-Efficient Knowledge Distillation for Supervised Fine-Tuning with NVIDIA NeMo Aligner
• NVFP4 — NVIDIA Transformer Engine
• serve CLI — vLLM
• Video Search and Summarization — NVIDIA
• Segment Anything Model 3 — Ultralytics
• Instance Segmentation — Ultralytics
• Model Export — Ultralytics