
What We Did to Entrust a Mini Factory to an AI Agent at the Japan NPN Partner Agentic AI Hackathon
This page has been translated by machine translation. View original
Introduction
Hello, I'm Shimada from the Manufacturing Business Technology Department at Classmethod.
From October 19 to 30, 2026, our in-house team took part in the Japan NPN Partner Agentic AI Hackathon hosted by NVIDIA.
Our team built a miniature factory whose production line, made up of a 3D printer, a robot arm, a conveyor, and an inspection camera, is managed by AI agents.
In this article, I write about the agent part that I was in charge of.
The concept is that once a person places an order, the agents autonomously keep everything running: printing, inspection, shipping, and even remaking items when defects occur.
To achieve this, I designed who makes which decisions and through which path the equipment is operated.
What the agents are responsible for
The flow of the mini factory is as follows: a person places an order, the 3D printer prints a duck, the conveyor carries it to the inspection stand, the VLM of the inspection camera judges pass or fail, and if it passes, it is shipped.
Of these, what the agents decide is how to allocate an order to production, what to print and when, and whether to remake an item or notify a person after a defect or failure.
I created two agents.
- Factory manager: Receives factory events, assesses the situation, and assigns work to the production agent. It is a Hermes Agent (Nous Research) profile, and the model is Qwen3.8-27B
- Production agent: Decides what to print and when. The model is NVIDIA Nemotron 3.5 Bolt E4B 4GB
The division of labor is that the factory manager asks production to "print the rest of this order," and production looks at the printer's state and the remaining quantity of the order and proposes starting a print.
For inspection and abnormal stops, the inspection program and jam monitor built by the respective owners publish events to MQTT, and the factory side receives them.
The loop that remakes items based on inspection results
The core of the factory loop is a record for each order: "how many were requested, how many were shipped, how many were removed as defective, and how many remain."
The capability server holds this record and updates it every time an inspection result arrives.
When inspection finds a defect, that one unit goes back into the order's remaining quantity.
If the printer is free and there is an order with remaining quantity, the event router wakes the factory manager.
The factory manager reads the order record and asks production to remake the item.
This path runs the reconciliation of checking the ordered quantity against actual results and remaking when they don't match.
There are also failures that agents alone cannot recover from, such as the nozzle clogging during printing.
In that case, production chooses "notify a human," and the factory manager issues a notification to the person.
When the person clears the bed, the router confirms that the printer is free, wakes the factory manager again, and the factory manager reassigns the remaining work.
The LLM only produces proposals
If an LLM directly operated the real 3D printer or arm, there would be a not-insignificant chance that unintended operations reach the equipment as they are.
So I did not give the agents any tools that write to the equipment, and allowed them only to output structured proposals (JSON).
This is the pattern found in NVIDIA's Multi-Agent Intelligent Warehouse Blueprint.
Proposals are accepted through proposal_submit on the capability server (an MCP server).
The decision engine judges each one as APPROVED, REJECTED, or DEFERRED, and the capability server sends only the APPROVED ones to the device MCP.
Proposals, judgments, and execution results are all recorded in a ledger.
The decision engine is a pure function that does not use an LLM, and it returns a judgment from the proposal, the factory state, and the judgment time.
The main rules are as follows.
| Judgment | Examples |
|---|---|
| REJECTED | The printer is not idle, printing more than the order's remaining quantity, the bed has not been cleared (BED_NOT_CLEARED), the agent is not allowed to submit that proposal (AGENT_NOT_ALLOWED) |
| DEFERRED | Operations that discard material, such as canceling a print, and the first print during unattended hours |
Who submitted a proposal is determined not by a self-declaration in the proposal's content but from the MCP Bearer token.
With the factory manager's token, only the order record (order.create) and requests to production (agent.dispatch) can be submitted, and a proposal for the factory manager to start a print itself is stopped with AGENT_NOT_ALLOWED.
When a proposal is REJECTED, a reason code (reason_codes) is returned, so the agent can read the reason and resubmit.
The A1 mini has no sensor to detect whether there is an object on the bed.
So, after a print finishes on the real machine, I made the decision engine stop the next print with BED_NOT_CLEARED until a signal that the bed has been cleared arrives.
The signal is any of: a person saying "I took it out," the real arm completing the removal, or an inspection result.
Even if the agent doesn't know about this situation, a print will never start on top of the previous finished product.
The factory manager is woken by events
The factory manager does not poll on its own.
The event router (a deterministic service) watches MQTT events and wakes the factory manager only when a decision is needed.
| When to wake | How it is checked |
|---|---|
| Production chose "wait" or "notify a human," or a proposal was REJECTED | The event of production's execution result |
| The printer is free and there is an order with a number of units not yet printed | Read the capability server after print, removal, and inspection events |
| An order is completed | The order completion event |
Fine-grained updates such as print progress do not wake it.
When waking it, the router calls the Hermes CLI once and puts the trigger and the 15 most recent events into the prompt.
prompt = ("工場からの知らせで起こした。必要なら minifab の道具で対応し、最後に何をしたかを日本語 2 文以内で答えて。\n"
f"きっかけ({kind}): {reason}\n最近のイベント(古い順):\n" + "\n".join(recent[-15:]))
out = subprocess.run([HERMES, "-p", PROFILE, "chat", "-Q", "-t", "minifab", "--reasoning", "low",
"-q", prompt], capture_output=True, text=True, timeout=TIMEOUT)
I assumed the factory manager does not remember the previous conversation, and had it re-read the state from the capability server every time.
Since it doesn't rely on conversational memory, it can resume its judgment where it left off even if the router or the factory manager is restarted.
When a person places an order or asks about the situation, they also talk to the factory manager through the same Hermes CLI.
The factory manager reads the capability server to answer, and for an order, it issues order.create and a request to production.
In the demo, the Reachy Mini described later became the front desk that calls this CLI.
Why I took the production agent off Hermes
I had originally planned to run the production agent as a Hermes profile as well.
However, when I used Bolt E4B from Hermes, the format broke, with tool calls coming back as plain strings, and only 1 out of 4 attempts was executed.
The prompt length (about 1,500 tokens) was not the cause.
So for production, I dropped Hermes and made it a short-loop program.
- The printer state, orders, and catalog are read by code from the capability server
- Only one tool,
choose, is given to the LLM, and it is made to call it just once withtool_choice: required - If the chosen move is starting or canceling a print, it is submitted to
proposal_submitwith production's token. If REJECTED, it is asked again just once with the reason attached
CHOICES = ("print.start", "print.cancel", "wait", "notify_human")
TOOL = {"type": "function", "function": {
"name": "choose",
"description": "次の 1 手を選ぶ",
"parameters": {"type": "object", "properties": {
"action": {"type": "string", "enum": list(CHOICES)},
"job_id": {"type": "string"},
"quantity": {"type": "integer"},
"order_id": {"type": "string"},
"rationale": {"type": "string", "description": "判断の根拠"},
}, "required": ["action", "rationale"]}}}
By limiting it to one round trip and one tool, it became able to choose the right move in the request, waiting, and failure scenarios, and each decision took 3 to 5 seconds.
Listing the options and having the model pick one was more stable than letting a small model search freely for tools.
Keep steps that need no judgment in code
After a print finishes, "removing it," "recording the inspection result," and "shipping if it passes, removing it if it fails" leave no room for judgment.
For this part, the line control inside the capability server submits proposals in a fixed procedure, and the agents are not involved.
Line control proposals also go through the decision engine, so they are stopped by the same rules as the agents.
The agents' role is limited to scenes that require judgment: how to allocate orders, and whether to remake or notify a person after a defect or failure.
Why I chose this technology stack
| Element | Choice | Reason |
|---|---|---|
| Factory manager framework | Hermes Agent | It can be woken by a single CLI call, and the set of usable tools (toolset) can be narrowed per profile |
| Factory manager model and location | Qwen3.8-27B on vLLM on NVIDIA DGX Spark | The policy was to bring the equipment to the venue and complete everything within it, so inference was also placed on the team's machine |
| Production model and location | Bolt E4B on llama.cpp on NVIDIA Jetson AGX Orin | Production's decision is a one-move choice at a time, so a small model is enough. It is placed on the same machine as the MQTT broker and the printer's MCP |
| Interaction with equipment | An MCP server per piece of equipment | The owner of each piece of equipment sets up its own MCP server, and only the capability server calls their write-type tools. The agents cannot see the equipment's MCP |
| Events | MQTT | The equipment is split across multiple Jetsons, and each publishes its own events. Topics and formats were unified between mocks and real machines |
| Safety pattern | Decision engine + capability server | The pattern from the Multi-Agent Intelligent Warehouse Blueprint. Because judgments can be reproduced afterward, there was also an idea to use the decision engine as the reward for reinforcement learning (RLVR) (we did not get as far as fine-tuning during the event) |
I also considered having the factory manager do everything, but in line with the structure of the NVIDIA Factory Operations Blueprint (FOX), I made it two layers in which the factory manager delegates to specialized agents and programs.
Run equipment without real machines using mocks
So that we could test end to end before the real machines were ready, I put equipment mocks inside the capability server.
The printer, arm, conveyor, and inspection stand are each mocked, and the clock can run at 20x speed, so even prints that take tens of minutes can be run through in a short time.
Starting with the equipment whose real machines are finished, I swap the mocks for real machines one by one.
For example, when the printer becomes a real machine, approved print starts are sent to the real machine, and the mock printer receives the real machine's completion events to match its state.
The whole factory can run even in a mixed state where the arm and conveyor remain mocks, which suited the way a hackathon goes, with real machines coming together one at a time.
Connecting a Reachy Mini for the demo
For running the factory loop, interaction with people is covered by the Hermes CLI.
To make the venue demo more impressive, I connected the robot Reachy Mini as the front desk.
Reachy converses using OpenAI's gpt-live-1, and for requests and questions to the factory, it passes the end of the conversation to the factory manager to answer.
The factory manager's reply takes 10 to 15 seconds each time.
Notifications to people (such as "please clear the bed") are consolidated by the event router into a single MQTT topic, and Reachy subscribes to it and reads them aloud.
However, for "start the factory," "stop the conveyor," and "I took it out," I don't let the factory manager decide; only fixed phrases are published directly to MQTT as human signals.
Stopping the conveyor is a safety-side operation, so I didn't want to delay it by putting an LLM's judgment in between.
"I took it out" is a declaration that the person cleared the bed, and if the LLM mixed it up, the next print would start with the finished product still on the bed.
Retrospective
Here is what I learned about the agent part.
- Because I unified the path that writes to the equipment into the capability server and the decision engine, I didn't have to touch the agents' prompts no matter how many times I changed the handling of approvals or bed clearing during the event
- When a small model is run in a general-purpose agent framework, the format of tool calls easily breaks. One round trip and one tool made it stable
- Leaving even steps that need no judgment to the agents makes things slower and adds more points of failure. I limited what I leave to the agents to scenes that require judgment, such as remaking and notifying people
- Because mocks and real machines can be swapped per piece of equipment, I could test the agent loop end to end even before the real machines were ready
I think part of the responsibility currently left to the decision engine could be replaced by NVIDIA OpenShell.
Since we used Hermes Agent this time, we are also considering introducing NemoHermes, which makes use of the OpenShell mechanism, in the future.
Conclusion
In the mini factory, I built the part where agents run everything from order to printing, inspection, and shipping, as well as remaking items when defects occur.
The LLM only produces proposals, and only those that pass through the deterministic decision engine are executed.
We settled on a design in which the factory manager is woken by events and production has a small model choose just once.
On the day of the demo, we were able to smoothly show everything from giving instructions to the Reachy Mini, to starting factory operation, to the coordination of the arm and conveyor. Since we also plan to exhibit it internally, we intend to further develop the whole mini factory and refine it to the point of running the loop from production to shipping.
