I tried reinforcement learning for a robot arm with DGX Spark (Isaac Sim + Isaac Lab + SO-ARM101)

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

At CES 2026, Physical AI was prominently featured in NVIDIA's keynote, and GTC 2026 scheduled for March also has a Physical AI Day set, so attention to the field combining robotics and AI is rapidly growing. I need to ride this wave too!

So this time, with the theme of "first touch and understand it myself," I decided to learn NVIDIA Isaac Sim and Isaac Lab from the basics. I'll move a robot arm in the Isaac Sim simulation environment, design reward functions myself, and observe how the learning results change.

Rather than just running existing training scripts, I'll dive into reading the reward design code, changing parameters, and writing my own reward functions. Since this is a learning log started from nearly zero experience with reinforcement learning, I hope it will be helpful for those who are similarly "interested in Physical AI but don't know where to start."

Goals for This Time

I'll proceed in 3 stages.

1. Build an Isaac Sim + Isaac Lab environment on DGX Spark
2. Manually interact with a robot arm in Isaac Sim's GUI
3. Customize reward functions and observe changes in learning results

Prerequisites

About SO-ARM101 and isaac_so_arm101

SO-ARM100/101 is an open-source robot arm developed by The Robot Studio. It features 6 degrees of freedom, Feetech STS3215 servo motors, and is characterized by its affordable price starting from $220. It has also been adopted as reference hardware in Hugging Face's LeRobot project. This time I'll use the improved SO-ARM101.

isaac_so_arm101 is an extension package for using this SO-ARM100/101 with Isaac Lab. It comes with a complete set of URDF models, environment configurations, and training scripts, so once the environment is set up, you can start reinforcement learning right away.

Environment Setup

I'll set up Isaac Sim, Isaac Lab, and isaac_so_arm101 on the DGX Spark ARM64 environment. On an x86_64 machine this would take just a few minutes with pip, but ARM64 requires a source build and there are several pitfalls. The details of the pitfalls are summarized in a table at the end of the article, so here I'll proceed focusing on the commands.

Source Build of Isaac Sim

First, install GCC 11. Ubuntu 24.04 defaults to GCC 13, but building Isaac Sim with GCC 13 fails. The same steps are described in the NVIDIA official Isaac Playbook.

sudo apt update && sudo apt install -y gcc-11 g++-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 200
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 200

Install Git LFS, then clone the repository and build.

sudo apt install -y git-lfs

git clone --depth=1 --recursive https://github.com/isaac-sim/IsaacSim.git
cd IsaacSim
git lfs install && git lfs pull
./build.sh -r  # Release build

It took about 13 minutes on DGX Spark. Please ensure at least 50GB of free disk space.

export ISAACSIM_PATH="${PWD}/_build/linux-aarch64/release"

Isaac Lab and isaac_so_arm101

git clone --recursive https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab

# Create a symbolic link to Isaac Sim (recommended method in Playbook)
ln -sfn "${ISAACSIM_PATH}" "${PWD}/_isaac_sim"

# Install (automatically upgrades to PyTorch 2.9.0+cu130)
./isaaclab.sh --install

Install isaac_so_arm101 with --no-deps. This is because uv sync mentioned in the README fails to resolve dependencies in the DGX Spark source build environment.

git clone https://github.com/MuammerBay/isaac_so_arm101.git
./isaaclab.sh -p -m pip install -e ~/works/robotics/isaac_so_arm101 --no-deps

Environment Variable Setup Before Running

Two environment variables are needed to run Isaac Sim on DGX Spark.

# Workaround for OpenMP linking issue
export LD_PRELOAD="/lib/aarch64-linux-gnu/libgomp.so.1"

# When running headless such as in an SSH session (DISPLAY is required)
export DISPLAY=:0
export XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.XXXXXX

Forgetting LD_PRELOAD will cause mysterious crashes. The XAUTHORITY filename changes with each session, so check it with ls /run/user/1000/.mutter-Xwaylandauth.*.

Interacting with the Robot in Isaac Sim's GUI

Now that the environment is ready, let's first open the Isaac Sim GUI and manually move the robot arm.

Launching the GUI and Loading SO-ARM101

Launch the Isaac Sim GUI and load the SO-ARM101 URDF.

cd ~/works/robotics/IsaacSim
./_build/linux-aarch64/release/isaac-sim.sh

The first launch takes a few minutes to generate caches for extensions and shaders. Once the GUI is up, load the SO-ARM101 URDF from File > Import.

The path is isaac_so_arm101/src/isaac_so_arm101/robots/trs_so101/urdf/so_arm101.urdf. During import, checking "Static Base" will load it as a robot arm with a fixed base.

The yellow robot arm is displayed in the center of the viewport. The Stage panel on the right shows the link structure as a tree, where you can confirm the hierarchy: base_link → shoulder_link → upper_arm_link → lower_arm_link → wrist_link → gripper_link.

Operating Joints with Physics Inspector

Isaac Sim has a tool called Physics Inspector that lets you manually operate each joint of the robot with sliders. Open it from Tools > Physics > Physics Inspector.

Once the panel is displayed, selecting the robot's articulation shows each joint's DOF (degrees of freedom) in a list with sliders.

SO-ARM101 has 6 joints. shoulder_pan controls the base rotation, shoulder_lift and elbow_flex control the arm's vertical movement, and wrist_flex and wrist_roll control the wrist angle. The last one, gripper, is for opening and closing the gripper.

When you actually move the sliders, you'll find that just shoulder_pan and shoulder_lift determine the arm's rough position. Conversely, wrist_flex and wrist_roll are for fine angle adjustments and affect the end effector's orientation. After actually interacting with it here, I could intuitively understand why "position tracking" and "orientation tracking" are defined as separate reward terms in the reward design that comes later.

The recommended slider mapping mode is "Joint Drive Target Position." Since it moves to the target position through physical simulation, you can observe behavior similar to actual servo motors.

By the way, there's also an approach called Imitation Learning where the trajectories manipulated this way are recorded and used as "demonstrations" for training. It's a method of learning a policy from demo data operated with a leader arm, and LeRobot and GR00T are exactly in this category. For now I'll focus on reward function-based reinforcement learning, but I'd like to try imitation learning with a real arm and a reference arm later.

Running the Existing Reaching Task

Now that I've gotten a feel for the robot's movements in the GUI, let's move on to driving the robot with reinforcement learning.

In robotics reinforcement learning, it's standard practice to incrementally increase task difficulty: Reach (bring the arm tip to a target position) → Grasp (grab an object) → Lift (pick it up) → Place (put it down). The Reaching task is the first step, a simple task of just "moving the end effector (arm tip) to a randomly generated target coordinate." There's no grabbing or lifting objects.

Even though it's simple, you need to coordinate 6 joints to reach any arbitrary point in 3D space, making it a good subject for learning the basics of reward design. isaac_so_arm101 has an environment and training script for this Reaching task ready to use.

Running the Training

cd ~/works/robotics/IsaacLab

./isaaclab.sh -p ~/works/robotics/isaac_so_arm101/src/isaac_so_arm101/scripts/rsl_rl/train.py \
  --task Isaac-SO-ARM101-Reach-v0 \
  --headless \
  --num_envs 64 \
  --max_iterations 1000

Learning with RSL-RL (PPO) in 64 parallel environments begins. The isaac_so_arm101 README writes SO-ARM101-Reach-v0, but note that the current version requires the Isaac- prefix.

1000 iterations completed in about 9 minutes. The throughput is about 3,500 steps/s with DGX Spark's GB10 GPU.

The position error dropped rapidly in the early stages and converged around 0.1.

Evaluating the Trained Policy

./isaaclab.sh -p ~/works/robotics/isaac_so_arm101/src/isaac_so_arm101/scripts/rsl_rl/play.py \
  --task Isaac-SO-ARM101-Reach-Play-v0 \
  --num_envs 4 \
  --video \
  --video_length 200

If you get ModuleNotFoundError: No module named 'isaaclab.utils.pretrained_checkpoint' in play.py, it's because this module doesn't exist in Isaac Lab 0.54.3. You can work around it by wrapping the relevant part with try-except.

When running Play, the trained policy is automatically exported in JIT and ONNX formats, which can be used for later transfer to real hardware.

Understanding the Reward Design

Now that the existing script is running, let's understand its internals. The reward function determines reinforcement learning behavior. Let me read the code to see what rewards are set in the Reaching task of isaac_so_arm101.

Manager-Based Reward Definition

isaac_so_arm101 uses Isaac Lab's Manager-Based approach. The RewardsCfg class in reach_env_cfg.py has a structure where reward terms (RewTerm) are declaratively listed.

@configclass
class RewardsCfg:
    # Task rewards
    end_effector_position_tracking = RewTerm(
        func=mdp.position_command_error,
        weight=-0.2,
        params={...},
    )
    end_effector_position_tracking_fine_grained = RewTerm(
        func=mdp.position_command_error_tanh,
        weight=0.1,
        params={..., "std": 0.1},
    )
    end_effector_orientation_tracking = RewTerm(
        func=mdp.orientation_command_error,
        weight=-0.1,
        params={...},
    )

    # Penalties
    action_rate = RewTerm(func=mdp.action_rate_l2, weight=-0.0001)
    joint_vel = RewTerm(func=mdp.joint_vel_l2, weight=-0.0001, params={...})

Five reward terms are defined. However, in SO-ARM101's Reaching task, orientation_tracking has its weight set to 0.0, so effectively 4 terms are functioning.

position_command_error (weight=-0.2) is a term that directly penalizes the L2 distance between the end effector and the target position. The farther away, the larger the negative reward, so it plays the role of "roughly moving the robot closer to the target."

position_command_error_tanh (weight=0.1, std=0.1) has an interesting design that maps the distance using a tanh kernel. Since tanh has the property of rapidly increasing reward when close to the target, it strongly motivates "final fine-tuning." The smaller the std parameter, the more it insists on precision at closer distances.

orientation_command_error (weight=-0.1) is end effector orientation tracking. In the base RewardsCfg the weight is -0.1, but in SO-ARM101's joint_pos_env_cfg.py, the orientation weight is overridden to 0.0, so it's effectively disabled in actual training. For a Reaching task, "how you arrive" matters less than "whether you arrived at the target position," so it's a reasonable design decision.

The remaining two, action_rate_l2 and joint_vel_l2, are penalties that encourage smooth movement. They aim to learn movements that are easy to reproduce on real hardware by suppressing sudden changes in actions and joint velocities. The initial weight is quite small at -0.0001, but this is related to curriculum learning.

Gradual Penalty Strengthening Through Curriculum Learning

Looking at CurriculumCfg, the weights for action_rate and joint_vel are configured to strengthen as training progresses.

@configclass
class CurriculumCfg:
    action_rate = CurrTerm(
        func=mdp.modify_reward_weight,
        params={"term_name": "action_rate", "weight": -0.005, "num_steps": 4500}
    )
    joint_vel = CurrTerm(
        func=mdp.modify_reward_weight,
        params={"term_name": "joint_vel", "weight": -0.001, "num_steps": 4500}
    )

The mechanism starts with penalties nearly at zero to prioritize "reaching the target first," then gradually strengthens penalties over 4500 steps to require "smooth movement." action_rate goes from -0.0001 to -0.005 (50x increase), and joint_vel from -0.0001 to -0.001 (10x increase).

I found this approach natural and interesting — "first learn to do it roughly, then gradually demand quality" — which resembles the human learning process.

Two Approaches in Isaac Lab

Besides the Manager-Based approach used here, Isaac Lab also has an approach called the Direct Workflow. The Direct Workflow is a style where you inherit DirectRLEnv and directly implement reward calculations in the _get_rewards() method, and SoftBank's Physical AI practical article uses exactly this approach.

Manager-Based is a "declaratively define the MDP" style, with the convenience that adding rewards and adjusting weights requires only configuration changes. Since I want to make use of the isaac_so_arm101 codebase as-is, I'll continue with Manager-Based.

Designing Custom Rewards

Now that I understand the reward design structure, let's actually customize it. I proceeded in 2 stages.

Step A: Adjusting Parameters

First, let me adjust only the weights and std without changing functions. It's as simple as inheriting SoArm101ReachEnvCfg and overriding parameters in __post_init__.

@configclass
class SoArm101ReachCustomACfg(SoArm101ReachEnvCfg):
    def __post_init__(self):
        super().__post_init__()

        # Strengthen distance penalty (-0.2 -> -0.5)
        self.rewards.end_effector_position_tracking.weight = -0.5
        # Increase tanh kernel sensitivity (std 0.1 -> 0.05)
        self.rewards.end_effector_position_tracking_fine_grained.params["std"] = 0.05
        # Require some smoothness from early on (-0.0001 -> -0.001)
        self.rewards.action_rate.weight = -0.001

Three changes were made: strengthening the distance penalty (-0.2→-0.5), increasing the tanh sensitivity (std 0.1→0.05), and adding motion smoothing from the beginning (action_rate -0.0001→-0.001). Each value change is noted in the code comments.

Step B: Adding a Reward Function

Next, let me write a new reward function myself. I referenced the tanh kernel pattern in object_ee_distance in isaac_so_arm101's mdp/rewards.py.

What I added is a "joint limit avoidance reward." The closer each joint gets to the end of its range of motion, the lower the reward, and staying near the center gives high reward. Since operating near joint limits on real robots poses a risk of failure, the intent is to train the habit of avoiding limits during simulation.

def joint_pos_limit_avoidance(
    env: ManagerBasedRLEnv,
    std: float,
    asset_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
) -> torch.Tensor:
    """Returns higher reward the farther joints are from the range of motion limits."""
    asset = env.scene[asset_cfg.name]
    joint_pos = asset.data.joint_pos
    soft_limits = asset.data.soft_joint_pos_limits

    # Distance to the nearest limit for each joint
    dist_to_lower = joint_pos - soft_limits[..., 0]
    dist_to_upper = soft_limits[..., 1] - joint_pos
    dist_to_nearest = torch.minimum(dist_to_lower, dist_to_upper)

    # tanh kernel: far from limit -> ~1.0, near limit -> ~0.0
    return torch.mean(torch.tanh(dist_to_nearest / std), dim=-1)

While referencing object_ee_distance's tanh kernel, that one uses 1 - tanh(d/σ) (higher reward when closer), whereas this one uses tanh(d/σ) (higher reward when farther from limits) — a reverse direction design. With std=0.2, the reward drops sharply when within 0.2rad (about 11 degrees) of a limit.

This function is registered as a new reward term in a RewardsCfg subclass, with the Step A adjustments also applied.

@configclass
class CustomBRewardsCfg(RewardsCfg):
    """Adds joint limit avoidance to existing rewards."""
    joint_limit_avoidance: RewTerm = RewTerm(
        func=joint_pos_limit_avoidance,
        weight=0.05,
        params={"std": 0.2, "asset_cfg": SceneEntityCfg("robot")},
    )

@configclass
class SoArm101ReachCustomBCfg(SoArm101ReachEnvCfg):
    rewards: CustomBRewardsCfg = CustomBRewardsCfg()

    def __post_init__(self):
        super().__post_init__()
        # Apply the same adjustments as Step A
        self.rewards.end_effector_position_tracking.weight = -0.5
        self.rewards.end_effector_position_tracking_fine_grained.params["std"] = 0.05
        self.rewards.action_rate.weight = -0.001

To avoid breaking existing code, I implemented it as a new Cfg class inheriting SoArm101ReachEnvCfg. The configuration adds reward terms in a RewardsCfg subclass and replaces it in the environment Cfg. Registering it as a custom environment in Gymnasium allows running training with the same train.py as the baseline.

Comparing Baseline and Custom Rewards

I ran training for 3 patterns under the same conditions (64 envs, 1000 iter) and compared the results.

The most surprising result was Custom A (orange). I expected that strengthening penalties would improve accuracy, but the position error actually worsened from 0.0987 to 0.1279. By increasing the distance penalty weight from -0.2 to -0.5, the agent seemed biased toward "avoiding punishment," which slowed its actual approach to the target. The fine_grained reward also dropped dramatically from 0.0229 to 0.0023 in the baseline, possibly because narrowing the tanh kernel's std to 0.05 made the bonus-earning range too small.

On the other hand, Custom B (purple), with the same parameter adjustments as Custom A plus the added joint limit avoidance reward, achieved a position error of 0.0802, surpassing the baseline. The joint limit avoidance seems to have functioned as a kind of regularization, guiding the arm to stay near the center of its range of motion, which also improved target-reaching efficiency as a result. Looking at the logs, joint_limit_avoidance stably earned 0.0461 in rewards, showing its contribution to learning stability.

What I felt from these results is that in reward design, "adding a reward from a different perspective as a supplement" may be more effective than "strengthening existing penalties." Custom A was a "try harder" approach of pushing harder, while Custom B was a "move with a comfortable posture" approach of guiding. The fact that the latter produced more stable learning is interesting in a way that parallels human learning.

Summary of DGX Spark aarch64-Specific Pitfalls

Here's a summary of DGX Spark-specific issues encountered during this environment setup.

I genuinely experienced "half a day lost on environment setup." I hope this table saves someone's time.

Summary

Starting from manually interacting with the robot arm in Isaac Sim's GUI, I ran existing training scripts, read through the reward design code, and observed how learning results changed with custom rewards.

For me personally, reading the reward design code was the most educational. Design patterns like "give a bonus for close-range precision with a tanh kernel" and "gradually strengthen penalties with a curriculum" are hard to really grasp just from reading tutorials. By actually changing parameters and seeing the differences in learning results, I think I've developed an intuitive understanding of how reward functions affect learning.

The DGX Spark ARM64 environment requires considerable effort to set up, but once configured, it delivers a throughput of about 3,500 steps/s in 64 parallel environments. With 1000 iterations finishing in about 9 minutes, it was comfortable to cycle through: change reward design → train → observe results.

Next, I'd like to learn about actual arm assembly, fine-tuning using GR00T N1.5, and the Sim2Real gap from simulation to real hardware.

Continued here

https://dev.classmethod.jp/articles/lerobot-so-arm101-assembly-teleop/

Reference Links

• Isaac Sim GitHub
• Isaac Lab GitHub
• isaac_so_arm101 GitHub
• SO-ARM100 GitHub
• DGX Spark Isaac Playbook (GitHub)
• Install and Use Isaac Sim and Isaac Lab | DGX Spark (Official Guide)
• Seeed Studio: Training SoArm101 Policy with IsaacLab
• Healthcare Robot Demo (CES 2026)
• SoftBank: Physical AI Practice with IsaacLab
• Isaac Lab: Creating a Manager-Based RL Environment
• Isaac Lab: Custom Reward Functions
• Isaac Sim: Basic Robot Tutorial
• Custom reward scripts for this article (GitHub)