I started an MVP for a reinforcement learning project for bottle grasping and lifting with Unitree G1 using Isaac Lab and PPO
This page has been translated by machine translation. View original
Introduction
In previous internship work, I developed a robot visual inspection agent for operator advisory using camera images from the Unitree Go2 and G1.
Starting now, as a new Physical AI project, I am working on reinforcement learning for bottle grasping and lifting targeting the Unitree G1 Inspire Hand.
This series is planned to proceed under the following title:
Isaac Lab and PPO-based Unitree G1 Bottle Pick-and-Lift Reinforcement Learning Project (G1 Bottle Pick-and-Lift Reinforcement Learning project)
In this project, PPO-based reinforcement learning is applied to a fixed standing G1 in Isaac Lab, targeting a bottle-shaped object on a table.
Reach hand toward the bottle
↓
Form grasp pose with Inspire Hand
↓
Lift from the table
↓
Retain
The ultimate goal includes sim-to-real, but currently we are in the simulation-first stage.
Rather than handling walking, autonomous navigation, handoff to humans, and online reinforcement learning from the start, the policy is to first confirm physically valid grasp and lift conditions in Isaac Lab, then build and evaluate a reinforcement learning policy around those conditions.
This article summarizes the content implemented and verified during the first week of the project.
- Isaac Lab RL environment using Unitree G1 Inspire Hand
- Reasons for dividing the bottle grasping task into stages
- Scripted arm trajectory and 12D hand residual PPO
- Adjustment of wrist / thumb / finger contact poses
- AI-assisted diagnosis of contact and physics parameters
- Scripted grasp / lift baseline
- PPO lift / hold showcase with fixed initial state
- Random placement curriculum to be developed next week
Note that this project is independent from the previous Go2 inspection project.
The Go2's camera, detector, dashboard, and VLM / LLM outputs are not connected to the G1's locomotion, arm, hand, or grasp control.
Goal: Stationary Bottle Pick-and-Lift Reinforcement Learning with G1
The task handled in this v1 is for a fixed standing G1 to grasp, lift, and retain a lightweight bottle-shaped object on a table.
fixed standing G1
↓
approach toward bottle-like object
↓
grasp formation
↓
lift
↓
retention above the table
The initial scope is limited to the following.
- fixed-base / fixed standing G1
- Inspire five-finger hand
- camera-free state-based observation
- DDS-free simulation environment
- teleoperation-free
- lightweight bottle-like rigid object
- large arm trajectory is scripted
- PPO handles residual control of the right hand's 12 joints
Online RL exploration on the real robot is not performed.
Also, success in simulation does not directly imply real robot performance or safety.
Why Not Start with 19D End-to-End PPO
Initially, a BottlePickLift task handling both the right arm's 7 degrees of freedom and the right hand's 12 degrees of freedom simultaneously was also prepared.
right arm:
7D
right Inspire Hand:
12D
total:
19D
However, bottle grasping simultaneously includes the following challenges.
arm approach
+ wrist orientation
+ thumb opposition
+ finger contact
+ object retention
+ lift trajectory
If PPO alone is used to explore all of these from the start, it becomes difficult to isolate the cause when failure occurs.
For example, when the bottle is dropped,
- whether the arm position was off
- whether wrist roll was misaligned
- whether the thumb closed too early
- whether finger contact was weak
- whether the lift trajectory was too abrupt
become difficult to determine.
Therefore, in this project the task was divided into stages.
Stage 1:
Create a safe pre-grasp pose
Stage 2:
Verify hand contact and retention
around a scripted arm trajectory
Stage 3:
Expand to broader approach + grasp + lift
Stage 4:
Introduce bounded object-position randomization
Currently, the large arm trajectory is fixed as scripted, and PPO is responsible for 12D residual actions of the right Inspire Hand.
The role of PPO is not to rediscover the entire arm motion from scratch, but to adjust finger contact, anti-slip, and retention around known contact poses.
What Was Accomplished This Week
During the first week of the project, the following were mainly implemented and verified.
- Built a fixed-base RL task using the Unitree G1 Inspire Hand asset
- Built a scene using dynamic object physics and contact sensors
- Confirmed PPO training, checkpoint, deterministic playback, and MP4 recording
- Built a safe pre-grasp task considering self-collision
- Implemented a scripted arm phase controller for safe → ingress → grasp → lift
- Built a PPO task using 12D residual actions for the right hand
- Confirmed failure where the thumb closes early and pushes the bottle out
- Introduced delayed hand activation and minimum pre-shape
- Diagnosed contact poses for wrist roll, thumb yaw, and thumb pitch
- Created a bottle-like rigid object referencing a 600 ml-class form factor
- Implemented slow contact / squeeze / lift trajectory
- Confirmed formal retention-and-lift success with the scripted baseline
- Built a BottleSlowPPO task centered on the scripted baseline
- Ran fresh PPO training with 128 parallel environments and 300 iterations
- Recorded a bottle lift / hold showcase with fixed initial state in deterministic PPO playback
Rather than simply running PPO, the major progress of the first week was being able to confirm the pose before contact is established, hand geometry, actual joint positions, and physical failures, and to gradually organize what should be learned.
Stage 1: Creating a Safe Final-Approach with BottlePreGrasp Curriculum
In the first Stage 1, a BottlePreGrasp curriculum was built to safely bring the G1's right arm close to the bottle as a preliminary step before grasping.
The goal here is not simply to reach a target marker.
It is to create a physically realizable final-approach trajectory that maintains hand orientation and arm pose leading to subsequent grasping, without unnaturally passing through the robot torso.
First, the green target marker position was lowered to a position closer to the lower middle part of the bottle.
This changed the setting from simply reaching for the top of the bottle to aiming for a pre-grasp geometry that leads to future thumb-to-finger opposition.
Initial 7D arm-only PPO
In the initial baseline, a 7D arm-only policy was used to control the right arm's 7 degrees of freedom with PPO.
right arm action:
7D
wrist:
controlled by PPO
With this policy, smooth approach toward the target marker and stopping near it were confirmed.
However, when checking grasp aperture geometry and performing deterministic playback, it became clear that because the wrist action was free, an approach from the thumb-web side with an unnatural posture was permitted.
In other words, it became clear that a reward that only brings the arm closer to the target does not necessarily yield a wrist posture suitable for subsequent grasping.
Failure case 1: Unnatural thumb-web side approach by free wrist policy
Based on this result, a 4D FixedWrist curriculum was implemented that removed wrist yaw, roll, and pitch from the PPO action and controls only the shoulder and elbow.
PPO action:
4D of shoulder / elbow
wrist:
fixed
In the FixedWrist task, behavior of approaching toward the bottle while keeping the wrist pose fixed was confirmed.
However, in deterministic playback after longer training, another problem was found.
A trajectory where the arm uses a self-intersection shortcut that unnaturally overlaps with the torso to reach the vicinity of the target.
Looking only at the target reward it appears the approach succeeded, but in reality it is an unnatural posture that does not lead to subsequent manipulation.
Failure case 2: Self-intersection shortcut passing through the torso
In response to this problem, articulation self-collision was enabled.
With self-collision enabled, trajectories like the previous ones crossing the torso could no longer be executed.
On the other hand, it also became clear that allowing PPO to explore from the default pose alone made it difficult to find a legitimate trajectory that approaches the bottle while avoiding self-collision.
Therefore, it was first confirmed that even with self-collision enabled, the bottle could be approached naturally with scripted actions.
A safe early arm pose was extracted from that trajectory and set as the reset / default pose for the new self-collision curriculum task.
Finally, the Stage 1 task was organized as follows.
task:
BottlePreGraspFixedWristSelfCollision
PPO action:
4D of shoulder / elbow
wrist:
fixed open pose
self-collision:
enabled
reset pose:
safe early arm pose extracted from scripted diagnostic
This task prioritizes the arm approaching the bottle from outside the torso.
In the final stochastic rollout of the best run, the following value was confirmed.
pregrasp_success:
0.5625
This result does not mean success in grasping or lifting.
However, the achievement of Stage 1 is that the RL curriculum was improved to maintain physically realizable trajectories including self-collision, rather than simply maximizing the reward.
Stage 1: self-collision enabled safe pre-grasp demo
Stage 2: Scripted Arm Trajectory and 12D Hand Residual PPO
In Stage 2, rather than leaving the large arm movement to PPO, it is given as scripted phase controller.
Phase 0:
safe pose
Phase 1:
ingress pose
Phase 2:
grasp pose
Phase 3:
lift / hold pose
What PPO outputs is the 12D residual action for the right Inspire Hand.
state observation:
53D
PPO action:
right hand residual 12D
residual scale:
0.08 rad
Conceptually, the configuration is as follows.
scripted arm trajectory
+ scripted hand target
+ PPO hand residual
↓
contact adaptation / anti-slip correction
With this configuration, the policy learns small finger adjustments during contact around the known grasp trajectory, rather than rediscovering the broad arm approach.
First Observed Failure: Thumb Closes Early and Pushes Bottle Away
In the initial hand-only PPO, a failure was observed where the thumb began closing before ingress was complete, pushing the bottle in the lateral direction.
thumb closes too early
↓
object is pushed away
↓
unstable contact / object-out-of-workspace
In response to this problem, the hand action was phase-gated.
before grasp phase:
default / open hand target
from grasp phase:
scripted pre-shape + PPO residual
Furthermore, a clamp was added to prevent the PPO residual from canceling the finger pre-shape and reopening the fingers during grasp / lift phases, ensuring they do not fall below the minimum pre-shape.
target:
max(scripted pre-shape, scripted pre-shape + PPO residual)
With this change, object-out-of-workspace due to early thumb push was greatly reduced.
Failure demo: Thumb closes early and pushes the bottle
AI-Assisted Diagnosis of Contact and Physics Parameters
Adjusting physical contact for robot hands, joint limits, thumb opposition, hand stiffness, and trajectory timing is not my area of expertise.
Therefore, this time AI was used as an assistive technical partner to form physical hypotheses and verify them with diagnostic scripts in Isaac Lab.
However, rather than directly adopting AI suggestions, the following were always confirmed.
AI-assisted hypothesis
↓
create scripted diagnostic
↓
confirm action targets
↓
confirm actual joint positions
↓
observe contact shape in video
↓
confirm termination result
Particularly important was realizing that having a joint target set and having that joint actually move as expected are different things.
During contact, some finger intermediate joints and thumb joints did not reach their targets and were affected by contact geometry and collisions.
Therefore, rather than continuing to increase the target angle alone, the relative positions of the wrist, thumb, and four fingers and the trajectory up to contact were reviewed.
Adjusting Wrist Roll and Thumb Opposition
Upon checking contact, a side-opposition approach where the thumb faces the finger bank from the side appeared more promising than a pose where the thumb pushes the bottle from above.
Representative settings used in scripted diagnostics are as follows.
wrist roll:
-0.25 rad
thumb yaw:
+0.50 rad
thumb pitch:
approximately +0.025 rad
contact close:
four fingers: +0.42 rad
thumb intermediate: +0.32 rad
thumb distal: +0.40 rad
deeper squeeze:
four fingers: +0.70 rad
thumb intermediate: +0.38 rad
thumb distal: +0.46 rad
hand stiffness:
50
These values are not optimal values for the real robot, but values confirmed for the current simulation asset, bottle geometry, and fixed initial state.
Also, since thumb pitch is near the soft joint limit, the effective target after clamping was confirmed in diagnostics, not only the requested target.
Creating a 600 ml-class Bottle-like Object
Initially, a smooth cylinder proxy was also used, but there was a desire to also check contact with a shape that has bottle-like shoulders, neck, and cap.
Therefore, a bottle-like object moving as a single rigid body was prepared.
type:
single rigid-body bottle-like object
approximate total height:
0.225 m
main body diameter:
0.068 m
mass:
0.08 kg
The outer shape references a general 600 ml-class round PET bottle.
However, this is a simulation proxy with an approximated outer shape.
It does not fully reproduce the mass distribution, flexibility, surface friction, or liquid sloshing of a commercially available PET bottle filled with beverage.
Abrupt Close Failure and Slow Contact Trajectory
When switching the hand target abruptly to the close side, the bottle sometimes toppled on first contact.
abrupt close
↓
first contact shock
↓
bottle topples / slips
From this failure, it became clear that not only the final close target but the temporal process of reaching it is important.
Therefore, the scripted grasp sequence was changed as follows.
step 80:
thumb side preposition
step 120–200:
slow contact ramp
step 201–270:
slow squeeze ramp
step 270–300:
squeeze hold
step 300–360:
slow arm lift ramp
step 360 onward:
lift hold
This slow trajectory is the result of verifying in simulation diagnostics the hypothesis, formed with AI assistance, of "reducing contact shock and first establishing stable contact."
Failure demo: Bottle topples due to abrupt close
Slow contact / squeeze / lift demo
Confirming Formal Retention-and-Lift Success with Scripted Baseline
Using the slow contact / slow squeeze / slow lift sequence, the scripted baseline was run on the bottle task with fixed initial state.
In this task, success is not counted just because the object moved slightly or a finger touched it.
The success condition includes requirements such as the following.
lift phase
+ bottle center height exceeds the resting reference by at least 1 cm
+ hand-to-object distance < 0.09 m
+ object uprightness >= 0.85
+ object speed < 0.15 m/s
+ 24 consecutive valid steps
The following were confirmed in the scripted diagnostic.
CONTACT_TARGET_VERIFICATION:
PASS
SQUEEZE_TARGET_VERIFICATION:
PASS
LIFT_TARGET_VERIFICATION:
PASS
retention_lift_success:
True
object_out_of_workspace:
False
maximum bottle-center lift delta:
+0.019683 m
The maximum bottle center lift delta was approximately 1.97 cm.
In other words, while limited to a fixed initial state and single simulator configuration, a scripted simulation baseline was confirmed where the bottle is lifted from the table and retained using the current wrist and thumb pose with slow trajectory.
What was confirmed here is a physically valid baseline for subsequent RL.
Scripted bottle grasp / lift baseline demo
BottleSlowPPO Centered on the Scripted Baseline
After the scripted baseline was established, a task was created to learn a 12D residual hand policy around that trajectory.
task:
BottleGraspRetentionUnderLiftBottleSlowPPO
arm:
scripted safe / ingress / slow lift
hand:
side-open → contact ramp → squeeze ramp
+ 12D PPO residual
object:
same bottle-like rigid object
hand stiffness:
50
PPO is not a policy that finds the bottle and moves the arm extensively.
The current goal is to confirm whether the following actions can be learned around the scripted baseline.
contact adaptation
anti-slip correction
small finger adjustments
retention support during lift
Training was run under the following conditions.
parallel environments:
128
iterations:
300
total timesteps:
921,600
action:
12D right-hand residual
PPO Training and Deterministic Showcase
In the fresh PPO training of 300 iterations, the entire workflow from training, checkpoint saving, deterministic playback, to MP4 recording was confirmed.
In the final stochastic rollout, the following values were obtained.
mean reward:
+1.69
lift_phase_retention:
+0.0586
object_out_of_workspace:
0.0547
retention_lift_success:
0.0000
Formal retention success in stochastic rollout has not yet been confirmed.
On the other hand, when model_299 was run as deterministic playback with fixed initial state, behavior where the bottle was lifted was confirmed.
Furthermore, in a showcase task where only the immediate reset upon success was disabled for demo purposes, a clip of approximately 10 seconds grasping, lifting, and retaining the bottle was recorded while maintaining dynamic physics and PPO inference.
The current primary demo is described as follows.
G1 Inspire-Hand Bottle Lift with PPO Residual Control
Isaac Lab Simulation — Fixed Initial Bottle State
Showcase Evaluation Conditions
The current PPO showcase is run under the following fixed conditions.
robot:
fixed-base G1
object:
fixed initial bottle state
arm:
scripted bounded trajectory
policy:
deterministic PPO checkpoint playback
hand control:
12D Inspire Hand residual action
observation:
camera-free state-based observation
This video does not include random placement, camera input, or real robot evaluation.
Current PPO showcase: lift / hold with fixed initial bottle state
The following are confirmed in this video.
deterministic PPO inference
+ scripted arm trajectory
+ dynamic bottle physics
+ visible grasp
+ tabletop lift
+ visible hold
On the other hand, the following are not demonstrated at this point.
stochastic formal success
random-placement robustness
arbitrary bottle generalization
real robot performance
sim-to-real transfer
Current Configuration
The current pipeline is conceptually as follows.
G1 Inspire Hand asset
↓
fixed-base Isaac Lab scene
↓
scripted safe / ingress / grasp / lift phases
↓
bottle-like dynamic rigid object
↓
state observation (53D)
↓
12D right-hand PPO residual
↓
retention-and-lift termination
↓
checkpoint / deterministic playback / MP4
In the previous Go2 inspection project, camera input, object detection, ROI geometry, and operator advisory were handled.
In this project, rather than directly controlling the robot with camera or LLM, the following elements are the focus in simulation.
robot hand kinematics
contact geometry
rigid-body dynamics
reward / termination design
PPO training
deterministic evaluation
This Week's Results and Current Limitations
As progress for the first week of the project, the following were confirmed.
- Isaac Lab RL environment using G1 Inspire Hand
- Pre-grasp task considering self-collision
- Scripted arm phase controller
- 12D hand residual PPO
- Diagnostic of wrist / thumb / finger contact poses
- Slow contact / squeeze / lift trajectory
- Scripted retention-and-lift baseline with bottle-like object
- PPO training / checkpoint / playback / video workflow
- Deterministic PPO lift-and-hold showcase with fixed initial state
On the other hand, the following remain as future challenges.
- Formal retention success in stochastic rollout
- Quantitative comparison of PPO residual against zero-residual scripted baseline
- Success rate when changing bottle initial position
- Evaluation with held-out offsets
- Mass / friction variation
- Perception integration
- Sim-to-real transfer
- Real G1 evaluation
The current results are not at the stage where "G1 can autonomously grasp a bottle at an arbitrary position."
More precisely, the current stage is as follows.
A physically valid scripted grasp / lift baseline was created for a bottle-like object at a fixed position, and an environment was built where a 12D hand residual PPO can be trained, played back, and visualized around it.
Next Week: Toward Random-Placement Curriculum
Next week, the plan is to proceed to a random-placement curriculum that slightly varies the bottle position on the table, rather than using only a fixed initial position.
Rather than introducing large randomization from the start, the process begins within the range reachable by the current scripted arm trajectory.
The initial plan is as follows.
bottle x:
nominal position ± 0.015 m
bottle y:
nominal position ± 0.015 m
yaw:
fixed initially
mass / friction:
fixed initially
The current policy controls only the hand residual, and the arm is scripted.
Therefore, what the next stage aims to verify is not "can an arbitrary-position bottle be grasped," but rather how well hand contact can accommodate small positional offsets near the nominal grasp geometry.
Also, the zero-residual scripted baseline and deterministic PPO policy with fixed state will be compared to check what influence the PPO residual has on the scripted prior.
After that, the plan is to proceed to an environment where success rate, termination distribution, and videos can be evaluated separately using held-out offsets.
Summary
This time, a reinforcement learning project handling bottle grasping and lifting in Isaac Lab targeting the Unitree G1 Inspire Hand was started.
During the first week of the project, the following workflow was built.
safe pre-grasp
↓
scripted arm phases
↓
AI-assisted contact geometry diagnosis
↓
slow contact / squeeze / lift baseline
↓
formal retention-and-lift validation
↓
12D hand residual PPO
↓
deterministic playback / showcase video
In particular, for robot hand tasks involving contact, the importance was realized of not simply running PPO for a long time, but of using AI as an assistant to confirm wrist roll, thumb opposition, finger targets, stiffness, and trajectory timing one by one with diagnostics.
At the current stage, a scripted baseline with fixed initial state and a deterministic lift / hold showcase incorporating PPO have been confirmed.
Next week, a small bounded random-placement curriculum will be introduced, gradually expanding beyond fixed-position conditions.
Without treating simulation success as equivalent to real robot performance or safety, the G1 bottle grasping and lifting reinforcement learning project will be progressed step by step, separating task scope, evaluation conditions, and failure cases.