Skip to content
Search

Robotic arm

lastmod 2026-07-27created 2026-02-139 edits

A robotic arm is generally a type of programmable mechanical arm with functions similar to a human arm.

The suitable model depends on the purpose and method of robotic arm control. Below is a summary by main approach.

Reinforcement Learning based

The most widely used approach.

  • Stable Baselines3 (SAC, PPO, TD3) - Integrates well with ROS2 and Gymnasium (formerly OpenAI Gym), and is suited to control in continuous action spaces (joint angle/torque). SAC (Soft Actor-Critic) in particular shows the most stable performance in robotic arm control.
  • NVIDIA Isaac Lab (formerly Orbit) - GPU-accelerated simulation enables large-scale parallel training, and it has a well-equipped sim-to-real transfer pipeline. Also supports a ROS2 bridge.

Imitation Learning based

An approach that learns from motions demonstrated by humans.

  • ACT (Action Chunking with Transformers) - Learns fine manipulation tasks from teleoperation data. Validated in the ALOHA project.
  • Diffusion Policy - Applies diffusion models to robot policies; represents multimodal action distributions well and is strong at complex manipulation tasks.

Foundation Model based

A recent trend that uses large-scale pretrained models.

  • π₀ (Pi-Zero, Physical Intelligence) - A general-purpose VLM-based model that takes language commands and controls a robotic arm. Can be applied to various tasks through fine-tuning.
  • RT-2 / Octo - Google DeepMind's robot transformer models, unifying vision, language, and action. Octo is released as open source, so it is easy to access.
  • GR00T (NVIDIA) - A foundation model focused on humanoids but also applicable to manipulation.

Traditional motion planning + AI hybrid

  • MoveIt2 + ML - A practical approach that uses MoveIt2's motion planning as the base and adds AI models only for grasp pose prediction or obstacle avoidance. The most stable option in industrial settings.

Point cloud based AI models for robotic arm control

Using point clouds as input lets you use 3D spatial information directly, which widens the options.

Grasp Pose prediction (object grasping)

The most practical area, applicable right away.

  • AnyGrasp / GraspNet - Predicts 6-DoF grasp poses directly from point clouds. Works on unseen objects, and community ROS2 wrappers exist. Most recommended starting point
  • Contact-GraspNet - A model released by NVIDIA that generates contact-based grasps from single-view point clouds. Easy integration with Isaac Sim
  • VGN (Volumetric Grasping Network) - Converts to a TSDF volume and then predicts grasp quality. Strong in cluttered environments.

Point cloud feature extraction backbones

3D encoders that the models above are built on.

  • PointNet++ / PointNext - The most proven point cloud processing networks. Capable of real-time processing, with plenty of references in robotics. Standard backbone
  • Point Transformer v3 - Transformer-based with high accuracy. Trade-off with inference speed
  • MinkowskiEngine (Sparse Conv) - Based on 3D sparse convolution, efficient on large point clouds. Large-scale data processing

Point cloud + manipulation policy (End-to-End)

Recent models that use point clouds directly as policy input.

  • DP3 (Diffusion Policy 3D) - A Diffusion Policy that takes 3D point clouds directly as input; being independent of camera view makes sim-to-real transfer easy. The model most worth watching
  • PerAct / PerAct2 - Converts point clouds to voxels and processes them with a Perceiver Transformer. Can be combined with language commands and performs various manipulation tasks with a single model. Can combine language commands.
  • Act3D - Predicts actions over a 3D feature field, with higher resolution than PerAct, which favors precision tasks. Precision tasks.
  • 3D Diffuser Actor - Recent research combining 3D scene representation with diffusion models. Recent research.

Point cloud based scene understanding

  • SAM3D / OpenScene - Performs open-vocabulary segmentation on 3D point clouds. Useful for identifying objects when handling natural language commands like "pick up the red cup". Natural language object identification.
  • FoundationPose - NVIDIA's 6-DoF object pose estimation model; works on point clouds/depth and can be used even without a CAD model. No CAD model needed.

Pipeline diagram 

포인트 클라우드 (ROS2 sensor_msgs/PointCloud2)
        │
        ├─ 간단한 pick-and-place ──→ AnyGrasp + MoveIt2
        │                            (빠르게 구현 가능, 산업용에 적합)
        │
        ├─ 복잡한 조작 작업 ────→ DP3 (Diffusion Policy 3D)
        │                          (비정형 작업, 높은 범용성)
        │
        └─ 언어 명령 기반 ─────→ PerAct2 + SAM3D
                                   (자연어로 작업 지시)
  1. First implement pick-and-place with the AnyGrasp + MoveIt2 + ROS2 combination.
  2. Then extend to DP3 or PerAct2 to handle complex tasks.

Recommendation summary by task type

Task type

Recommended model

Difficulty

Pick-and-place (structured)

AnyGrasp + MoveIt2

★★☆☆☆

Pick-and-place (unstructured)

Contact-GraspNet + MoveIt2

★★★☆☆

Complex manipulation (assembly, etc.)

DP3 (Diffusion Policy 3D)

★★★★☆

Natural language command based control

PerAct2 + SAM3D

★★★★☆

General-purpose robot control

π₀ / Octo

★★★★★

See also

Favorite site

author Zer0 sitemap