AIWeldingRobot:Basic
lastmod 2026-09-30created 2026-02-2438 edits
Basics for developing an AI Welding Robot.
A learning roadmap for approaching the field from the basics to SOTA, from a developer's perspective.
Final goal
Pipeline for building an Korean: 자동 용접 로봇 시스템 (automatic welding robot system) using a Korean: 로봇팔 (robot arm) + Korean: 포인트 클라우드 (point cloud) + Korean: 머신 비전 카메라 (machine vision camera):
- Understanding the 3D data Point Cloud format -> (Open3D, .ply .pcd)
- Simulator (RoboDK)
- Renderer (dedicated viewer)
- 3D camera (Structured Light / Stereo Camera)
- Weld seam detection & segmentation
- Weld path planning
- Hand-Eye calibration (camera coordinates → robot coordinates transformation)
- Robot arm motion planning (MoveIt2, etc.)
- Welding execution + real-time Seam Tracking
- Post-weld quality inspection (defect detection)
Phase 0: Prerequisite check
See 3DSegmentation:Basic#Phase 0: Prerequisite check for details.
Phase 1: Understanding 3D data representations
See 3DSegmentation:Basic#Phase 1: Understanding 3D data representations for details.
Phase 2: Classical 3D Segmentation
See 3DSegmentation:Basic#Phase 2: Classical 3D Segmentation for details.
- RANSAC-based plane segmentation: detect the workpiece planes first → the intersection line of two planes = the weld seam. The most basic method for extracting weld lines
- Euclidean Clustering (DBSCAN): separating individual parts in a scene with multiple workpieces, segmenting candidate seam regions
Phase 3: Deep learning-based Point Cloud Segmentation
See 3DSegmentation:Basic#Phase 3: Deep learning-based Point Cloud Segmentation for details.
- Sparse Convolution-based: WeldNet - a paper that specializes Sparse Conv + RPN for weld seam detection (annular weld seam)
Phase 4: 2D→3D lifting (using Foundation Models)
See 3DSegmentation:Basic#Phase 4: 2D→3D lifting (using Foundation Models) for details.
Phase 5: 3D Gaussian Splatting + Segmentation
See 3DSegmentation:Basic#Phase 5: 3D Gaussian Splatting + Segmentation for details.
Phase 6: Open-Vocabulary & Language-guided 3D Segmentation
See 3DSegmentation:Basic#Phase 6: Open-Vocabulary & Language-guided 3D Segmentation for details.
Phase 7: Welding-specific — Weld Seam Detection & Path Planning
This Phase is the step most directly tied to the final goal. It applies the techniques from Phases 2–3 to the welding domain.
Overall weld seam detection pipeline
The approach most widely used in academia and industry today is the two-stage Coarse-to-Fine method:
- Coarse Stage - rough localization based on 2D images:
- semantic segmentation with YOLOv5/v8/v11-Seg, Faster R-CNN, K-Net, etc.
- Extract the weld seam ROI, detect start/end points
- Fine Stage - precise extraction based on 3D point clouds
- Precise scan of the ROI with Structured light / laser sensor
- Extract plane intersection lines from the point cloud (RANSAC, normal-based)
- B-spline / NURBS curve fitting → generate the weld path
2D-based Weld Seam Detection
- Weld seam object detection: fusion of 2D images and 3D point clouds (2024)
- EfficientWeld - lightweight DNN that detects the seam region and start/end points simultaneously. 36% faster inference
- YOLOv11-Seg - segmentation based on the latest YOLO. Segments workpiece and seam simultaneously
- K-Net - unified Semantic/Instance/Panoptic framework. Segments welding workpieces with transfer learning
- YOLOv5-based - identifies seam position, arc start/end points, and weld type in a single inference
3D-based Weld Seam Extraction
- WeldNet: voxel-based weld seam detection (2024)
- Plane intersection method - detect two planes with RANSAC → intersection line = weld line. The most basic yet powerful
- Concave-Convex Feature Descriptor (Flatness) - extracts seams from the concave/convex features of the point cloud
- WeldNet - a network that detects annular weld seams with Voxel encoding + Sparse Conv + RPN
- DGCNN - detects weld surface defects with a Dynamic Graph CNN
- Farthest Point Sampling (FPS) - curve fitting after keypoint denoising and resampling
Weld path planning (Path Planning)
- Generate a smooth path from the extracted seam keypoints with B-spline curve fitting
- Welding torch pose planning - dihedral angle principle based on the normals of neighboring points
- Teaching-free programming - automatically generates weld paths from vision alone, without teaching
- Multi-seam handling - determining the optimal order based on the TSP (traveling salesman problem)
Key papers
- Weld seam detection: coarse-to-fine path planning - combines 2D EfficientWeld + precise 3D point cloud extraction (2025)
- Weld seam extraction with semantic segmentation and point cloud feature - K-Net semantic seg + Flatness descriptor (2025)
- Weld seam extraction by fusing 2D semantic seg + 3D point cloud - YOLOv11-Seg + NGDT-RANSAC (2025)
- WeldNet: voxel-based weld seam detection - Sparse Conv + RPN for annular seams (2024)
- Weld seam object detection: fusion of 2D + 3D - Faster R-CNN + orthogonal plane intersection (2024)
- Off-line 3D seam extraction and path planning - arc welding path planning based on point cloud segmentation (2020)
Phase 8: Hardware integration - cameras, calibration, robots
Robot simulators
If you have no hardware, you can test with a simulator first.
Choosing a 3D vision camera
| Type | Representative products | Features | Suitability for welding |
| Photoneo PhoXi, Mech-Mind, LMI Gocator | High precision (0.05–0.1mm), robust to ambient light | ⭐⭐⭐ Best | |
| Stereo Camera | Intel RealSense D435i, ZED 2 | General-purpose, inexpensive, real-time | ⭐⭐ For prototyping |
| Laser Line Scanner | Keyence LJ-X8000, SICK Ruler3000 | High-speed line profile scanning | ⭐⭐⭐ Real-time Seam Tracking |
| ToF | Basler blaze, Azure Kinect | Long range, real-time | ⭐ Not precise enough for welding |
Eye-in-Hand vs Eye-to-Hand configuration:
- Eye-in-Hand - camera mounted on the robot end effector. Can scan from various angles while moving. Advantageous for securing a view of the weld.
- Eye-to-Hand - camera installed at a fixed position. Wide field of view, stable. Suitable for large workpieces.
Hand-Eye Calibration ⭐ Required
The key step that links the camera coordinate system and the robot coordinate system. Without it, the robot cannot know "the weld line position the camera sees".
- Mathematical essence: the AX = XB problem (Tsai-Lenz method)
- Eye-in-Hand: robot end effector → camera frame transformation (T_hand_camera)
- Eye-to-Hand: robot base → camera frame transformation (T_base_camera)
- Calibration precision directly determines welding precision. Papers report achieving around 0.2mm
ROS2-based tools:
- easy_handeye - the most widely used Hand-Eye calibration package (a ROS2 version exists)
- handeye_calibration_ros2 - OpenCV Tsai-Lenz implementation, includes RealSense examples
- moveit2_calibration - calibration integrated with MoveIt2
- hand-eye-calibration_ROS2 - automatic calibration with UR5e + RealSense + MoveIt2
Calibration tips:
- Collect data from at least 15–17 different poses
- Make the angles between rotation axes large and the translation distances small
- Keep the camera close to the calibration target
Robot arm control & motion planning
- MoveIt2 - the standard ROS2 motion planning framework. Supports 126+ robots
- MoveIt Servo - real-time jog control. Used for real-time path correction during seam tracking
- Example robot choices: Universal Robot (UR5e/UR10e), AUBO, xArm, FANUC, etc.
- xArm ROS2 - xArm + RealSense D435i integration example (includes Hand-Eye calibration)
Real-time Seam Tracking
During welding, arc heat, deformation, spatter, and so on can shift the seam position, so real-time correction is needed:
- Laser Vision Sensor (LVS) - a laser line sensor mounted in front of the welding torch acquires profiles in real time
- Siamese Network-based seam tracking - tracks the seam across frames
- Adaptive ROI - adjusts the search region of the next frame based on the previous frame's result
- LVS_ST_CALIB - open-source spatiotemporal calibration of a Laser Vision Sensor
Phase 9: Weld quality inspection (Post-Weld Inspection)
The step of automatically inspecting quality from point clouds/images after welding.
Defect types
- Undercut
- Overlap
- Porosity
- Crack
- Uneven bead
Inspection methods
- 3D point cloud comparison - registration of pre-/post-weld point clouds → difference = bead shape analysis
- DGCNN - detects 3D point cloud surface defects with a Dynamic Graph CNN
- 2D vision - UNet- and YOLO-based defect detection on weld bead images
- 3DWS - 3D convolution-based intelligent welding system (23% higher detection rate than RANSAC)
- Use YOLO models.
- mmdetection/mmsegmentation
See also
- 3D Segmentation
- Computer Vision
- Point Cloud
- Deep Learning
- Welding Robot
- Robotics
- SAM3
- Segment Any 3D Gaussians
- 3D scanning
- Image segmentation
- NeRF
- 3D Gaussian Splatting
- Robot arm
- Welding
- Hand-eye calibration
- ROS2
- MoveIt2
- 3DSegmentation:Basic
- Abnormal Detection
- Welding:SurfaceSampleDataGenerator - how to generate virtual "surface" data as ply by taking one line of csv data from LVS that has only the
(x, z)axes and adding multiple lines along theyaxis.
Favorite site
- clij2-docs - 3D Image Segmentation
- Papers With Code - 3D Semantic Segmentation
- Pointcept GitHub
- emergentmind - SAM 3D topic
- Weld seam detection: 2D+3D fusion (Sci Rep, 2024)
- WeldNet: voxel-based weld seam detection (2024)
- easy_handeye - Hand-Eye calibration (ROS1/2)
- handeye_calibration_ros2 - ROS2 Hand-Eye calibration
- xArm_ros2 - xArm + RealSense + MoveIt2 integration example
- LVS_ST_CALIB - Laser Vision Sensor spatiotemporal calibration (welding-specific)