Skip to content
Search

3DSegmentation:Basic

lastmod 2026-09-30created 2026-02-2426 edits

A learning roadmap for approaching 3D Segmentation, from the basics to SOTA, from a developer's perspective.

Phase 0: Prerequisite check

Before starting, check that you are familiar with the concepts below.

Core library setup 

pip install open3d numpy torch torchvision trimesh
  • open3d - the Swiss Army knife of 3D data visualization/processing
  • numpy
  • pytorch
  • torchvision
  • trimesh - Mesh loading/manipulation

Phase 1: Understanding 3D data representations

To understand 3D segmentation, you first need to know "what 3D data is represented with".

Point Cloud ⭐

Basic flow in the welding context:

  1. Scan the workpiece using a structured light camera (e.g. Photoneo PhoXi, Intel RealSense)
  2. Acquire the Korean: 점군 (Point Cloud)
  3. Detect the joint 
import open3d as o3d
pcd = o3d.io.read_point_cloud("example.ply")
o3d.visualization.draw_geometries([pcd])

Voxel

  • 3D space divided into a uniform grid (3D pixels)
  • 3D CNNs can be applied, but memory consumption is high (O(n³))
  • Improved with Sparse Convolution (MinkowskiEngine, SpConv)
  • Welding context - used in WeldNet and others to detect weld joints with voxel encoding + sparse conv

Mesh

  • A surface representation made of vertices + faces
  • Mainly used in CAD, games, and AR/VR
  • Can be processed with a Graph Neural Network

Implicit Representation

  • NeRF (Neural Radiance Fields) - represents a 3D scene as a continuous function
  • 3D Gaussian Splatting (3DGS) - represents a scene as a set of 3D Gaussians
  • Recently emerged as a key backbone for 3D segmentation -> rarely used directly in welding -> useful for following research trends

Range Image / BEV (Bird's Eye View)

  • Projects a LiDAR point cloud onto a 2D image to make use of existing 2D techniques
  • Well suited to real-time processing in autonomous driving

Phase 2: Classical 3D Segmentation

Understanding the traditional methods from before deep learning lets you feel why deep learning was needed.

Region Growing

  • Starts from a seed point and adds similar neighboring points to the group
  • Available out of the box in Open3D and PCL

RANSAC-based plane segmentation

  • Quickly extracts planar structures such as floors and walls

Sample code from Open3D: 

# Step 1: Detect the first plane (첫 번째 평면 검출)
plane1, inliers1 = pcd.segment_plane(distance_threshold=0.01, ransac_n=3, num_iterations=1000)

# Step 2: Remove inliers and detect second plane (두 번째 평면 검출)
remaining = pcd.select_by_index(inliers1, invert=True)
plane2, inliers2 = remaining.segment_plane(distance_threshold=0.01, ransac_n=3, num_iterations=1000)

# Step 3: Intersection of two planes = weld seam line (교차선 = 용접 이음)

Euclidean Clustering (DBSCAN)

  • Separates individual objects with distance-based clustering

Graph Cut / Felzenszwalb

  • Extends image-based graph segmentation to 3D
  • Used for post-processing in SAM3D (Pointcept) and others

Phase 3: Deep learning-based Point Cloud Segmentation

PointNet (2017) ⭐

PointNet++ (2017) ⭐

Sparse Convolution-based

Transformer-based

  • Point Transformer (v1/v2/v3) - applies self-attention to point clouds
  • PointTransformerV3 (PTv3) - currently SOTA on many benchmarks
  • Stratified Transformer - effective at capturing long-range dependencies

Major benchmark datasets

Dataset

Domain

Features

ScanNet

Indoor

RGB-D reconstruction, semantic/instance segmentation

S3DIS

Indoor

Large-scale building scans

SemanticKITTI

Autonomous driving

LiDAR sequences, 28 classes

ModelNet40

Classification

CAD models in 40 categories

ShapeNetPart

Part segmentation

Object part labels

nuScenes

Autonomous driving

Multi-sensor (camera+LiDAR)

Unified frameworks

  • Pointcept - a unified framework for various point cloud models

Phase 4: 2D→3D lifting (using Foundation Models)

The paradigm of extending Foundation Models trained in 2D to 3D.

SAM (Segment Anything Model)

SAM → 3D extension lineage

  • SAM (2D, 2023)
    • SAM 2 (Video, 2024) — adds tracking across frames
      • SAM 3 (Open-vocab, 2025) — text prompts + multiple objects
        • SAM3D (2025) — single image → 3D reconstruction
    • 3D Scene Segmentation derivatives
      • SAM3D (Pointcept, ICCV'23) — 2D masks → 3D back-projection + merging
      • SA3D (NeurIPS'23) — cross-view self-prompting with SAM on NeRF
      • SAMPro3D (3DV'25) — optimizes SAM prompt locations in 3D space
      • SAM-guided Graph Cut (ECCV'24) — 3D instances with SAM + graph cut
    • Medical 3D extensions
      • SAM-Med3D (ECCV'24) - fine-tuning specifically for 3D medical imaging
      • 3DSAM-adapter - adapter-based extension of 2D SAM to 3D volumes
      • SAM3D (ISBI'24) — processes the whole volume at once
    • Native 3D
      • P3-SAM (2025) — PointTransformer backbone, true 3D point prompting
  1. Run the SAM 2D demo first → understand the prompt-mask pipeline
  2. Run the SAM3D (Pointcept) code on ScanNet → understand 2D-3D lifting
  3. Practice NeRF scene segmentation with SA3D

Phase 5: 3D Gaussian Splatting + Segmentation

Basic understanding of 3D Gaussian Splatting (3DGS)

Segment Any 3D Gaussians ⭐

  • Semantically segments individual Gaussians in a 3DGS scene
  • Trained by back-projecting SAM's 2D masks onto the 3D Gaussians
  • Code: Jumpat/SegmentAnythingin3D

Phase 6: Open-Vocabulary & Language-guided 3D Segmentation

The latest trend of specifying the desired target with text for 3D segmentation. -> applying LLMs

Key concepts

  • Open-Vocabulary - can segment even categories not seen during training
  • Grafts Vision-Language Models such as CLIP / SigLIP onto 3D features

Major models

  • OpenScene - assigns CLIP features to 3D points, 3D search with text queries
  • LERF (Language Embedded Radiance Fields)
  • OV-SAM3D - Training-free. Combines SAM + RAM + CLIP
  • Reason3D (2025) - uses an LLM to infer the 3D segmentation target
  • SAM 3 (Meta, 2025) - unified model based on text prompts

Direction of the field

Transferring the capabilities of 2D Foundation Models to 3D is the mainstream approach.

Zero-shot / training-free approaches that work without separate 3D training data are gaining ground.

Phase 7: Domain-specific applications

Autonomous driving

  • LiDAR point cloud semantic segmentation
  • Major models: Cylinder3D, WaffleIron, CENet, RangeViT
  • Benchmarks: SemanticKITTI, nuScenes, Waymo Open Dataset

Medical imaging

  • Organ/lesion segmentation in CT/MRI volumes
  • Major models: nnU-Net (still strong), SAM-Med3D, UNETR
  • Benchmarks: BTCV, BraTS, TotalSegmentator

Robotics/AR

  • Real-time scene understanding
  • 3DGS-based interactive segmentation is on the rise

See also

Favorite site

author Zer0 sitemap