Pointcept:Examples:Weld:LVS:SetupAndOOM

lastmod 2026-04-22created 2026-04-213 edits

Setting up the Pointcept Weld training environment and resolving OOM (Out Of Memory) issues.

Continued from the Pointcept:Examples:Weld:LVS page (a sample that trains on data acquired with LVS).

Overview

A record of resolving, step by step, a series of environment and memory problems that came up while training a custom-built WeldDataset (3-class semantic segmentation of weld regions) with the PTv3 model in the Pointcept project.

Final result: PTv3 completed 100 epochs on an 8 GB RTX PRO 2000 Blackwell (sm_120). Best mIoU 0.3093 (bead IoU 0.9279).

Environment

Problem 1: Missing NCCL symbol

Symptom

ImportError: .venv/.../torch/lib/libtorch_cuda.so:
  undefined symbol: ncclCommWindowDeregister

Cause

• ABI mismatch between the installed torch version and the NCCL package
• Findings:
  • torch-2.11.0.dist-info installed (+cu130 bundle)
  • Companions such as torchvision-0.22.0+cu128 and torchaudio-2.7.0+cu128 are for pt27
  • Both nvidia-nccl-cu12-2.26.2 and nvidia-nccl-cu13-2.28.9 installed
• ncclCommWindowDeregister is a symbol introduced in NCCL 2.27+, so it cannot be resolved with 2.26.2

Fix

Reviewed and tightened requirements.cu128.txt and uv-install-cu128.sh, then rebuilt the venv.

Problem 2: Tightening the install script and dependency pins

Diagnosis

All URLs and wheels were valid, but the pins were loose, so a downgrade could fail:

• The torch==2.7.0 pin can also match the PyPI CPU build (no local version identifier)
• PyG extensions (torch-cluster, torch-scatter, torch-sparse, torch-geometric) versions unpinned
• spconv-cu126 version unpinned
• uv pip install cannot downgrade automatically (--reinstall needed)

Changes

requirements.cu128.txt:

torch==2.7.0+cu128
torchvision==0.22.0+cu128
torchaudio==2.7.0+cu128

torch-cluster==1.6.3
torch-scatter==2.1.2
torch-sparse==0.6.18
torch-geometric==2.7.0

spconv-cu126==2.3.8

uv-install-cu128.sh improvements:

• set -e → set -euo pipefail
• Added an upfront check that uv exists
• Added an upfront warning check for g++-13
• Wrapped read -p with -t 0 to prevent non-interactive hangs
• uv pip install -r → uv pip install --reinstall -r
• Added 10.0 (datacenter Blackwell) to TORCH_CUDA_ARCH_LIST

Problem 3: uv.lock forcing a torch 2.11 pin

Symptom

Even after rerunning the script, torch was still 2.11.0. ./.venv/bin/python -c "import torch; print(torch.__version__)" → 2.11.0+cu130.

Cause

• peft>=0.19.0 in pyproject.toml requires torch as a transitive dep
• uv.lock fixed torch==2.11.0 (PyPI, cu130 bundle)
• When the user ran with uv run --with open3d, uv restored the lock-based environment and reinstalled torch 2.11
• uv pip install --reinstall (pip-compat layer) and uv run (lock layer) are separate systems

Fix

rm -rf .venv uv.lock
./uv-install-cu128.sh

From then on, invoke directly instead of using uv run:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  PYTHONPATH=$PWD:$PYTHONPATH \
  ./.venv/bin/python tools/train.py ...

Problem 4: First OOM — attention upcast

Symptom

File ".../point_transformer_v3m1_base.py", line 203, in forward
    attn = attn.float()
torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 1.56 GiB.
GPU 0 has a total capacity of 7.53 GiB of which 91.12 MiB is free.

Analysis: resolution strategy by priority

By size of accuracy loss: model capacity (channels·depth) >> grid_size (voxel) > patch_size > point_max > upcast flags.

Steps applied (creating cumulative variants)

Created variant configs cumulatively per step in configs/weld/. Kept all past configs as comments in train.sh for quick rollback.

Problem 5: val OOM (softmax 2.55 GiB)

Symptom

Training succeeded for the first epoch, but OOM occurred during Evaluation:

File ".../point_transformer_v3m1_base.py", line 204, in forward
    attn = self.softmax(attn)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.55 GiB.

Cause

The training transform has SphereCrop, but the val transform lacks it, so the entire scene (about 200k+ points) is passed forward. The softmax tensor gets allocated large.

Fix 1 (failed)

Added SphereCrop(point_max=25600, mode="center") to val.

Result: IndexKernel.cu:93: index out of bounds. The inverse indices from GridSample(return_inverse=True) point to the original voxel set, but after SphereCrop the pred size shrinks, causing a mismatch.

Fix 2 (succeeded)

Removed Copy, return_inverse=True, origin_segment, and inverse from the val transform.

The if "inverse" in input_dict branch in evaluator.py is not taken, so evaluation uses only the cropped region. Since this is for monitoring single-scene overfit, the impact is negligible.

Problem 6: backward OOM (allocator fragmentation)

Symptom

At epoch 11, a 100 MiB allocation failed during loss.backward(). Occurred even though expandable_segments:True was on.

Hypothesis testing

• Hypothesis A: empty_cache=True resolves fragmentation between epochs
  • Actual result: OOM at epoch 3 — worse
  • expandable_segments and empty_cache() interfere with each other, causing large contiguous chunk reallocation to fail
• Hypothesis B: keep empty_cache=False + halve point_max (25600→12800)
  • Actual result: completed 100 epochs

Lesson

Mixing PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and torch.cuda.empty_cache() can actually break the effect. Use only one of them.

Problem 7: End-of-training hook failure

Symptom

After 100 epochs of training, at the PreciseEvaluator stage:

AttributeError: 'ConfigDict' object has no attribute 'test'

Cause

PreciseEvaluator is included in hooks in _base_/default_runtime.py, so SemSegTester is built after training ends. The weld config has no data.test section.

Fix

Overrode hooks in the variant config to remove only PreciseEvaluator:

hooks = [
    dict(type="CheckpointLoader"),
    dict(type="ModelHook"),
    dict(type="IterationTimer", warmup_iter=2),
    dict(type="InformationWriter"),
    dict(type="SemSegEvaluator"),
    dict(type="CheckpointSaver", save_freq=None),
    # PreciseEvaluator 제거 — weld는 data.test 없음
]

With a single training scene, test metrics have limited meaning, so removing the hook is more reasonable than adding a proper data.test.

Side task: Inference script

Added a standalone script that uses the trained model to generate predictions for an arbitrary point cloud:

• tools/infer_weld.py: Loads a checkpoint, runs a single forward pass, outputs a colored .ply or int .npy. Uses center SphereCrop (default 12800 points) to fit in 8 GB.
• infer.sh: Wrapper. Sets PYTORCH_CUDA_ALLOC_CONF/PYTHONPATH, auto-selects the default weight and input, supports the WEIGHT env var.

Final config chain

configs/weld/semseg-pt-v3m1-0-base-noattention_nosoftmax-smaller-eighthpoint-valcrop-noinv-nopreceval.py

Cumulative changes:

1. upcast_attention=False, upcast_softmax=False
2. enc/dec_patch_size=256
3. SphereCrop point_max=12800 (both train and val)
4. Removed Copy/inverse/origin_segment from val
5. empty_cache=False
6. Removed PreciseEvaluator from hooks

Training results

Observations:

• Best mIoU peaked mid-run (epoch 54), then dropped due to overfitting
• face1, face2 IoU are always 0 — fundamentally unlearnable from single-scene data
• Until more data is collected, this only means a sanity check that "the training pipeline runs"

Lessons

1. Loose pins block downgrades: Specify the local version with torch==2.7.0+cu128 instead of torch==2.7.0.
2. Don't mix the two uv layers: uv pip and uv run/sync use different dependency sources and can conflict. Use only one per project.
3. Start OOM fixes with the zero-cost ones: upcast flag → patch_size → point_max → grid_size → model capacity.
4. Be careful combining allocator tuning options: Using expandable_segments and empty_cache() together backfires.
5. Inverse mapping in the val transform is incompatible with SphereCrop: Restoring the original resolution and cropping cannot coexist.
6. Cumulative variant file strategy: Creating a new config file at each step and keeping the history as comments in train.sh makes rollback easy. The downside is that file names get long.

See also

• Pointcept

Favorite site

• PyTorch CUDA Memory Management
• spconv (Blackwell support: cu126 wheel + PTX JIT)