Pointcept:Arm64

lastmod 2026-09-23created 2026-08-2021 edits

Notes on issues that come up while building and using Pointcept on ARM64 (specifically on NVIDIA DGX Spark).

How to build

This covers running natively on DGX Spark (GB10 = sm_121, aarch64). In this case QEMU is not needed at all (if arch == host_arch it passes straight through), and Docker is all you need.

System specs

• NVIDIA DGX Spark (GB10 = sm_121, aarch64)
• NVIDIA-SMI 580.173.02
• NVIDIA Driver Version: 580.173.02
• CUDA Version: 13.0
• Ubuntu 24.04.4 LTS (noble)
• Python 3.12
• Torch 2.9.1 (the minimum version in the cu130 index is 2.9.0)

Source code

• Uses https://github.com/cvprun/Pointcept. (Forked from the original code - https://github.com/pointcept/pointcept)

Simple way to run it all in one go

arm64+cu130+torch2.9.1+py3.12+sm_121

./scripts/build_wheels.sh build --arch arm64 --accel cu130 --torch 2.9.1 --python 3.12 --cuda-arch 12.1 --jobs 8 -v
./scripts/verify_wheels.sh wheelhouse/linux-arm64/cu130/torch2.9.1-cp312/

arm64+cu130+torch2.13.0+py3.12+sm_121

./scripts/build_wheels.sh build --arch arm64 --accel cu130 --torch 2.13.0 --python 3.12 --cuda-arch 12.1 --jobs 8 -v
./scripts/verify_wheels.sh wheelhouse/linux-arm64/cu130/torch2.13.0-cp312/

Pinning down the target

DGX Spark's sm_121 can only be code-generated with CUDA 12.9 or later. In the script's cuda_supported_arches, cu128 only goes up to 12.0, so 12.1 fails at the validation step. Therefore:

It is convenient to group these into common variables:

cd /path/to/Pointcept
BW=./scripts/build_wheels.sh
T="--arch arm64 --accel cu130 --torch 2.9.1 --python 3.12 --cuda-arch 12.1 --jobs 8 -v"

• --jobs defaults to nproc (max 16), but DGX Spark shares 128GB of unified memory with the GPU. nvcc uses ~2.5GB per job, so lowering it to 8-10 is recommended.
• -v streams the compiler output. Essential if you want to watch a build that takes hours.

Check before starting (builds nothing):

## 조합·패키지 목록 출력:
$BW matrix $T

## 실제 실행될 docker 커맨드 출력:
$BW build $T --dry-run

Step-by-step build

The key is --only {패키지}. The output directory and the pip/uv cache volume (pointcept-build-cache) are shared across runs, so even if you split the runs, the results accumulate in one place.

wheelhouse/linux-arm64/cu130/torch2.9.1-cp312/

manifest.txt records (by appending) whether each wheel was downloaded prebuilt or compiled.

• Dependencies are pulled in automatically. Passing just --only spconv builds all three in the order pccm → cumm → spconv, so if you want to merge steps 2-3, running step 3 alone is enough.

Each step runs in a separate container and redoes "apt toolchain install + uv venv + torch install" from scratch every time (a fixed overhead of about 2-3 minutes). Thanks to the cache volume, from the second run onward it is a cache copy rather than a download.

Two things to know on sm_121

cumm/spconv

cumm has its own supported_arches table (cumm/common.py), and 12.1 is not in it. The script automatically maps it to 12.0+PTX (cumm_arch_for), so PTX is shipped instead of cubin and the driver JIT-compiles it on first load.

If you see a log like this during the build, everything is fine:

-> cumm arch set: 12.0+PTX (from 12.1)

flash-attn

flash_attn_arch_for only knows the values 80/90/100/120; 12.1 has no mapping, so the package itself is skipped:

!! flash-attn skipped: no target in '12.1' has flash-attn 2.x kernels

On Blackwell, sm_120 cubin does not load on sm_121 devices, so this is a real constraint rather than a script bug. Skip this step and carry on; if you really need flash-attn, it has to be checked separately.

If you want, I can tweak the script to pass 12.0+PTX and check whether the PTX path actually works.

After completion

cat wheelhouse/linux-arm64/cu130/torch2.9.1-cp312/manifest.txt
pip install wheelhouse/linux-arm64/cu130/torch2.9.1-cp312/*.whl

Good to know

• The build does not use the GPU at all (no --gpus flag, nvcc cross-compilation). The driver/GPU check shown at startup only tells you "whether this wheel will load on this machine"; it does not block the build.
• The repo is mounted read-only at /src, and libs/ is copied to /tmp inside the container before being built, so no build/ or *.egg-info appears in the working tree.
• Even on failure, it is wrapped in set +e, so the last 40 lines of that package's log are printed. The full log is /tmp/<package>.log inside the container, but the container is --rm so it disappears; if you need to look closely, enter the same environment with $BW shell $T and run it manually.
• If you feel like running everything in one go, use $BW build $T --keep-going — it skips failed packages and goes to the end.

Troubleshooting

Pointcept (PTv3) training crash : cudaErrorIllegalAddress

PTv3 training on a DGX Spark node died with cudaErrorIllegalAddress. The error was always reported in spconv's pair generation, but the iteration at which it died and the scene shape differed from run to run.

While ruling out hypotheses one by one, at one point torch 2.13.0 was wrongly confirmed as the cause. A production run with torch downgraded to 2.9.1 died the same way, so that conclusion was withdrawn.

Finally, a production run was done with spconv_native: true, which removes the implicit gemm path entirely, and it ran the full 100 epochs (2,500 iter). This switch is now on by default.

Symptoms

[Exception|implicit_gemm_pair]indices=torch.Size([49093, 4]),bs=1,ss=[757, 621, 546],
  algo=ConvAlgo.MaskImplicitGemm,ksize=[3, 3, 3],...,subm=True,transpose=False
  File ".../spconv/pytorch/ops.py", line 550, in get_indice_pairs_implicit_gemm
    SpconvOps.sort_1d_by_key_allocator(pair_mask_tv[j],
RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress:
  an illegal memory access was encountered

• merge_sort: failed to synchronize means thrust noticed, late, an invalid memory access by another kernel that ran earlier. So the traceback points not to where the fault occurred but to the first synchronization point after it.
• Where the production runs died:

Investigation log

Lessons from the wrong conclusion

Hypotheses 5, 6, and 7 were all judged on results from running each configuration only once. If a fault blows up at a different place each time, passing once may just be luck. In fact, the pass on the 2.9.1 side was luck. The experiment with real discriminating power was removing the faulty path entirely (hypothesis 8).

The torch version of the production image was confirmed from the line numbers in the traceback. The _wrapped_call_impl line in module.py is 1775 in 2.9.1 and 1778 in 2.13.0.

Key facts

• cumm 0.8.2 does not accept 12.1. supported_arches in cumm/common.py ends at "12.0". So spconv for sm_121 is built as 12.0+PTX and the driver JIT-compiles it. torch itself runs the same way with compute_120 PTX.
• spconv stores all weights as KRSC regardless of algorithm (ALL_WEIGHT_IS_KRSC=True). So it is safe to switch to ConvAlgo.Native after the model is built. The hook verifies this assumption with an assert.
• The cost of Native was about 4% on an sm_120 laptop.

Fixes

Troubleshooting PointGroup instance segmentation training

Written 2026-09-23. Target: instance_segmentation / pointcept / pg-ptv3 (PointGroup PG-v1m2 + PTv3 backbone), GB10 (sm_121) single GPU, 62 user PLY files (50 train / 12 val), single class.

The plan document is the Phase 6 progress notes in docs/plans/50_pointgroup_instance_segmentation_plan.md, and this file collects only the problems encountered during execution.

1. At a glance

The single most important fact confirmed in round 3: bfs_cluster in pointgroup_ops actually runs on sm_121. It was the biggest risk in the plan, and there is no longer any need to prepare a CPU fallback.

2. Round 1: no np.in1d

mask = ~np.in1d(segment, self.segment_ignore_index)
AttributeError: module 'numpy' has no attribute 'in1d'

np.in1d was removed in NumPy 2.0. The numpy in the image is 2.5.2. There are only three places in the Pointcept fork that use this function, and all of them are in the instance path (InstanceParser, InsSegEvaluator, tester). That is why semantic segmentation ran 100 epochs to completion with the same image without ever hitting this error.

The fix is pointcept_semseg.patch_numpy_compat(). It restores np.in1d with np.isin. All three places use 1-D label arrays, so the two are equivalent. The dispatcher calls it before choosing a trainer, and pointcept_insseg.main() also calls it once at startup. The fork has no other aliases removed in NumPy 2 (checked exhaustively).

As a side note, the test for this fix was using patch.dict(sys.modules, ...), which on exit restores the whole dictionary and so deletes modules first imported inside the block. After that there were two copies of numpy, and an unrelated test died with _NoValueType. It was replaced with _stub_module, which changes and restores just one key.

3. Round 2: CUDA device-side assert in backpropagation

ScatterGatherKernel.cu:163 ... scatter gather kernel index out of bounds   (수백 줄)
spconv indice_conv_backward ... dev2host ... error 710

Data preparation and the first forward pass succeeded. CUDA errors are asynchronous, so where it blew up (spconv backpropagation) and the culprit kernel are different. Neither the Pointcept nor the PointGroup code calls ATen gather/scatter directly (only comm.gather and torch_scatter.segment_csr), and the same combination of backbone, spconv, and data has run 100 epochs of semantic segmentation to completion before. Reading the code could not narrow it down further, so instrumentation was added.

• Hyperparameter cuda_launch_blocking (off by default). When on, the trainer sets CUDA_LAUNCH_BLOCKING=1 before importing torch so the traceback points to the real kernel. The trade-off is that it gets slower.
• describe_scene() checks the segment and instance ranges for each scene. If a class index exceeds the model's number of classes, or there are "points that have a class but no instance", it fails before training starts. It logs statistics for the first three scenes and a list of scenes with no instances at all.

It was not reproduced in round 3 (run with blocking on). It was treated as a transient glitch in the asynchronous path and only the instrumentation was kept. If it shows up again, the order is: get a traceback with cuda_launch_blocking: True → batch_size: 1 (rule out scene-boundary problems) → enable_amp: False.

4. Round 3: completed, but AP is all 0

seg_loss: 0.0000   bias_l1_loss: 150 ~ 225
Val result: mAP/AP50/AP25 0.0000/0.0000/0.0000
Best validation AP50 updated to: 0.0000

seg_loss: 0.0000 is normal. With only 1 class, the semantic loss is constant and seg_head has nothing to learn. At inference, pred_classes is always 0.

The problem is bias_l1_loss. These are values from re-measuring the 12 user PLY files.

The 150-225 in the log is exactly this 172. In other words, the offset head was sitting right at the loss of a model that has learned nothing. PointGroup's offset head regresses, for each point, "the displacement to my instance center", pulls the points together using that displacement, then clusters them with BFS to make proposals. If the offsets don't converge there are no proposals, and with no proposals AP is 0.

The cause is coordinate scale. PointGroup's clustering radius (cluster_thresh) and Pointcept's augmentation values (RandomJitter sigma=0.005 clip=0.02) are numbers tuned for meter-scale indoor scenes (0.02 m voxels). We were feeding in data at 100× that scale as is. In a 749-unit scene, a jitter of 0.005 does nothing. On top of that, only 2 epochs were run.

Fix

Staging applies only a scale factor to the coordinates. It scales and saves the data so that the user-chosen grid_size (in data units, default 0.6) becomes TARGET_VOXEL = 0.02, and the generated config's grid_size and voxel_size use that value.

• The voxel grid stays the same. The same points are grouped into the same voxels. The actual distance covered by cluster_thresh is also still 0.9 data units. The only things that changed are the size of the regression target (172 to 5.7) and the size of the augmentation.
• The scale factor is carried in the model as model_config["coord_scale"]. The serving pyfunc applies the same factor, so it sees the same scale as training. The output is per-point IDs, so it is not affected by the coordinate scale.
• Staging logs mean |offset| to the instance centre.

5. How to read a healthy log

What to check in the staging section.

-> scene 10_final: {'points': 45470, 'instances': 3, 'labelled': 45470, 'classes': [0]}
-> coordinates scaled by 0.0333333 so the voxel is 0.02 ...
-> mean |offset| to the instance centre: 5.7 (PointGroup's bias_l1_loss starts near this; ScanNet sits at 1-2)
-> prepared 50 train / 12 val scene(s), 249 instance(s)

• The more scenes have 0 instances, the less the offset loss can learn. These are reported on a separate line.
• If labelled is much smaller than points, there are many unassigned instances. Fill them in with the assignment table on the dataset screen or with a CSV.
• If mean |offset| is a single digit, the scale is right. If it is three digits, you are in the situation described in section 4 of this document.

What to check in the training section.

• bias_l1_loss should start near the mean |offset| above and go down. If it stays flat, training is not working.
• seg_loss: 0.0000 is normal for a single class.
• The evaluator picks the best checkpoint by AP50. Not mAP.
• The MLflow metric names are metrics/mAP, metrics/AP50, metrics/AP25.

6. What to tweak, in order, when AP is still low

1. Epochs. 2 epochs give you nothing. Start looking from 50-100.
2. cluster_thresh (default 1.5, in voxels). Raise it if there are no proposals at all; lower it if all objects merge into one.
3. cluster_min_points (default 50), cluster_propose_points (default 100). Lower them if small objects get thrown away entirely.
4. batch_size: 1, enable_amp: False. Only when narrowing down unstable runs.

62 files with 2-5 instances each is a small dataset. A ScanNet-pretrained backbone is used, but there is plenty of room for overfitting, and with only 12 validation files the variance of AP is also large. It is better not to draw conclusions from a single number.

7. Changes that require republishing

The trainer and pyfunc live inside the training runtime image. That is, code changes only take effect in runs once the runtime is republished. The changes that have required republishing so far:

• Added the pointgroup_ops wheel (Phase 0)
• Restored np.in1d (round 1 fix)
• cuda_launch_blocking and scene-check instrumentation (round 2 fix)
• Coordinate scale factor (round 3 fix)

The serving image uses the training image as its FROM, so only models built after a new training image is published are served with the new pyfunc. Serving of already registered semantic segmentation models is unaffected. The pyfunc branches on geo_task, and the default is semantic segmentation.

8. File locations

See also

• Pointcept

Favorite site

• Github - cvprun/Pointcept