Pointcept:BuildMatrix

lastmod 2026-09-28created 2026-09-281 edits

Pointcept prebuilt wheel matrix (amd64 · arm64)

Written: 2026-09-28 · Target: the pointcept-wheelhouse package of the app.cvp.run Pointcept agent app

Overview

The Pointcept agent app builds its environment only from the pointcept-wheelhouse uploaded to the registry (roadmap decision R5). If there is no wheelhouse matching the host, it does not fall back to a source build; it stops in a blocked state. So for the agent to work with downloads alone, a wheelhouse must be prebuilt and uploaded for each host combination.

The PyPI and PyG indexes are mostly missing wheels for CUDA 13 (cu130) and for arm64. This page lays out the amd64·arm64 matrix to prebuild with scripts/build_wheels.sh, and the constraints confirmed by actually building and running.

Conclusion: Rather than making many combinations, building one wheelhouse per accel with a wide cuda_arch gives the widest coverage. 3–4 for amd64 and 3–4 for arm64 are enough.

The agent's wheelhouse selection rules

Based on wheelhouseFits·selectWheelhouse in app.cvp.run/cli/src/pointcept/packages.ts.

Recommended build matrix

Every row is python 3.12, torch 2.9.1.

† 7.5 (Turing) can only be added after the agent is fixed first. See flash-attn.

Combinations recommended for exclusion

Confirmed constraints

torch wheel index

List of cp312 wheels in download.pytorch.org/whl/<accel>/torch/ as of 2026-09-28.

The versions common to cu126·cu128·cu129·cu130 and both arches are 2.9.x, 2.10, 2.11. Of these, only 2.9.1 is verified.

torch lower bounds filtered by the builder (cuda_min_torch): cu126 ≥ 2.6.0, cu128 ≥ 2.7.0, cu129 ≥ 2.8.0, cu130 ≥ 2.9.0, cu132 ≥ 2.12.0.

PyPI · PyG prebuilt gaps

Therefore arm64 compiles all 12 native packages from source. Only the pure Python packages pccm, torch-geometric, and ocnn use prebuilts.

Per-package pins and patches

flash-attn

• flash-attn 2.x only produces cubins for sm_80, 90, 100, 120. 8.6·8.7·8.9 run on the sm_80 cubin.
• 10.3 and 12.1 are in a different Blackwell family, so the cubins do not load. The builder builds with 100+PTX, 120+PTX so they are reached via driver JIT (commit c35c4d4, 2026-09-10).
  • It builds and works on DGX Spark (sm_121). Since 2026-09-11 it has been recorded in the manifest as compiled flash_attn-2.8.3.post1-…-linux_aarch64.whl, and on 2026-09-21 it completed 60 epochs of training with enable_flash=True.
  • Before 2026-09-10 there was no arch matching 12.1, so it was silently skipped.
• Builds targeting only 7.5 (Turing), 11.0 (Thor), or Orin have nothing to compile and are skipped.
• The agent decides whether to use flash only by whether import flash_attn succeeds (scripts.ts). So if you put 7.5 in the tag of a wheelhouse containing flash-attn, flash turns on on T4 and fails. To add 7.5, the agent must first be fixed to enable flash only when capability ≥ 8.0.
• PTX is JIT-compiled on first load, so CUDA_CACHE_PATH must point to a writable path to pay the JIT cost only once.
• It is the package with the longest build time. Building just 12.1 on GB10 took about 18.7 minutes. Since there are at most 4 cubins, widening the arch list does not increase the cost endlessly.

glibc compatibility

Updated 2026-09-28: the build_wheels.sh default changed to --os manylinux_2_28, which resolves this issue. Below are measurements from the earlier ubuntu24.04 builds (including 2026.09.11, currently published in the registry). See manylinux_2_28 build.

The current wheels are built in an ubuntu24.04 container. The maximum symbol versions confirmed with objdump are as follows (amd64 cu130).

On Ubuntu 22.04 hosts (glibc 2.35, GLIBCXX 3.4.30), importing cumm·spconv fails. The agent's os tag is only linux, so it cannot filter out such hosts in advance. There are two alternatives.

1. Build with --os ubuntu22.04. Wheels built on 22.04 also run on 24.04. Not yet tried.
2. Add the glibc version to the tag and check it in wheelhouseFits.

Prebuilts and tag accuracy

When prebuilt spconv·cumm·PyG are downloaded on amd64 as in A1·A2, the arch set of those wheels is decided upstream and may differ from the cuda_arch tag. To make the tag exact, use --force-source. The build time increases in exchange.

Upload size

Widening the arch list makes the cumm·spconv wheels larger. First check whether the Workers CPU limit problem in the admin package upload relay (pointcept-agent-app-roadmap.md) has been resolved.

How to build

amd64 is built directly on an amd64 host and arm64 directly on DGX Spark. QEMU cross builds are not used because nvcc is 10–30x slower. Keep the output directory separate so it does not mix with existing single-arch results. If the same directory is used, manifest.txt gets appended to.

The two loops below are in the repository as scripts/build_matrix_amd64.sh and scripts/build_matrix_arm64.sh. Output goes to wheelhouse-wide/ (git-ignored), and trailing arguments are passed to build_wheels.sh (e.g. --jobs 8, --dry-run).

# amd64
for spec in "cu126|8.0 8.6 8.9 9.0" \
            "cu128|8.0 8.6 8.9 9.0 10.0 12.0" \
            "cu130|8.0 8.6 8.9 9.0 10.0 10.3 12.0"; do
  bash scripts/build_wheels.sh build --arch amd64 --accel "${spec%%|*}" \
    --torch 2.9.1 --python 3.12 --cuda-arch "${spec#*|}" \
    --force-source --out wheelhouse-wide --keep-going -y
done

# arm64 (DGX Spark)
for spec in "cu126|8.0 8.6 8.9 9.0" \
            "cu128|8.0 8.6 8.9 9.0 10.0 12.0" \
            "cu130|8.0 8.6 8.9 9.0 10.0 10.3 12.0 12.1"; do
  bash scripts/build_wheels.sh build --arch arm64 --accel "${spec%%|*}" \
    --torch 2.9.1 --python 3.12 --cuda-arch "${spec#*|}" \
    --out wheelhouse-wide --keep-going -y
done

To include A3·B3 (cu129), add "cu129|… 10.3 12.1" in the same format.

Verification and publishing

1. Check installation, import, kernels, and PTv3 forward with scripts/verify_wheels.sh <wheelhouse 디렉터리>.
2. If possible, check through training with scripts/verify_mlops.sh.
3. Publish each combination to the registry with a unique version and attach tags.

{"os":"linux","arch":"arm64","accel":"cu130","torch":"2.9.1","python":"3.12",
 "cuda_arch":"8.0 8.6 8.9 9.0 10.0 10.3 12.0 12.1"}

Only two have been verified on real GPUs: sm_120 (RTX PRO 2000) and sm_121 (GB10). The other arches have only been compiled, so note "unverified" in the package description.

Runtime conditions: is matching arch and GPU enough?

No. Even assuming the whole matrix above is built, there are more host-side conditions, and some GPUs need agent code changes to run properly even when the wheels match.

Host conditions

GPUs that require agent changes first

Verification status

"Built" and "runs" are different. The only combinations confirmed end-to-end on real GPUs are amd64 cu130 sm_120 and arm64 cu130 sm_121 (in mlops, with Native pinning). For the other arches, it is only confirmed that nvcc compiled them successfully. 8.0–9.0, 10.0, and 12.0, which are built as cubins, are low risk, but 10.3·12.1, which rely on PTX JIT, need verification on real hardware.

Summary

On a host with Linux, driver CUDA ≥ accel, glibc 2.28 or later (for manylinux builds), and internet access, installation and inference work as long as arch·GPU match. However, two things are needed first.

1. GB10 and B300: spconv Native pinning must be added to the agent.
2. Ubuntu 22.04·RHEL hosts: the wheelhouse must be rebuilt with manylinux_2_28 and published. The currently published 2026.09.11 is an ubuntu24.04 build.

Single-bundle distribution (download once and run without sudo)

Everything the agent uses for training and inference can be packed into one per (arch, accel), so that it runs just by downloading and extracting. The NVIDIA driver and glibc cannot be put in the bundle and must be on the host. For the format, a compressed archive (tar.zst) per (arch, accel) is recommended over AppImage.

What can and cannot go in the bundle

Why AppImage is not recommended

• AppImage must be mounted with FUSE to run. On hosts without libfuse2, sudo apt install libfuse2 is required, which defeats the purpose. Containers often lack /dev/fuse. Using --appimage-extract-and-run ends up extracting every time anyway.
• The bundle is around 5 GB, so there is no benefit to mounting squashfs on every run.
• The agent already has the necessary structure. The registry's archive type, the CLI's extractAll, version markers, and the ~/.cache/cvp/<app> root can be used as-is. Download the archive once, extract it, and run with bin/python3; no sudo, FUSE, or network (PyPI, download.pytorch.org) is needed.

Recommended layout

pointcept-runtime-<arch>-<accel>.tar.zst   # 예: arm64-cu130
└─ runtime/
   ├─ python/          # python-build-standalone 3.12. venv가 아니라 이 python의 site-packages에 직접 설치
   │  └─ lib/python3.12/site-packages/   # torch, nvidia-*, Pointcept 휠, 순수 의존성
   ├─ lib/             # 호스트보다 새 공유 라이브러리가 필요할 때만
   ├─ src/pointcept/   # 고정 커밋 소스
   ├─ scripts/         # 추론·학습 워커
   └─ env.sh           # PYTHONPATH, LD_LIBRARY_PATH, TRITON_CACHE_DIR, CUDA_CACHE_PATH

• Do not use a venv. Absolute paths are baked into pyvenv.cfg and script shebangs, so moving it breaks them. Always run via python/bin/python3 -m … or the script path.
• Size: the agent's current rfdetr·sam2 venvs are 4.7–5.3 GB extracted (torch 1.1 + nvidia 2.7 + triton 0.7 GB). Pointcept adds about 0.2 GB to this, so it is expected to be about 2.5–3 GB compressed. For the 6 combinations in the recommended matrix, the total is about 15–18 GB.
• Do not put weights in the bundle. They are 555 MB per model and optional, so keep them as separate packages as now.
• Reuse: the torch and nvidia parts (about 4.5 GB) are the same as rfdetr·sam2. If multiple apps are later moved to bundles, splitting into two layers, a "torch base per (arch, accel)" and an "app layer", lets them share downloads and disk. Currently the python versions differ (rfdetr·sam2 use 3.11, Pointcept uses 3.12).

Remaining host conditions

1. Linux, and an NVIDIA driver matching (arch, accel). nvidia-smi must work.
2. At least the glibc lower bound set by the build base (2.28 for manylinux_2_28)
3. About 8 GB or more of disk (the archive and the extracted result)
4. GB10·B300 need spconv ConvAlgo.Native pinning in the agent scripts. This is a code problem unrelated to the bundle.

Sticking points

• Upload: uploading multi-GB files to the registry requires the Workers CPU limit problem in the admin upload relay to be solved first. Switching to presigned direct uploads is one option.
• Verification method: run everything from download and extraction through inference·training on a base image with nothing installed. This means a state where only the driver is injected via nvidia-container-toolkit, such as docker run --gpus all ubuntu:22.04, debian:12, rockylinux:9. Whether Triton finds a compiler also shows up at this stage.

manylinux_2_28 build

manylinux is not a way to remove the glibc dependency. It standardizes the approach of building on an old glibc to lower the lower bound. build_wheels.sh now uses this as its default (--os manylinux_2_28).

What manylinux actually does

As a result, it works on any Linux with glibc 2.28 or later (RHEL 8·9, Ubuntu 20.04 or later, Debian 10 or later). This lower bound is the same as that of the torch wheel itself (manylinux_2_28), so the additional constraints caused by the Pointcept wheels are zero. Upstream spconv-cu124·cu126 and cumm-cu126 are also distributed as manylinux_2_28.

Since the platform tag is in the wheel filename, on hosts with insufficient glibc pip rejects it at the install stage and states the reason. The problem of crashing with an obscure error at import goes away.

Build base images

nvidia/cuda:<버전>-devel-rockylinux8 is distributed for amd64 and arm64 for all of CUDA 11.8, 12.1, 12.4, 12.6, 12.8, 12.9, 13.0, 13.1, and 13.2 (Docker Hub, checked 2026-09-28). For CPU targets, rockylinux:8 is used.

There are also pytorch/manylinux2_28-builder (x86_64) and pytorch/manylinuxaarch64-builder (aarch64), which PyTorch uses to build its own wheels. But they are separate repositories per architecture, so the rockylinux8 image, which provides both platforms under one name, was chosen.

Behavior of build_wheels.sh

• The --os default is manylinux_2_28. ubuntu24.04 and ubuntu22.04 can still be used.
• Inside the container, it enables EPEL and PowerTools with dnf and installs gcc-toolset, cmake, git, openblas-devel, and sparsehash-devel.
• The gcc-toolset version is matched to the range nvcc allows. 11 for cu118, 12 for cu121, and 13 otherwise.
• Compiled wheels and prebuilt wheels with the linux_* tag (PyG, etc.) all go through auditwheel repair --plat manylinux_2_28_<arch>. If a prebuilt fails the policy, that package is compiled from source.
• Libraries excluded from repair: libtorch*, libc10*, libshm*, libcaffe2*, libgomp* (provided by torch), libcudart*, libnvrtc*, libcublas*, libcusparse*, libcurand*, libcufft*, libcusolver*, libnvJitLink*, libnvToolsExt*, libcudnn*, libnccl*, libcupti*, libnvshmem* (provided by nvidia-* wheels), libcuda.so* (provided by the driver). Putting these in the wheel loads a different copy from the one torch uses, causing conflicts.
• ROCm only has Ubuntu images. Without --os, it builds on ubuntu24.04 (no tag); if --os manylinux_2_28 is specified, that combination is skipped.
• --native builds use the host OS, so no manylinux tag is added.
• The output path does not distinguish OS (<out>/linux-<arch>/<accel>/torch<ver>-cp<py>/). Rebuilding into the same directory replaces the old ubuntu wheels with manylinux wheels.

Comparison with other approaches

Verification results (2026-09-28, amd64 · cu130 · torch 2.9.1 · sm_120)

Built pccm, cumm, spconv, torch-scatter, and pointops with --os manylinux_2_28.

When the same combination was built on ubuntu24.04, cumm required GLIBC 2.38 / GLIBCXX 3.4.32.

The PyG prebuilts did not pass the policy. auditwheel rejected torch_scatter-2.1.2+pt29cu130-…-linux_x86_64.whl (data.pyg.org) for "too-recent versioned symbols", and the builder fell back to compiling from source. This means the PyG prebuilts used on amd64 so far could also have broken on older hosts. In exchange, the amd64 build time grows by the compile time of the 3 PyG packages.

Runtime check (docker run --gpus all, with only uv, python 3.12, torch 2.9.1+cu130, and the wheels installed on a base image with nothing else installed):

Not yet confirmed

• flash-attn, swin3d, torch-sparse, torch-cluster, pointops2, pointgroup_ops, pointseg, pointrope built with manylinux
• arm64 (DGX Spark) manylinux build
• cu118·cu121 using gcc-toolset-11·12

Remaining decisions

• glibc: whether to build with ubuntu22.04 or distinguish by tag → resolved with the manylinux_2_28 default
• 7.5 (Turing): whether to add it to the tag after fixing the agent to decide flash use by capability
• Whether to publish cu129 (A3·B3)
• Whether to add spconv ConvAlgo.Native pinning to the agent (app.cvp.run). Required to support GB10·B300

References

• Pointcept
• Pointcept fork: scripts/build_wheels.sh, scripts/verify_wheels.sh, scripts/verify_mlops.sh
• mlops: src/geo_mlops/resources/training_runtime/Dockerfile.pointcept, src/geo_mlops/resources/training_runtime/pointcept_semseg.py(GeoSpconvAlgo),docs/plans/36_pointcept_training_runtime_plan.md, docs/plans/50_pointgroup_instance_segmentation_plan.md
• app.cvp.run: cli/src/pointcept/packages.ts, cli/src/pointcept/env.ts, cli/src/pointcept/scripts.ts, docs/plans/pointcept-agent-app-roadmap.md