Pointgroup:TrainingEvaluationOOM

lastmod 2026-10-01created 2026-10-012 edits

Issue where running evaluation during training causes OOM

• Diagnosis and fix of PointGroup training exit 137
• Written: 2026-10-01
• Scope: the mlops platform's Pointcept training runtime (PointGroup, pg-ptv3) and Kubernetes log collection
• Related document: docs/plans/54_training_pod_memory_and_log_follow_plan.md
• Status: code changes done; commit, deployment, runtime image republishing, and retraining measurements not done yet

Summary

A PointGroup instance segmentation training run ended with exit code 137 after about 16 hours. Two causes overlapped.

1. The memory used by the evaluation step grows as training progresses. The evaluator builds a mask the size of the original point count for every proposal and holds them until all validation scenes are done.
2. The validation data loader forks new worker processes at every evaluation. Forking a main process that has already grown creates one more copy of that memory.

Together they exceeded the 64Gi pod memory limit, and the kernel killed the pod. The diagnosis also revealed a separate defect. The platform's log collection stopped 4 hours after the pod started and never reconnected, so the failure screen was showing logs that were 12 hours old.

GPU unified memory, /dev/shm, and batch size were not the cause.

Symptoms

The training screen showed the following message.

학습이 실패했습니다 (exit 137). 학습 단계에서 멈췄습니다
메모리 부족(OOM)으로 종료되었습니다. 배치 크기나 입력 크기를 줄이거나 /dev/shm 을 키우세요.
파드 메모리 상한(GEO_MLOPS_TRAINING_RUNTIME_MEMORY_LIMIT)에 걸렸을 수도 있습니다.
trainer exited with code 137:
[2026-09-30 08:47:35,902 INFO misc.py line 117 1] Train: [8/100][310/494] ... loss: 4.4757 seg_loss: 0.0000 ...

From the message alone, it looks as if the run died in the middle of a training step in epoch 8. That is not what actually happened. See the Timeline below.

seg_loss: 0.0000 is not a defect. A run with no classes selected trains with the single class object, and with only one class the cross-entropy is a constant 0. Training is driven by the offset loss.

Diagnosis

Initial hypotheses

Values checked on the node

I read the kernel log on the deployment node.

sudo dmesg -T | grep -i -B2 -A6 "out of memory"
sudo dmesg -T | grep -B45 -A3 "oom-kill:constraint"
sudo dmesg -T | grep -A45 "Memory cgroup stats for"

The key lines are as follows.

oom-kill:constraint=CONSTRAINT_MEMCG, ... task=python,pid=442561
Memory cgroup out of memory: Killed process 442561 (python) total-vm:114837024kB, anon-rss:36670684kB, file-rss:95364kB, shmem-rss:231896kB
Tasks in ... are going to be killed due to memory.oom.group set
Memory cgroup out of memory: Killed process 443443 (pt_data_worker) total-vm:21504120kB, anon-rss:1190224kB
Memory cgroup out of memory: Killed process 1672569 (pt_data_worker) total-vm:108775600kB, anon-rss:31221212kB

Memory cgroup stats for ...:
anon 68195393536
file 237563904
kernel 286511104
shmem 237547520
inactive_anon 57443008512
active_anon 10989826048

How to read the numbers

• constraint=CONSTRAINT_MEMCG means it hit the pod cgroup limit, not the whole node's.
• The RSS of a forked process includes pages shared with the parent. So the process table alone cannot tell whether the worker's 31.2GB is shared or its own.
• The cgroup counts each page only once. The sum of the three processes' anonymous memory (69.1GB) is nearly equal to cgroup anon (68.2GB), so the worker's 31.2GB is not shared; it is a copy the worker owns.
• Without this copy, usage would have been about 38GB, well below the limit.
• There are only two workers, and their pids differ greatly. One was created at the start of training, the other much later. The later one is the validation loader worker newly forked at evaluation.
• inactive_anon is 57GB. Most of the memory had not been used recently. This is the copy the worker was holding and the masks the main process was merely holding on to.

Causes

Evaluation memory grows with each epoch

At inference, PointGroup groups points into clusters to make proposals. The evaluation path is as follows.

1. The model builds a dense int32 tensor of size [number of proposals, number of voxels] on the CPU.
2. Pointcept's InsSegEvaluator.eval expands it to the original point count with pred_masks[:, idx]. It becomes an int32 tensor of [number of proposals, number of original points].
3. associate_instances builds a bool mask the size of the original point count for each proposal and puts it in pred_inst["mask"]. This mask stays alive until all validation scenes are done and aggregated.
4. The aggregation (evaluate_matches) reads only the intersection counts and point counts. It does not read the masks.

The number of proposals increases as the model gets better at grouping clusters. In a single-class run, proposal scores are always 1.0, so no proposals are filtered out either. That is why evaluation memory grows with each epoch, and why early evaluations pass without trouble and only later epochs reach the limit.

Forking validation workers duplicates memory

Pointcept creates the training loader with persistent_workers=True. Its workers are forked only once, at the start of training. At that point the main process is small.

The validation loader does not have this option. <span style="color: red">It forks new workers at every evaluation. A forked worker inherits the parent's entire memory at that moment. It is shared at first, but when the parent writes to that memory again during evaluation, copy-on-write gives the parent new pages and leaves the worker alone with the old ones. As a result, there are two copies of the memory.

Log collection stops at 4 hours

The server's follow_job_logs opened the pod log stream only once. When the stream ended, it returned as is and did not reconnect. The stream ends quietly, without an error, in two cases.

• The kubelet closes the follow connection at the streaming timeout (default 4 hours).
• The HTTP client ends it with a read timeout when there is no data for an hour.

In this run, the logs stopped exactly 4 hours after the pod came up.

Timeline

Container log times are in UTC, and the node's dmesg -T is in KST. Lining them up shows a 12-hour gap between the last log and the termination.

Fixes

I implemented four changes, split into phases.

Phase 1. The evaluator does not hold on to masks

• File: src/geo_mlops/resources/training_runtime/pointcept_insseg.py
• Added a new associate_instances function. It builds the mask of one proposal at a time, counts its points, and discards it. When mapping back to the original points, it also moves only one proposal row at a time. It never builds the [number of proposals, number of original points] tensor.
• The intersection with ground-truth instances is obtained by counting, once, the instance IDs of the points a proposal covers. Unlike the original, it does not rescan all points for every ground-truth instance, so evaluation is also faster.
• GeoInsSegEvaluator inherits Pointcept's InsSegEvaluator and replaces only eval. The AP calculation and the best checkpoint criterion (AP50) stay the same.
• The evaluator name referenced by the generated config changed to GeoInsSegEvaluator.

Phase 2. The validation loader reads without workers

• File: src/geo_mlops/resources/training_runtime/pointcept_semseg.py
• GeoTrainer.build_val_loader builds the loader with the worker count set to VAL_LOADER_WORKERS = 0. No fork happens at evaluation.
• Validation uses batch 1 and has few scenes, so workers save almost no time.
• Applies to both semantic segmentation and instance segmentation. The training loader's worker count (num_workers) is unchanged.

Phase 3. The log follower reconnects

• File: src/geo_mlops/integrations/kubernetes.py
• When the stream ends, it queries the pod status.
  • If running, it reconnects from the time of the last line received.
  • If finished, it reads once more to pick up the tail, then stops.
  • If the pod is gone, it stops.
  • If the cluster cannot be read, it assumes the pod is running and retries.
• Logs are requested with timestamps. _LogCursor drops lines that come again on reconnect and passes on only the lines with the timestamps stripped. Lines are not lost even when several share the same timestamp.
• The reconnect interval starts at 2 seconds and grows up to 30 seconds on consecutive failures.
• This defect affected every training run longer than 4 hours. It was not specific to PointGroup.

Phase 4. Memory is logged every epoch

• File: src/geo_mlops/resources/training_runtime/pointcept_semseg.py
• GeoProgress.after_epoch prints one line per epoch.

-> memory epoch 30: rss 3.00GiB (peak 36.00GiB), pod 5.00GiB of 64.00GiB (peak 40.00GiB), cuda reserved 8.00GiB (peak allocated 6.00GiB)

Exit 137 does not tell you the cause after the fact. With this line, you can see which value grew and since when, without logging in to the node.

What was left unchanged

• The [number of proposals, number of voxels] int32 tensor inside the model is code from the Pointcept fork, so I left it alone. It is voxel-based and therefore much smaller than a tensor based on original points. If the Phase 4 logs still show a large evaluation peak, I will fix it then.
• The failure message wording and the default memory limit (64Gi) were not changed.

Verification

The one failure is test_flash_attention_stays_on. It fails because flash-attn is not installed in the development environment and is unrelated to this change.

One thing could not be verified. The body of GeoInsSegEvaluator.eval was never run, because torch and pointcept are not available in the development environment. The association calculation was cross-checked, but the evaluation loop itself will run for the first time in the retraining measurement.

Remaining work

1. Commit the changes.
2. Deploy the server. Phase 3 is server code, so this alone applies it.
3. Republish the Pointcept training runtime image. Phases 1, 2, and 4 are trainer code inside the image.
4. Retrain pg-ptv3 on the same data. Three things to check:
   • Does the first evaluation finish normally and produce AP values?
   • Do the logs continue after 4 hours?
   • How much does the peak in the -> memory lines grow with each epoch?

If you hit the same symptom again

First, read the kernel log on the node.

sudo dmesg -T | grep -A45 "Memory cgroup stats for"
sudo dmesg -T | grep -B10 -A6 "oom-kill:constraint"

Then read things in the following order.

1. Do not take at face value the epoch that the log tail in the failure message points to. Look at the -> memory lines in the log and the last step in MLflow first.
2. Check whether constraint is CONSTRAINT_MEMCG. If so, it hit the pod limit, not the whole node.
3. Check which of anon, shmem, and kernel is large in the cgroup stats.
   • If anon is large, it is the process heap.
   • If shmem is large, it is /dev/shm.
   • If kernel is large, it is on the driver or kernel side.
4. Compare the sum of anonymous memory in the process table with cgroup anon. If the sum is much larger, pages shared through fork were counted twice; if they are similar, each process holds its own share.
5. If you need the trend, look at the pod's container_memory_working_set_bytes in monitoring.

See also

• Pointcept
• Pointgroup