RF-DETR:Troubleshooting:SlowTraining
lastmod 2026-09-03created 2026-09-023 edits
Record of tracking down and fixing RF-DETR training that ran abnormally slowly even while holding the GPU.
Bottom line first: What was slow was not training but the validation metric computation run every epoch. While torchmetrics' MeanAveragePrecision.update() RLE-encoded predicted masks one by one on the CPU, the GPU sat idle for 85–90% of wall-clock time. The default hyperparameter eval_interval: 1 made it pay that cost every epoch.
Symptoms
- RF-DETR training feels abnormally slow
-
nvidia-smishows GPU-Util at 0%, raising the suspicion that it is running on the CPU
+-----------------------------------------+------------------------+----------------------+
| 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A |
| N/A 44C P0 10W / N/A | Not Supported | 0% Default |
+-----------------------------------------+------------------------+----------------------+
| 0 N/A N/A 2817 G /usr/lib/xorg/Xorg 125MiB |
| 0 N/A N/A 3348 G /usr/bin/gnome-shell 18MiB |
| 0 N/A N/A 2727029 C python 5843MiB |
+-----------------------------------------------------------------------------------------+
Environment
| Item | Value |
| Node | DGX Spark (NVIDIA GB10, aarch64, 20 cores) |
| Driver / CUDA | 580.173.02 / 13.0 |
| Runtime image | Based on |
| Model | |
| Orchestration | k3s, namespace |
Diagnosis
1. Is it actually using the GPU
The nvidia-smi process list has a python of type C(compute) holding 5843MiB. That means a CUDA context is alive, so this is not CPU training.
Xorg(125MiB) and gnome-shell(18MiB) in the same list matter later — these desktop processes constantly put a floor of 3–4% under GPU-Util.
| NOTE |
| A single GPU-Util snapshot tells you nothing beyond "no kernel was running at that instant". You must sample. |
2. Sampling — finding two phases
| Phase | Observation | Interpretation |
| A (most of the time) | 0–4% / 10.5W, a 72–92% spike about once every 7 seconds (still 13W even then) | Training submits work to the GPU only once every 7 seconds. The 3–4% floor is the desktop processes |
| B (a stretch of about 30 seconds) | 61–89% sustained / 35–38W | The real training speed of this hardware |
The GPU is idle for 85–90% of wall-clock time. The bottleneck is upstream of the GPU.
Power was the decisive clue. 10.5W is idle, 38W is real load. Since even the "92%" spike in phase A drew only 13W, it was not doing much work even at that moment.
3. Is it CPU / I/O bound
The container has no procps, so top does not work. Read /proc directly.
kubectl exec -n tenant-<코드> <파드> -- python -c "
import time, os
def snap():
v = list(map(int, open('/proc/stat').readline().split()[1:]))
return sum(v), v[3] + v[4], v[4]
t0, i0, w0 = snap(); time.sleep(3); t1, i1, w1 = snap()
dt = t1 - t0
print('cores :', os.cpu_count())
print('cpu busy : %.1f%%' % (100 * (dt - (i1 - i0)) / dt))
print('iowait : %.1f%%' % (100 * (w1 - w0) / dt))
print('loadavg :', open('/proc/loadavg').read().strip())
"
1.6 of 20 cores, iowait near 0. Nothing is saturated. So it is not a throughput limit; something is serialized.
At this point the first hypothesis (too few data loaders / bound by CPU augmentation) is rejected. If CPU augmentation were the bottleneck, 4–5 cores would be busy.
| NOTE |
| Even inside a container, |
4. Which process is eating the CPU
kubectl exec -n tenant-<코드> <파드> -- python -c "
import os
for pid in sorted((p for p in os.listdir('/proc') if p.isdigit()), key=int):
try:
st = open('/proc/%s/stat' % pid).read().rsplit(') ', 1)[1].split()
print(pid, open('/proc/%s/comm' % pid).read().strip(),
'cpu_s=%.1f' % ((int(st[11]) + int(st[12])) / os.sysconf('SC_CLK_TCK')),
'threads=' + st[17])
except Exception: pass
"
1 python cpu_s=5124.7 threads=35
136-139 pt_data_worker 각 ~136s (학습 로더 4개)
145-148 pt_data_worker 각 ~30s (검증 로더 4개)
The 8 workers are running normally and total 664 CPU-seconds, while the main process alone has 5124 CPU-seconds — about 7.7×. num_workers took effect and data loading is not the bottleneck.
In normal GPU training, the main process barely uses the CPU because it is waiting for the GPU. This is the opposite.
5. Wrong answer — the PTX JIT compilation hypothesis
GB10 is Blackwell sm_121. If the wheel has no SASS for that architecture, the driver JIT-compiles every kernel from PTX, and that is single-threaded CPU work inside the process — it fit the observed pattern well.
kubectl exec -n tenant-<코드> <파드> -- python -c "
import torch
print('torch :', torch.__version__, 'cuda', torch.version.cuda)
print('device :', torch.cuda.get_device_capability())
print('archs :', torch.cuda.get_arch_list())
"
torch : 2.13.0+cu130 cuda 13.0
device : (12, 1)
archs : ['sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
sm_121 is missing. But this alone is not evidence — CUDA guarantees forward compatibility across minor versions within the same major, so an sm_120 cubin normally loads on a 12.1 device.
The direct evidence is whether the JIT cache is being updated. Size alone is not enough: if the cache hits its limit and rotates entries, the contents change while the size stays the same.
kubectl exec -n tenant-<코드> <파드> -- python -c "
import os, time
newest = 0; n = 0; total = 0
for root, _, files in os.walk('/root/.nv/ComputeCache'):
for f in files:
try: st = os.stat(os.path.join(root, f))
except OSError: continue
n += 1; total += st.st_size; newest = max(newest, st.st_mtime)
print('files :', n)
print('total : %.1f MiB' % (total / 2**20))
print('newest write : %.1f s ago' % (time.time() - newest))
"
Untouched for 88 minutes. The 248MB was compiled once at the start and has nothing to do with the current CPU usage. Hypothesis rejected.
6. Which thread
kubectl exec -n tenant-<코드> <파드> -- python -c "
import os
d = '/proc/1/task'
rows = []
for t in os.listdir(d):
try:
st = open('%s/%s/stat' % (d, t)).read().rsplit(') ', 1)[1].split()
rows.append(((int(st[11]) + int(st[12])) / os.sysconf('SC_CLK_TCK'),
t, open('%s/%s/comm' % (d, t)).read().strip()))
except Exception: pass
rows.sort(reverse=True)
for cpu, t, comm in rows[:12]: print('%9.1f %-8s %s' % (cpu, t, comm))
"
5078.6 1 python
137.5 366 pt_autograd_0
29.3 140 pt_data_pin
4.8 149 pt_data_pin
2.3 135 cuda-EvtHandlr
The main thread accounts for 5078.6 (99%) of 5124.7. Against the 5307-second observation window, it is running non-stop at 96%.
This result alone, however, cannot tell "PTX JIT" from a "Python hot loop" — driver JIT also runs inline on the calling thread. You have to look at the stack directly.
7. Getting the stack with py-spy
Running it directly inside the container fails:
The default Kubernetes capability set does not include SYS_PTRACE. The general profile of kubectl debug adds it.
kubectl debug -n tenant-<코드> <파드> --target=main --profile=general \
--image=python:3.11-slim -- sh -c "pip install -q py-spy && py-spy dump --pid 1 --nonblocking"
# 출력은 임시 컨테이너의 로그로 간다
kubectl logs -n tenant-<코드> <파드> -c debugger-xxxxx
| NOTE |
| |
Result:
Thread 1 (active): "MainThread"
_get_safe_item_values (torchmetrics/detection/helpers.py:644)
update (torchmetrics/detection/mean_ap.py:533)
wrapped_func (torchmetrics/metric.py:549)
on_validation_batch_end (rfdetr/training/callbacks/coco_eval.py:447)
_call_callback_hooks (pytorch_lightning/trainer/call.py:228)
_evaluation_step (pytorch_lightning/loops/evaluation_loop.py:455)
run (pytorch_lightning/loops/evaluation_loop.py:146)
on_advance_end (pytorch_lightning/loops/training_epoch_loop.py:406)
run (pytorch_lightning/loops/training_epoch_loop.py:154)
...
train (rfdetr/detr.py:998)
train (rfdetr_detect.py:467)
Cause
Mask RLE encoding
_get_safe_item_values inside torchmetrics' MeanAveragePrecision.update() RLE-encodes predicted masks one at a time on the CPU. rf-detr-seg-* outputs a mask for every query, so each batch has hundreds of them.
rfdetr turns on backend="faster_coco_eval" (rfdetr/training/callbacks/coco_eval.py:197), but that backend only replaces the compute() stage. The bottleneck, update(), is still the torchmetrics implementation.
Paying the cost twice
The default template sets use_ema: True, while rfdetr's eval_ema_only defaults to False. With this combination, on_validation_batch_end runs one more independent forward pass with the EMA model for every validation batch and updates the metrics again (coco_eval.py:463). Validation cost doubles exactly.
What the two observed phases were
- Phase B (38W, 30 s) = an actual training epoch
- Phase A (10.5W, long) = validation
Since the default was eval_interval: 1, this alternation repeated every epoch.
Fix
Commit 30d5069 perf(training): Cut the RF-DETR validation cost
1. Default validation interval
RF-DETR default template in src/geo_mlops/training/hyperparams.py:
rfdetr passes this value straight to Lightning's check_val_every_n_epoch (rfdetr/training/trainer.py:721), so on other epochs not only the metric computation but the whole validation loop is skipped. rfdetr validates the last epoch again on its own.
2. Exposing two validation-cost switches
These are rfdetr TrainConfig fields, but they were missing from the platform whitelist, so the wizard rejected them.
eval_ema_only: false # true = EMA 모델로만 검증, 중복 forward 제거 (비용 절반)
log_per_class_metrics: true # false = 클래스별 AP 계산·로깅 생략
The template keeps the rfdetr defaults. Turning them on changes what is measured — with eval_ema_only, only metrics for the EMA model remain, and with log_per_class_metrics: false, per-class AP disappears. Judging that this should not be changed silently for the sake of performance, they were only exposed, with the trade-off stated in the comments.
| WARNING |
| |
3. Deployment scope
Only the server needs to be redeployed. The training runtime image stays as is — the trainer passes values through model.train(**overrides), so once a new key passes the whitelist it reaches rfdetr. The frontend has no hard-coded parameter list either (it renders the backend YAML as is).
Verification
Confirmed in the field that eval_interval: 40 resolves the problem.
Lessons learned
- A single GPU-Util snapshot is not evidence. Only 1Hz sampling together with power separates idle from load. On a node running a desktop, a 3–4% floor is always there.
- If nothing is saturated, suspect serialization. If it is slow while CPU, I/O, and GPU are all idle, it is not a throughput problem.
- Look at the stack instead of guessing. Both hypotheses (data loader, PTX JIT) were wrong, and measurement ruled out each one. One py-spy run was more accurate than all the reasoning before it.
- Judge a cache by mtime, not size. A cache at its limit rotates its contents while its size stays fixed.
- Validating every epoch is a trap for segmentation models. Computing mask metrics can cost more than training.
Reusable diagnostic sequence
-
nvidia-smiprocess list — is there aC-type process (whether the GPU is in use) -
nvidia-smi --query-gpu=utilization.gpu,power.draw --format=csv -l 1— idle ratio and load periods -
/proc/stat— CPU busy / iowait (rule out CPU/I/O bound) - Walk
/proc/<pid>/stat— which process (workers vs main) -
/proc/1/task/*/stat— which thread -
py-spy dump— which function (kubectl debug --profile=generalor directly on the node)