If you're running training jobs, inference services, or any GPU-bound workload, you already know the pain. A lack of a trend history, alerting, and correlation with the host is a gap this release closes.
What you get with integrated GPU monitoring
The agent queries the GPU drivers directly and pulls real-time and historical metrics across the full stack of what actually matters:
- Utilization and compute: It covers GPU utilization, encoder and decoder usage, and GPM deep-profiling counters: SM activity, SM occupancy, Tensor Core activity, and FP16, FP32, and FP64 pipeline activity. This is the difference between the GPU being busy and the GPU being busy doing the math you're paying for.
- Memory: It covers the total, used, and free VRAM; memory utilization; and memory bandwidth utilization.
- Temperature and power: It covers the core temperature, power draw, power limit, power usage percentage, and fan speed. Catch thermal or power incidents before they throttle you.
- Clocks and throttling: It covers graphics and memory clock speeds and usage percentages, the performance state, and the explicit throttle status with reasons.
- Reliability: It covers ECC single- and double-bit error counts (current and lifetime), retired memory pages, and the row remap status: early-warning signals for hardware degradation.
- Interconnections: It covers PCIe TX and RX throughput, generation, and link width plus full NVLink telemetry: per-link throughput, CRC errors, replay errors, and recovery events.
- Topology and modes: It covers the Multi-Instance GPU mode, compute mode, persistence mode, and display state.
- Processes: It covers the per-process GPU memory, command line, executable path, and compute versus graphics process counts. Find the job eating your VRAM.
- Virtual GPU (vGPU) licensing: It covers the NVIDIA vGPU license status, product, and expiration, tracked directly.
Why it matters for distributed workloads
If you run multi-GPU training or inference over NVLink, watch the NVLink error counters. Rising CRC or replay errors usually mean cable issues, improper seating, or cooling problems affecting link stability. These issues degrade throughput long before anything hard-fails. Catching them here means fixing a cable seating problem instead of debugging a mysteriously slow training run for two days.
Getting started
- Prerequisite: You need the Linux Full-Stack Agent v22.3.00 or above.
- Auto-discovery: GPU monitors are created automatically when the agent detects GPUs on a host. If you need to add or remove GPUs later, just rerun the discovery process.
- Access: Open a server monitor and check the GPU tab, or go to Server > GPU to see every GPU monitor across your account.
To learn more in detail, visit our help documentation here.
Supported vendors today
- NVIDIA and AMD (some metrics like for the CUDA version, architecture, GPM counters, NVLink, and vGPU are NVIDIA-only)
Happy monitoring!
Like (1)
Reply
Replies (0)
