Some processes on a Linux host occasionally run much slower and miss real-time deadlines. A reboot can seem to help, but the behavior isn't consistent. How would you tell whether NUMA memory placement is involved?
How do you diagnose an intermittent NUMA slowdown?
Answers
Start by checking whether the slowdown follows particular cores or NUMA nodes. Run the same small workload pinned to different cores, then compare local and remote memory latency when the host is fast and when it's slow.
Also look at where the affected process's memory is allocated, not just where its threads are running. In a NUMA system, a core can access memory on another node, but that access can take longer. A long-running process with memory spread across nodes can behave differently from one whose memory is mostly local.
Reply to this answer
Check what happens as the page cache fills. Heavy reads from NFS can populate the cache, and memory allocation can spill onto another NUMA node rather than reclaiming local cache first. That can leave application memory spread across nodes.
Full cache by itself isn't a problem. The useful test is whether the process's memory placement and latency change as the cache grows, and whether that lines up with the missed deadlines.
Reply to this answer
Don't assume the fastest result is the normal hardware baseline. Firmware behavior can complicate the comparison. In one reported case, corrected ECC events caused the firmware to disable a NUMA lookup directory, which improved latency while using more inter-node bandwidth.
So compare firmware settings and hardware event logs across fast and slow runs. That behavior is specific to the system, not something to expect from every ECC event.
_s84+vordali.7:
That also makes a reboot a tricky test. It can change the condition you're measuring rather than simply fixing it.
Reply to this answer
I'd avoid replacing hardware based on the slowdown alone. Record the per-core results, memory placement, event logs and firmware state first. If the issue tracks cache growth or a firmware change, that gives you a more specific lead than just a slow host.
Reply to this answer
Add your answer
Answers are reviewed before publication.
p.847!brassfunnel:
Keep the workload and CPU placement consistent between runs. Otherwise changes in scheduling can muddy the comparison.