NUMA

NUMA (Non Uniform Memory Access) is a multiprocessor system where it does not take the same amount of time to access any given memory location. The system memory (RAM) is split contiguously between the CPUs (first half to CPU0 and second half to CPU1 on a 2-CPU NUMA) such that accessing memory associated with a different CPU might take longer than accessing memory of one’s own CPU. It takes longer because the data needs to travel through the interconnect between the CPUs. Most of the NUMA magic is supported by the CPUs itself, so that normal load-store machine instructions do all this behind the scenes.

• A NUMA-capable system where NUMA is not enabled runs in UMA (Uniform Memory Access) mode. By definition, UMA is supposed to provide constant-time access to memory locations, but realistically on Intel systems the memory access times seems to vary - faster for memory close by and slower for memory attached to a different CPU.

• You might need to check your BIOS or use tools provided by the CPU or system vendor to enable or disable NUMA and to choose from among the NUMA configs supported by the CPU or system vendor.

• numactl is a common tool that can be used to view and set NUMA configs.

• APIs provided in the libnuma library (set_mempolicy) can be used to isolate all succeeding memory allocations use the memory associated with specified CPU when they need to use RAM. This is easiest to do with the set-and-fork multi-process model. This can be used with multi-threading model too, but the benefits might vary.

• numastat tool (that ships with numactl) can be used to print per-NUMA-node memory statistics:

$ numastat
                           node0           node1           node2           node3
numa_hit                 2098759          453124          626116          663329
numa_miss                      0               0               0               0
numa_foreign                   0               0               0               0
interleave_hit             62341           62613           62331           62595
local_node               2067691          369541          543901          566566
other_node                 31068           83583           82215           96763

• If you do not want to mess with manual tuning of NUMA nodes, the Linux kernel can do automatic tuning. To check if that is enabled:

$ sudo cat /proc/sys/kernel/numa_balancing
1

Echo a 0 or 1 to that file to toggle the tuning.

• On systems with multiple GPUs and multiple CPUs, configuring the right affinity between GPUs and CPUs might be important to get optimal performance for your application. The topology related options of nvidia-smi can be used to find the CPU affinities and NUMA nodes associated with the GPUs.

Alternatively, find the PCI bus IDs of the GPUs (as shown here) and check the NUMA node of that PCI bus ID like this:

$ cat /sys/bus/pci/devices/0000:01:00.0/numa_node
3

$ cat /sys/bus/pci/devices/0000:c2:00.0/numa_node
0

• AMD EPYC CPUs have a configuration called NPS (NUMA Nodes Per Socket) that needs to be configured in the BIOS. Details about what value to set for NPS can be found in the High Performance Tuning Guide for AMD EPYC 7002 Series Processors.