NUMA (Non Uniform Memory Access) is a multiprocessor system where it does not take the same amount of time to access any given memory location. The system memory (RAM) is split contiguously between the CPUs (first half to CPU0 and second half to CPU1 on a 2-CPU NUMA) such that accessing memory associated with a different CPU might take longer than accessing memory of one’s own CPU. It takes longer because the data needs to travel through the interconnect between the CPUs. Most of the NUMA magic is supported by the CPUs itself, so that normal load-store machine instructions do all this behind the scenes.
A NUMA-capable system where NUMA is not enabled runs in UMA (Uniform Memory Access) mode. By definition, UMA is supposed to provide constant-time access to memory locations, but realistically on Intel systems the memory access times seems to vary - faster for memory close by and slower for memory attached to a different CPU.
You might need to check your BIOS or use tools provided by the CPU or system vendor to enable or disable NUMA and to choose from among the NUMA configs supported by the CPU or system vendor.
numactl is a common tool that can be used to view and set NUMA configs.
APIs provided in the libnuma library (set_mempolicy) can be used to isolate all succeeding memory allocations use the memory associated with specified CPU when they need to use RAM. This is easiest to do with the set-and-fork multi-process model. This can be used with multi-threading model too, but the benefits might vary.
numastat tool (that ships with numactl) can be used to print per-NUMA-node memory statistics:
$ numastat node0 node1 node2 node3 numa_hit 2098759 453124 626116 663329 numa_miss 0 0 0 0 numa_foreign 0 0 0 0 interleave_hit 62341 62613 62331 62595 local_node 2067691 369541 543901 566566 other_node 31068 83583 82215 96763
$ sudo cat /proc/sys/kernel/numa_balancing 1
Echo a 0 or 1 to that file to toggle the tuning.
Alternatively, find the PCI bus IDs of the GPUs (as shown here) and check the NUMA node of that PCI bus ID like this:
$ cat /sys/bus/pci/devices/0000:01:00.0/numa_node 3 $ cat /sys/bus/pci/devices/0000:c2:00.0/numa_node 0