numactl

📅 2021-Feb-25 ⬩ ✍️ Ashwin Nanjappa ⬩ 🏷️ numa, numactl ⬩ 📚 Archive

numactl is a popular tool that is used to view and set the NUMA settings on a computer with multiple processors. It ships along with a complementary tool named numastat.

Installing the tool is easy:

$ sudo apt install numactl

The --hardware option can be used to check the NUMA status of your processors.

With a CPU that has no NUMA support:

$ numactl --hardware
No NUMA available on this system

With a single CPU that has NUMA support:

$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 7983 MB
node 0 free: 2066 MB
node distances:
node   0
  0:  10

In the above output, the CPU has 4 cores, but NUMA sees them as 4 CPUs. This is because the NUMA concept is older than the concept of multi-core CPUs.

With a 4-CPU system with NUMA support:

$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128801 MB
node 0 free: 125628 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 129016 MB
node 1 free: 128157 MB
node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
node 2 size: 129016 MB
node 2 free: 127929 MB
node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 3 size: 129002 MB
node 3 free: 127553 MB
node distances:
node   0   1   2   3
  0:  10  12  12  12
  1:  12  10  12  12
  2:  12  12  10  12
  3:  12  12  12  10

Each CPU in the above system has 32 cores, which NUMA sees as 32 different CPUs. We see 4 NUMA nodes in the above output, meaning that each CPU is in it’s own NUMA node. Each CPU (or NUMA node) is associated with 25% of the total system memory.

The node distance matrix shows that accessing memory from any other node is slower than accessing one’s own memory. In Linux, the typical distance values are 10, 12 and 32. The distance of node to itself is 10, the Near-Hop Distance (different NUMA node, but same socket) is 12 and Far-Hop Distance (different NUMA node, different socket) is 32.

The --show prints the NUMA policy of the current process.

This is what I got on a 4-CPU system:

$ numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3

Not quite sure what the output means when executed at the shell.

Tried with: numactl 2.0.11 and Ubuntu 20.04