PCI Express (PCI-E) is the popular high-speed serial bus used to connect GPUs and NICs to the CPU.
The max bus speed per lane has been doubling every generation in PCI-E:
If you see a 2-socket system with 2x PCI-E Gen3x16 links for each socket, to get the total bandwidth of the system:
2 sockets, each with 2x of Gen3x16 lanes
= 2 x 2 x 16 x (Gen3 per-lane bandwidth)
= 64 x (Gen3 per-lane bandwidth)
= 64 x 1 GB/s
= 64 GB/s