Why FC weights are in transposed form

The torch nn.linear operator is used to implement what is traditionally called the fully connected (FC) layer in neural networks. Looking at the nn.linear documentation I noticed that though the weights matrix is used as (in_features, out_features) in the FC, it is stored as its transpose (out_features, in_features). Why is that?

The consensus seems to be that:

• During matrix multiplication, the weights matrix is read column-by-column, which is typically not as efficient as reading row-by-row. So, it is stored in the transposed form.
• Another reason seems to be that FC is typically mapped down to a BLAS GEMM routine. And in BLAS GEMM, transposes are cheap.

References

• https://discuss.pytorch.org/t/why-does-the-linear-module-seems-to-do-unnecessary-transposing/6277