Linear algebra essentials

Vectors and matrices

A vector is an ordered list of numbers, or an arrow: \(v=(3,4)\) has two components. A matrix is a table of numbers, or several vectors placed side by side (or one under another).

Machine learning spends most of its time doing arithmetic with these objects. A row of data is a vector. A batch of rows is a matrix. The weights of a layer are a matrix.

The dot product

The dot product multiplies matching entries and adds them up:

\[ (1,2,3)\cdot(0.5, 0.1, 2)=0.5+0.2+6=6.7. \]

That is already a neuron without the activation: a weighted sum of inputs. In matrix notation people write \(x^\top w\) or \(x\cdot w\).

import numpy as np
np.dot([1, 2, 3], [0.5, 0.1, 2])   # 6.7

Matrix multiplication versus \(\odot\)

Matrix multiplication mixes rows of the first matrix with columns of the second. Element-wise multiplication (\(\odot\)) never mixes: it only pairs the entry with its neighbour in the same place. They are different operations, and both appear in models.

\[ \begin{pmatrix}1&2\\3&4\end{pmatrix} \begin{pmatrix}5&6\\7&8\end{pmatrix} = \begin{pmatrix}19&22\\43&50\end{pmatrix}, \qquad \begin{pmatrix}1&2\\3&4\end{pmatrix} \odot \begin{pmatrix}5&6\\7&8\end{pmatrix} = \begin{pmatrix}5&12\\21&32\end{pmatrix}. \]

import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
A @ B    # matrix product: [[19, 22], [43, 50]]
A * B    # element-wise:    [[ 5, 12], [21, 32]]

Transpose and norms

The transpose \(A^\top\) flips rows and columns. It is how you turn a column into a row, or line up shapes so a product is legal.

A norm \(\lVert v\rVert\) answers “how big is this vector?”. The L2 norm \(\lVert v\rVert_2=\sqrt{v_1^2+v_2^2+\cdots}\) is ordinary length. The squared L2 norm \(\lVert v\rVert_2^2\) is what you usually put in a loss, for example each term of k-means inertia.

import numpy as np
A = np.array([[1, 2], [3, 4]])
A.T                      # [[1, 3], [2, 4]]
np.linalg.norm([3, 4])   # 5.0

Why machine learning is matrix multiplication

One dense layer, on one input, is “multiply, add a bias, then activate”:

\[ y = f(xW+b). \]

A whole mini-batch \(X\) (rows are examples) is the same product, computed at once: \(Y=f(XW+b)\). That is why GPUs matter, and why shapes (\(n\times d\) times \(d\times k\)) are worth checking twice.

import numpy as np
x = np.array([1., 2., 3.])
W = np.array([[0.2, 0.5],
              [0.1, 0.3],
              [0.4, 0.0]])
x @ W   # two units: [1.6, 1.1]

flowchart TD
  A[Inputs x] --> B[Matrix multiply xW plus bias]
  B --> C[Activation f]
  C --> D[Next layer]

The non-linearity \(f\) is an activation function. The multiply-and-add is the linear algebra on this page.

See also

Back to top