Vector Operations

Knowing what a vector is — a directed arrow with magnitude and direction — gets you started. Knowing how to operate on vectors is what lets you do ML.

Three operations cover most of what you will ever need: addition, scalar multiplication, and the dot product.

Vector Addition

Two vectors can be added together by placing the tail of the second at the tip of the first. The result is the single arrow that goes from the start of the first vector to the tip of the second.

Algebraically, addition is component-wise:

The order does not matter — vector addition is commutative: a + b = b + a.

For example: a hiker walks (2, 3) — two kilometres east and three north — then turns and walks (1, −1) — one east and one south. The net displacement is (2 + 1, 3 + (−1)) = (3, 2). The single arrow from start to finish is the sum — regardless of which leg of the journey you take first.

This exact operation appears in every residual network and every Transformer. A residual connection adds a layer’s input directly to its output: y = F(x) + x. That addition — literally vector addition applied at every layer — is one of the most important architectural decisions in modern deep learning. It allows gradients to flow through very deep networks without vanishing, and it is why models like ResNet and GPT can be trained at all.

Scalar Multiplication

A scalarScalarA single number with magnitude but no direction. Scalars scale vectors — multiplying a vector by a scalar changes its size but not its direction. is a plain number with no direction. Multiplying a vector by a scalar scales its length without changing its direction (unless the scalar is negative, which reverses the direction).

Multiplying v = (2, 1) by the scalar 3 gives (6, 3) — the same direction, three times the length. Multiplying by −1 flips the vector to point the opposite way.

Scalar multiplication appears directly in the gradient descent update rule: w ← w − η∇L. The term η∇L scales the gradient vector by the learning rate η. A small η produces a short step in the gradient’s direction; a large η produces a long step in the same direction. The direction of the update is determined by the gradient; the magnitude of the step is determined by η.

The Dot Product

The dot productDot ProductA measure of how similar two vectors are in direction. If the dot product is zero, the vectors are perpendicular. The larger the dot product, the more aligned the vectors. takes two vectors and returns a single number — a scalar — that captures how aligned they are. The sign of the result tells you the relationship between the directions.

When two vectors point in the same general direction — angle less than 90° — the dot product is positive. When they are perpendicular — exactly 90° apart — the dot product is zero. When they point in opposing directions — angle greater than 90° — the dot product is negative.

Computing the dot product in two dimensions: multiply matching components and add. For vectors (3, 1) and (1, 3):

Positive — the vectors point in the same general direction. For (3, 0) and (0, 3):

Zero — the vectors are perpendicular. For (3, 0) and (−3, 0):

Negative — the vectors point in opposite directions.

In 2D: a · b = a₁b₁ + a₂b₂. In higher dimensions, add more terms: a₁b₁ + a₂b₂ + a₃b₃ + … one term per dimension. In a 768-dimensional embedding space, the dot product has 768 terms — but the meaning is identical: multiply matching components and sum.

The geometric formula connects the algebraic calculation to the angle θ between the two vectors:

Both formulas give the same number. The geometric form makes the sign rule transparent: cos θ is positive for acute angles, zero at 90°, and negative for obtuse angles.

Every neuron in a neural network computes a dot product: it takes the dot product of its weight vector with the input vector, adds a bias, and passes the result through an activation function. A linear layer with 512 output neurons computes 512 dot products simultaneously — one per output neuron. The dot product is the fundamental arithmetic operation performed billions of times per second during every forward pass of every neural network in production.

Cosine Similarity

If you want to know whether two vectors point the same way — without caring how long they are — divide the dot product by both magnitudes. The result is always between −1 and +1 and depends only on the angle, not the scale.

Rearranging the geometric dot product formula gives:

This quantity is called cosine similarityCosine SimilarityThe cosine of the angle between two vectors, computed as their dot product divided by the product of their magnitudes. It ranges from −1 (opposite directions) through 0 (perpendicular) to +1 (identical directions) and is magnitude-invariant — scaling either vector leaves the value unchanged.. It ranges from −1 (opposite directions) through 0 (perpendicular) to +1 (identical directions), and it is magnitude-invariant: scaling either vector up or down does not change it.

import numpy as np

a = np.array([3, 1])
b = np.array([1, 3])

dot = np.dot(a, b)                                         # 6
cos_sim = dot / (np.linalg.norm(a) * np.linalg.norm(b))   # ≈ 0.6

That invariance is exactly why ML systems prefer cosine similarity over the raw dot product when comparing embeddings.