Orthogonal/Orthogonality

Orthogonality can be represented in many different ways depending on the situation where they are used. I'll start from the picture that everyone knows, two arrows at a right angle. Then we move to the inner product, which works in any dimension, and to functions, which work like vectors with infinitely many elements. The last section explains why engineers care, and where the idea of zero correlation needs a warning.

What does orthogonality look like in a graph ?

The word orthogonal comes from the Greek words for right and angle. So the first definition is geometric, and it only needs a drawing of two vectors that start from the same point.

When we talk about Orthogonality in vectors represented in a graph (in a coordinate system), the orthogonal means the angle between any two vectors are 90 degree (right angle). It doesn't matter of magnitude of the vectors.

Two vectors v1 and v2 at a right angle

This kind of graphical representation would be easy to understand or intuitive, but you cannot represent orthgonality in this way if the dimension of a vector is larger than 3.

The right angle in the drawing also has a numerical form, and it is the old theorem of Pythagoras. The vectors v₁ and v₂ are orthogonal exactly when |v₁ + v₂|² = |v₁|² + |v₂|². For any other angle, an extra cross term appears in the sum. That cross term is 2 v₁ ⋅ v₂, and it leads directly to the next section.

Orthogonal means a right angle between the vectors : the length of each vector does not matter, so scaling either vector keeps them orthogonal.
The zero vector is orthogonal to every vector : it has no direction, and its inner product with any vector is zero.
The picture stops at three dimensions : the Pythagoras form does not, and it is the bridge to the inner product.

How does the inner product test orthogonality ?

We need a test that works for vectors of any size, because a signal with 1,000 samples is a vector with 1,000 elements. The inner product gives that test with one multiplication per element and one sum.

Another way of expressing the orthogonality of the two vectors is to use the inner product. When the inner product of two vectors is zero, we say the two vectors are orthognal to each other. In this way, we can check the orthogonality of two vectors with any dimension (size). This is the most commonly used concept of Orthogonality in engineering.

For real vectors v₁ = (a₁, ..., a_n) and v₂ = (b₁, ..., b_n), the inner product is v₁ ⋅ v₂ = a₁b₁ + a₂b₂ + ... + a_nb_n. In two and three dimensions it also equals |v₁| |v₂| cosθ, where θ is the angle between the vectors. Since cos 90 deg = 0, the inner product test and the right angle of the graph agree. In higher dimensions, the formula with cosθ becomes the definition of the angle.

Let's check two small examples. The vectors (1, 2, 3) and (3, 0, -1) give 1 x 3 + 2 x 0 + 3 x (-1) = 0, so they are orthogonal. The four-element vectors (1, 1, 1, 1) and (1, -1, 1, -1) give 1 - 1 + 1 - 1 = 0, so they are orthogonal as well. We cannot draw these two, but the test still works. They are also two rows of a 4 x 4 Hadamard matrix, which is where Walsh codes come from.

Complex vectors need one change. The inner product conjugates one of the two vectors, <u, v> = u₁v₁^* + ... + u_nv_n^*, which is v^Hu in matrix form. Take u = (1, j) and v = (1, -j). With the conjugate, the result is 1 + j x j = 0, so the two vectors are orthogonal. Without the conjugate, the result is 1 + j x (-j) = 2, and the test gives the wrong answer.

A zero inner product means orthogonal, in any dimension : this is the working definition, and the inner product page covers the operation itself.
Conjugate one vector for complex data : I/Q samples, channel vectors and OFDM symbols are complex, so the plain element-by-element sum is not enough.
Orthonormal adds unit length : a set of vectors is orthonormal when every pair is orthogonal and every vector has length 1. The columns of an orthogonal matrix form such a set.

How can two functions be orthogonal ?

A function can be seen as a vector with one element for every value of x. Once we accept that view, the sum in the inner product turns into an integral, and orthogonality carries over to functions without any new idea.

We can define orthogonality not only for two vectors but also for any two functions. Let's suppose we have two function f(x) and g(x) as follows.

Two functions f(x) and g(x) between a and b

If you multiply two functions and integrate over a certain span and the result is zero, we say the two functions are orthogonal over that span. For example, if we have two function that satisfy the following relation, we say f(x) and g(x) are orthogonal between 'a' and 'b'.

Integral of f(x) g(x) from a to b equals 0

The two curves in the sketch above only illustrate the notation. You cannot tell from the drawing whether their product integrates to zero. The areas where f(x)g(x) is positive and the areas where it is negative must cancel exactly over the span from a to b.

The span is part of the definition. Take f(x) = x and g(x) = 1. Over -1 to 1, the integral of x is 0, so the two functions are orthogonal there. Over 0 to 1, the same integral is 1/2, so they are not orthogonal.

The most useful orthogonal functions are sines and cosines over one full period. Over 0 to 2π, the integral of sin x cos x is 0. The integral of sin(mx) sin(nx) is also 0 for any two different integers m and n, and it is π when m = n. This set of relations is what makes the Fourier series work, because each coefficient can be found by one integral that ignores all the other terms. The Fourier transform page builds on the same idea.

For functions, the integral replaces the sum : sampling f(x) and g(x) turns the integral back into the vector inner product, apart from the sample spacing.
Orthogonality always holds over a stated span : the same two functions can be orthogonal over one interval and not over another.
Harmonics are orthogonal over a full period : this is the property that Fourier analysis, and OFDM subcarriers, depend on.

What is the practical meaning of orthogonality ?

The definitions above are clean, but an engineer needs a reason to care about them. The reason is separation. Orthogonal signals can share the same channel, and a receiver can still take them apart with one inner product each.

What is the practical meaning of the orthogonality ? This is the most important in engineering application.

If two vectors are orthogonal, it means there is zero correlation (no correlation at all) between the two vectors (or two data sets).

That statement needs a condition, and the condition is zero mean. The correlation coefficient first subtracts the mean of each data set, and only then takes the inner product. So orthogonal means uncorrelated only when both data sets have zero mean. The examples below show both ways the two ideas can differ.

The vectors (1, 2, 3) and (3, 0, -1) are orthogonal, because their inner product is 0. But their correlation coefficient is about -0.96, which is a strong negative correlation.
The vectors (1, 2, 3) and (2, 1, 2) have an inner product of 10, so they are not orthogonal. But their correlation coefficient is exactly 0, because the mean-removed vectors (-1, 0, 1) and (1/3, -2/3, 1/3) are orthogonal.

Most signals in communication have zero mean, such as spreading codes, OFDM subcarriers and noise. For them, the two statements agree, and engineers often use the words orthogonal and uncorrelated for the same thing.

Here is how the separation works. Let's send the code c₁ = (1, 1, 1, 1) with the value 3 and the code c₂ = (1, -1, 1, -1) with the value -2 at the same time. The receiver sees the sum s = 3c₁ - 2c₂ = (1, 5, 1, 5). The inner product s ⋅ c₁ is 12, and dividing by 4 gives 3. The inner product s ⋅ c₂ is -8, and dividing by 4 gives -2. Each code removes the other code completely, because the two codes are orthogonal. CDMA uses this with Walsh codes, and OFDM does the same thing with subcarriers.

Orthogonal signals can be separated without interference : an inner product with one signal removes every signal orthogonal to it.
Orthogonal and uncorrelated are equal only for zero-mean data : check the mean before using one word for the other.
Orthogonality breaks when the conditions break : a timing offset between codes, or a frequency offset between OFDM subcarriers, makes the inner product non-zero and brings the interference back.