Linear Independence is an indicator of showing the relationship among two or more vectors. Putting it simple, "Linear Independence" imply "No correlation between/among the vectors". The mathematical definition of linear independence is as follows.

Let's read the definition slowly. The choice a₁ = a₂ = ... = a_n = 0 always satisfies the equation, so that solution tells you nothing. The real question is whether any other choice works. If one does, at least one coefficient, say a_k, is not zero. You can then divide by a_k and write v_k as a combination of the other vectors. So "linearly dependent" means that at least one vector adds no new direction. "Linearly independent" means that every vector adds a direction that the others cannot reach.

Keep in mind that "no correlation" above is a loose picture, not the definition. Two vectors can point in almost the same direction and still be linearly independent. For example, the vectors [1, 1] and [2, 1] used later on this page are only about 18.4 degrees apart, and they are independent. Nonzero orthogonal vectors are always independent, but independent vectors do not have to be orthogonal.

• How does the definition work for two vectors?
• How do you check independence with concrete numbers?
• Why does linear independence matter?

How does the definition work for two vectors?

Two vectors are the smallest case where the definition can fail, and they can be drawn on paper. So let's use them to see what the equation means as a picture. The whole question becomes whether a₁v₁ and a₂v₂ can cancel each other.

Like many other mathematical definitions, it is hard to grasp a clear understanding without going through examples.

Let's suppose we have two vectors and want to check if the two vectors are 'linearly independent" or not.

Applying the two vectors into the definition of linear independecy, we can express it as follows.

If I assume the two vectors are 2x1 vector, we can describe each component of the above mathemtical expressions as follows. As you see in this example, if we have only two vectors and the direction of vectors are different, they are 'linearly independent'.

If the two vectors are aligned in the same direction or in completely opposite direction (180 degree difference), we can easily find a non-zero a1,a2 value to make these two vectors 'NOT linear independent'.

The two pictures above lead to a simple rule for two vectors. Two vectors are linearly dependent exactly when one of them is a scalar multiple of the other. In that case both arrows lie on one line through the origin. The zero vector is a special case. It is a multiple of every vector, because 0 = 0 x v, so any set that contains the zero vector is dependent.

For two vectors in the plane, you can test the rule with a single number. Write v₁ = [p, q] and v₂ = [r, s], and put them as the columns of a 2 x 2 matrix. The determinant of that matrix is ps - qr. Its absolute value is the area of the parallelogram that the two vectors span. When the two arrows lie on one line, the parallelogram is flat and the area is zero. So the two vectors are independent exactly when ps - qr is not zero.

• Two vectors are dependent only when they lie on one line : same direction and opposite direction both count, as in the picture above.
• A set that contains the zero vector is always dependent : a nonzero coefficient on the zero vector gives the zero sum by itself.
• The determinant gives a one-number test in the plane : ps - qr is zero exactly when the two vectors are dependent.

How do you check independence with concrete numbers?

Pictures work for two vectors in the plane, but they do not scale to more vectors or more dimensions. The general method is to write the definition as a set of simultaneous equations in a₁, a₂, ... and solve it. Let's do this once with numbers.

Now let's see another example showing concrete numbers. Let's assume that we have two vectors as shown below.

Let's plug these two vector into the definition of linear independence. It becomes as follows.

If we plug the values into the expression, we get following expression.

Now the question is "Can we find any non-zero a1, a2 to satisfy this equation ?" and the result and the conclusion comes as follows.

You can solve these two equations by hand. Subtract the second equation from the first, and you get a₂ = 0. The second equation then gives a₁ = 0. So a₁ = a₂ = 0 is the only solution, and the two vectors are independent. The determinant test agrees. With v₁ and v₂ as columns, the matrix is [1 2; 1 1], and its determinant is 1 x 1 - 2 x 1 = -1, which is not zero.

Now let's compare this with a dependent pair, v₁ = [1, 2] and v₂ = [2, 4]. The same steps give the two equations below.

 a1 + 2a2 = 0
2a1 + 4a2 = 0      (this is 2 x the first equation)

a1 = 2, a2 = -1  :  2 x [1, 2] - [2, 4] = [0, 0]
det [1 2; 2 4] = 1 x 4 - 2 x 2 = 0

The second equation is only a multiple of the first, so there is really one equation for two unknowns. It has infinitely many solutions, and a₁ = 2, a₂ = -1 is one of them. The vectors are therefore dependent, because v₂ is simply 2v₁. In Matlab, rank([1 2; 1 1]) returns 2 and rank([1 2; 2 4]) returns 1. The rank counts the independent columns, so it gives the same answer without solving anything by hand.

• The test is a homogeneous system of equations : the vectors are independent when the only solution is all coefficients equal to zero.
• A dependent set gives infinitely many solutions : any multiple of one nonzero solution, such as a₁ = 2, a₂ = -1, also works.
• rank and det give the same answer faster : full rank, or a nonzero determinant for a square matrix, means independent columns.

Why does linear independence matter?

Then the last question would be "Why the linear independency is important ?", "How do we utilise this concept ?".  The importance of this concept would be for calculating the Rank of a matrix or for investigating the existence of solution of a simultaneous equation.

Let's start with the rank. The rank of a matrix is the largest number of linearly independent columns in it. It is also the largest number of linearly independent rows, and the two counts are always equal. With more than two vectors, you need this count, because checking the vectors in pairs is not enough. Take the three vectors below. None of them is a multiple of another, and yet they are dependent.

v1 = [1, 0, 1],  v2 = [0, 1, 1],  v3 = [1, 1, 2]

v3 = v1 + v2   so   1 x v1 + 1 x v2 - 1 x v3 = 0
rank([v1 v2 v3]) = 2,   det([v1 v2 v3]) = 0

The third vector lies in the plane of the first two, so the three vectors span only a plane and not the whole 3-dimensional space. This also shows a hard limit. In a space of n dimensions, at most n vectors can be independent. So three vectors in the plane are always dependent, whatever their values are.

Next, the simultaneous equation. Take a square system Ax = b with n equations and n unknowns. It has exactly one solution for every b when the columns of A are linearly independent. The same condition means that A has rank n, that det(A) is not zero, and that A^-1 exists. When the columns are dependent, the system has either no solution or infinitely many, depending on b. The same idea appears in the least square method. There, A^TA can be inverted only when the columns of A are independent.

In numerical work, the hard case is a set that is almost dependent. The determinant is then very small, and rounding errors can make it zero or nonzero by chance. For this reason, Matlab computes rank() from the singular values with a small tolerance, instead of testing the determinant for zero.

• Rank counts independent columns : and it always equals the number of independent rows.
• Pairwise checks are not enough for three or more vectors : v₃ = v₁ + v₂ is dependent even though no vector is a multiple of another.
• Independent columns mean a unique solution : for a square matrix this is the same as rank n, a nonzero determinant and an existing inverse.
• n dimensions allow at most n independent vectors : any larger set is dependent before you compute anything.