Eigenvector and Eigenvalue

One of the most useful/important but very hard to understand the practical meaning would be the concept of Eigenvector and Eigenvalue. You can easily find the mathematical definition of eigenvalue and eigenvector from any linear algebra books and internet surfing.

Eigen 'in German' means 'Characteristics' in English. So you may guess, 'Eigen vector' would be a special vector that represents a specific characteristics of a Matrix (a Square Matrix) and 'Eigen value' would be a special value that represents a specific characteristics of a Matrix (a Square Matrix)

If you think of a Matrix as a geometric transformer, the Matrix usually perform two types of transformational action. One is 'scaling(extend/shrink)' and the other one is 'rotation'. (There are some additional types of transformation like 'shear', 'reflection', but these would be described by special combination of scaling and rotation).

Eigenvector and Eigenvalue can give you the information on the scaling and rotational characteristics of a Matrix. Eigenvector would give you the rotational characteristics and Eigenvalue would give you the scaling characteristics of the Matrix. (Refer to Matrix-Geometric/Graphical Meaning of Eigenvalue and Determinant)

It means with Eigenvalue and Eigenvector, you may reasonably guess about the result of geometrical transformation of a vector without really calculalting a lot of Matrix/Vector multiplication (transformation).

Mathematical Definition of Eigenvalue

Every later section on this page builds on one exact statement, so let's fix it before any intuition. The definition is a single short equation, and each symbol in it has a specific role.

I will start with the samething, i.e mathematical definition. Mathematical definition of Eigenvalue and eigenvectors are as follows.

Eigenvalue equation Ax = lambda x

Let's think about the meaning of each component of this definition. I put some burbles as shown below.

Eigenvector and eigenvalue labelled in Ax = lambda x

When a vector is transformed by a Matrix, usually the matrix changes both direction and amplitude of the vector, but if the matrix applies to a specific vector, the matrix changes only the amplitude (magnitude) of the vector, not the direction of the vector. This specific vector that changes its amplitude only (not direction) by a matrix is called Eigenvector of the matrix.

Let me try explaining the concept of eigenvector in more intuitive way. Let's assume we have a matrix called 'A'. We have 5 different vectors shown in the left side. These 5 vectors are transformed to another 5 different vectors by the matrix A as shown on the right side. Vector (1) is transformed to vector (a), Vector (2) is transformed to vector (b) and so on.

Five vectors before and after transformation by matrix A

Compare the original vector and the transformed vector and check which one has changes both its direction and magnitude and which one changes its magnitude ONLY. The result in this example is as follows. According to this result, vector (4) is the eigenvector of Matrix 'A'.

Only vector 4 keeps its direction

I hope you clearly understand the meaning of eigenvectors.

Now we know eigenvector changes only its magnitude when applied by the corresponding matrix. Then the question is "How much in magnitude it changes ?". Did it get larger ? or smaller ? exactly how much ? The indicator showing the magnitude change is called Eigenvalue. For example, if the eigenvalue is 1.2, it means that the magnitude of the vector gets larger than the original magnitude by 20% and if the eigenvalue is 0.8, it means the vector got smaller than the original vector by 20 %. The graphical presentation of eigenvalue is as follows.

Eigenvalue as the ratio of magnitudes

Now let's verbalize our Eigenvector and Eigenvalue definition.

Eigenvalue equation Ax = lambda x

Matrix multiplied to its Eigenvector is same as the Eigenvalue multiplied to its Eigenvector.

Another way to understand the meaning of the eigen vector and eigen value directly from the equation would be as follows :

Let's look into the right hand side of the equation, which is a vector x multiplied by a scalar (lamda). What does it mean ?. It mean that this is just scaling the vector and the angle of the vector does not change. It means that this part can scale the vector x but does not do any rotation to the vector.

Now let's look into the left hand side of the equation A x, which is a vector multiplied by a matrix A. When you multiply a Matrix to a vector, the vector would scale in some degree, shear in some degree and rotate in some degree at the same time. But there can be a special vectors which only scales and does not do any rotation by the matrix, that specific vector (or vectors) are called eigen vector of the matrix. You can put the statement "does not do any rotation by the matrix" in another way, saying "does not change the direction of the vector". (NOTE : You need to pay attention to the meaning of 'change the direction' in this context. In reality, eigen vector can point to opposite direction of the orignal vector which is same as 180 degree rotation. But in the context of eigen value/vector, we don't call it as 'change of the direction' / 'rotation'. We call it as 'scaling by negative number'.)

Two details of the definition are easy to miss. The first is that x must not be the zero vector. A0 = λ0 holds for every λ, so the zero vector would make every number an eigenvalue. The second is that an eigenvector fixes a direction, not a length. If x is an eigenvector, then 2x, -x and any other nonzero multiple of x are eigenvectors with the same eigenvalue. This is why software such as Matlab and Octave returns eigenvectors scaled to length 1, and why two tools can print different signs for the same eigenvector.

An eigenvector keeps its direction : A only stretches, shrinks or flips it, and the eigenvalue is the factor.
The eigenvalue is the scale factor : λ above 1 stretches, λ between 0 and 1 shrinks, and a negative λ also flips the vector.
Only the direction matters : any nonzero multiple of an eigenvector is also an eigenvector.

How do you calculate eigenvalues and eigenvectors ?

The definition tells you what an eigenvector is, but not how to find one. Let's turn Ax = λx into a calculation, because the same two steps work for any square matrix.

Move everything to one side: (A - λI)x = 0. A nonzero x can solve this only when A - λI is singular. So the first step solves the characteristic equation det(A - λI) = 0 for λ. The second step takes each λ in turn and solves (A - λI)x = 0 for x.

Let's try A = [4 1; 2 3]. The characteristic equation is (4 - λ)(3 - λ) - 2 = λ² - 7λ + 10 = 0, so λ = 2 or λ = 5. For λ = 2, the rows of A - 2I are both [2 1], so x = [1 -2]^T. For λ = 5, A - 5I = [-1 1; 2 -2], so x = [1 1]^T. In Matlab or Octave, [V,D] = eig(A) gives the same pair, with each eigenvector scaled to length 1.

Two quick checks catch most mistakes. The sum of the eigenvalues equals the trace of A, here 4 + 3 = 7 = 2 + 5. The product of the eigenvalues equals det(A), here 12 - 2 = 10 = 2 x 5.

Eigenvalues come first : they are the roots of det(A - λI) = 0, a polynomial of degree n for an n x n matrix.
Eigenvectors come second : each one spans the null space of A - λI for its own λ.
Trace and determinant check the result : the eigenvalues add up to the trace and multiply to the determinant.

What does complex number eigen value mean ?

Some real matrices give eigenvalues with an imaginary part, and the definition above seems to leave no room for them. Let's see what a complex eigenvalue says about the transformation that the matrix performs.

Simply put, this mean that there is no real valued x and lamda that satisfies the following equation

Eigenvalue equation Ax = lambda x

As mentioned above, this equation mean as follows :

there can be a special vectors which only scales and does not do any rotation by the matrix, that specific vector (or vectors) are called eigen vector of the matrix

Interpreting the statement "there is no real valued x and lamda that satisfies the following equation" based on what I mentioned above, it would mean as follows.

Every vectors (every points in the coordinate) gets rotated by the matrix.

For example, let's think about a rotational matrix as shown below. this matrix rotate every point in the coordinate by pi/4 radian.

Rotation by pi/4 applied to a shape

If you do the calculation of eigen value and eigen vector for this matrix, you would get complex numbered eigen value and eigen vectors as shown below.

>> M = [cos(pi/4) -sin(pi/4);sin(pi/4) cos(pi/4)]

M =

0.7071 -0.7071

0.7071 0.7071

>> [V,D,W] = eig(M)

V =

0.7071 + 0.0000i 0.7071 + 0.0000i

0.0000 - 0.7071i 0.0000 + 0.7071i

D =

0.7071 + 0.7071i 0.0000 + 0.0000i

0.0000 + 0.0000i 0.7071 - 0.7071i

W =

-0.7071 + 0.0000i -0.7071 + 0.0000i

0.0000 + 0.7071i 0.0000 - 0.7071i

In other words, if you get the complex eigen values you can use it as the indicator that the matrix performs a rotation operation and use it as an important information about the matrix. Look into this page and see how you can utilize / interpret the complex valued eigen value/vector.

The numbers in D carry both parts of the transformation. The eigenvalues are 0.7071 +/- 0.7071i, which is e^+/-jπ/4. Their magnitude is 1, so the matrix does not scale anything. Their angle is π/4, which is exactly the rotation angle of M. In general, a real 2 x 2 matrix with the eigenvalues re^+/-jθ acts as a rotation by θ combined with a scaling by r, when you describe it in a suitable pair of axes.

Complex eigenvalues of a real matrix always come in conjugate pairs, as D shows, and so do the eigenvectors in V. The two columns of V are also orthogonal under the complex inner product, because a rotation matrix is a normal matrix. The orthogonality section below comes back to this point.

A complex eigenvalue means rotation : no real direction survives the transformation unchanged.
The magnitude and the angle split the job : |λ| is the scaling and the angle of λ is the rotation.
Real matrices give conjugate pairs : if λ is an eigenvalue of a real matrix, so is its complex conjugate.

Why Eigenvalue/Eigenvector ?

Then very important question would be "Why we need this kind of Eigenvector / Eigenvalue ?" and "When do we use Eigenvector / Eigenvalue ?".

The answer to this question cannot be done in a short word, the best way is to collect as many examples as possible to use these eigenvector/eigenvalues. You can find one example in this page, the section Geometric/Graphical Meaning of Eigenvalue and Determinant

i) When you have a situation in which you have to multiply a matrix to a vector repeatedly (for example, let's assume that you have to multiply a matrix 100 times), it would require a lot of calculation. But you can predict this result just by looking at the eigenvalues without doing 100 times matrix multiplication. (You can see this kind of cases a lot in stochastics).
ii) You can use the eigenvectors and eigenvalues to get the solution of linear differential equations (see here). // Maybe this would be one of the most important and common application of Eigenvalues and Eigenvectors.
iii) In computer graphics, a matrix is multiplied to several hundreds of points (vectors) to change the shape of an image represented by the points. By analyzing the eigenvectors of the matrix (transformation matrix), you can predict the overall result of the change of the shape without doing the several hundreds matrix x vector multiplication. (see here)

Let's make case i) concrete with a two state Markov chain. A system stays in state 1 with probability 0.9 and moves to state 2 with probability 0.1. From state 2 it moves back with probability 0.5. The transition matrix is P = [0.9 0.1; 0.5 0.5], and a probability row vector p moves one step as pP.

The eigenvalues of P are 1 and 0.4. The eigenvector for 1 gives the long run distribution, p = [5/6 1/6] = [0.833 0.167]. The other eigenvalue says how fast the chain gets there, because the difference shrinks as 0.4ⁿ. Starting from [1 0], the distribution is [0.835 0.165] after 5 steps and [0.8333 0.1667] after 20 steps. You can read all of this from the two eigenvalues without multiplying P twenty times.

Case ii) follows the same pattern. The solution of dx/dt = Ax is a sum of terms e^λt, one for each eigenvalue, so the signs of the real parts decide whether the system is stable. The Eigen Decomposition page shows how both results follow from A = QΛQ^-1.

Repeated multiplication is ruled by the eigenvalues : the largest one sets the long run behaviour and the next one sets the speed of convergence.
Differential equations are ruled by the eigenvalues too : each eigenvalue gives one mode e^λt of the solution.
Eigenvectors are the natural axes : along them, a complicated matrix acts like a set of independent scalars.

Are Eigenvectors orthogonal to each other ?

The answer is 'Not Always'. Eigenvectors of a matrix is always orthogonal to each other only when the matrix is normal, and a real symmetric matrix is the most common normal matrix. One of the examples of real symmetric matrix which gives orthogonal eigen vectors is Covariance Matrix (See this page to see how the eigenvectors / eigenvalues are used for Covariance Matrix).

NOTE : compare this with Singular Value matrix in SVD(Singular Value Decomposition). The two matrix(U and V) obtained by SVD always gives you the orthogonal vectors whereas the matrix (Q) obtained by Eigen Decomposition gives you the orthogonal vector only in special case (the special case is when the matrix is normal, for example real symmetric).

A normal matrix is one that commutes with its conjugate transpose, AA^H = A^HA. Real symmetric matrices are normal, and so are rotation matrices and Hermitian matrices. When a normal matrix has a repeated eigenvalue, its eigenvectors for that eigenvalue are not automatically orthogonal, but they can always be chosen orthogonal.

Let's compare two examples. The symmetric matrix [2 1; 1 2] has the eigenvectors [1 -1]^T and [1 1]^T, and their inner product is 0. The matrix [4 1; 2 3] is not normal. Its eigenvectors [1 -2]^T and [1 1]^T have the inner product -1, so the angle between them is 108.4 deg, not 90 deg. The rotation matrix in the complex eigenvalue section is not symmetric, but it is normal, and its two complex eigenvectors are orthogonal.

Orthogonal eigenvectors need a normal matrix : real symmetric, Hermitian and rotation matrices all qualify.
Most matrices are not normal : their eigenvectors are independent but lean toward each other, as in the 108.4 deg example.
SVD always gives orthogonal vectors : so it is the tool to use when orthogonality matters and the matrix is not normal.

Can I reconstruct the orignal matrix from eigenvectors and eigenvalues ?

If eigenvectors and eigenvalues are the fundamental characteristics of a matrix. You may think it may be possible to reconstruct a matrix from its eigenvalues and eigenvectors.

Let's suppose that you have eigenvectors and eigenvalues. Using these information, you can reconstruct the original matrix as follows based on the definition of eigen decomposition.

i) construct a matrix (let's call this Q) using the eigenvectors as column vectors

ii) construct a diagonal matrix (let's call this S (sigma)) using eigenvalues.

iii) reconstruct the orignal matrix (let's call this M) by performing Q *S * inv(Q) // '*' denote matrix multiplication.

Let's take an example.

I am using this example matrix from Guzinta Math/Eigenvalue Decomposition so that you can see the same thing with more diverse aspects.

Let's assume that we have two eigentvectors for a 2 x 2 matrix [-1,2] and [-1,1], and two eigen values -2,-1.

Using these information you can construct Q and S (Q and S is the matrix mentioned the above procedure).

Q = [-1 -1

2 1]

S = [-2 0

0 -1]

Using these two matrix, you can reconstruct the original matrix M using Q *S * inv(Q). I have done this in Wolfram Alpha as shown below.

Wolfram Alpha reconstruction of the matrix from eigenvectors and eigenvalues

The capture above shows the product Q x S x Q^-1 and its result, M = [0 1; -2 -3]. Two of its side results confirm the eigenvalues at once. The trace is -3, which is -2 + -1, and the determinant is 2, which is -2 x -1. Wolfram Alpha names the factors S and J, where J is the diagonal matrix of eigenvalues, and it gives Q^-1 = [1 1; -2 -1].

Now let's check if the reconstructed matrix produce the eigenvalues and eigenvectors same as that we used. I did this with Wolfram Alpha as shown below.

Wolfram Alpha eigenvectors of the reconstructed matrix

The second capture returns the eigenvectors (-1, 2) and (-1, 1) with the eigenvalues -2 and -1, which are the values the reconstruction started from. The matrix M also has a meaning of its own. It is the state matrix of y'' + 3y' + 2y = 0 with the state [y y']^T, and its eigenvalues -1 and -2 are the roots of λ² + 3λ + 2 = 0.

The reconstruction works only when the eigenvectors are linearly independent, because Q must have an inverse. A matrix with too few independent eigenvectors, such as [1 1; 0 1], cannot be rebuilt this way.

Eigenvectors and eigenvalues determine the matrix : M = Q S Q^-1, as long as Q is invertible.
The order must match : the i-th column of Q belongs to the i-th diagonal entry of S.

Reference

For more examples, refer to

For useful video, try followings :