I was able to complete the last assignment of the course and I understand how to use NumPy for linear algebra related work. What I still do not understand is the applications of the Eigenvalues and Eigenvectors. I have tried going to the the last section of the final lab again and again but do not seem to understand what is the purpose of Eigenvalues and Eigenvectors in finding probabilities. Till now, I just thought that these two features are used only in computer graphics manipulation.
Another thing in which I am confused is how come simple linear transformation and Eigenvalues are different from each other since they both do things like stretching, reflecting and similar functions.
Kindly explain my two queries in as simple words as possible. I really want to learn the mathematical foundation but I am kind of getting de-motivated.
I will try to answer your query in the simplest way I can. Please feel free to reply to this topic if you are still confused after reading it.
The very first thing I would link to mention is the fact that eigenvalues and eigenvectors are objects related to a specific linear transformation. So, given a linear transformation, we can ask for its eigenvalues and eigenvectors.
The linear transformation is indeed what you mentioned: a map that - usually - stretchs, rotates, reflects vectors. Related to a linear transformation there are the eigenvectors. Such vectors are very special for that specific transformation - they don’t change direction, only are scaled by some real number. I will show you an example, from Wikipedia:
The eigenvector is the blue vector, note that the red vector is not an eigenvector, since its direction changes.
Apart from its applications, knowing the eigenvectors and eigenvectors of a specific linear transformation helps you with a variety of calculations, since they are only “stretched”, if you can find a basis consisting of eigenvectors, you can look at this linear transformation in this new basis and it will have a “nice looking” matrix related to it - a so called diagonal matrix - where performing computations is extremely easy.
Speaking about the assignment - I would recommend not to dive too much into its intricacies for now, since Markov matrices and (discrete) dynamical systems are a whole world of theory and usually they fall out from simple explanations. The main idea of the section is to show how using eigenvectors can help you in making some computations.
I hope that helps you, if you need any further clarification, just reply to this topic and I will do my best to help you.
I was able to understand that eigenvectors are special vectors because they do not change the direction in linear transformation. But i am still confused about three things:
If they do not change directions, how can they perform operations like stretching?
Why do we have to use them as linear transformation can also perform the same task?
How can they improve computation?
I apologize if my questions sounds dumb.
If you can refer to some sources that you think might help gaining the intuition of eigenvectors and eigenvalues, I’ll appreciate it.
I did not take the Math4ML course, so I do not know the course specific context. But I still hope to contribute to answer your question:
As some physics motivation:
In structural dynamics, when you encounter a specific structure, it reacts after excitement (e.g. hitting a steel beam with a hammer): in general after the system is excited, it usually oscillates at the natural frequency of the system (= eigenfrequency) which can be characterised by eigenvalues. The modal forms of the vibration are described by the eigenvectors, which so to speak describe the direction of vibration in the space.
So, in general eigen value problems are of particular benefit if linear systems are analyzed or modelled - also in data analysis of course!
you can conduct a transformation on the eigenvectors or a subset of them (meaning you change the basis of the original coordinate system to the eigenvector coordinate system [which results in uncoupled equations - called diagonal matrix, see @lucas.coutinho’s excellent response - in structural dynamics which is nice for analysis purposes]). In the transformation case you will do a linear projection which can mean: stretching, scaling etc.
often is it used in the course of a PCA / SVD to reduce the dimensions of the feature space, see also this thread here: C3_W2 - PCA Question Often you can dramatically speed up the calculation of linear (but also linearized nonlinear) problems (like contact mechanics in structural dynamics) because here inverting a matrix (iteratively) is needed which has a cubic complexity. If you can reduce the matrix with a PCA / SVD only using the most important eigen vectors / eigen values describing most of the information (e.g 98% or so) , you barely loose any accuracy in the linear or linearised system model but you are much faster - mainly due to the model order reduction aspect when it comes to matrix inverting step.
Calculating the linear transformation by mutliplying a matrix by a vector can be very resource intensive.
Therefore, you can use the eigenvalues and eigenvectors of a given matrix and a given vector to compute the linear transformation faster and with less resources.
In addition, I would say that the eigenvalues and eigenvectors are a kind of signature or meta-information of a given matrix, because they give you an idea of what the linear transformation of the matrix looks like.
one can improve computational efficiency (one can solve the problem in a smaller dimensional space and often you need to invert a matrix in a smaller space which is cubic effort and this especially means a strong benefit for non-linear problems which you need to solve iteratively which means in total you get much faster of course)
but you sacrifice accuracy or let’s say you lose some information in this dimensionality reduction.
