# Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors & eigenvalues.

I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a machine. But I didn't understand it. I didn't get the purpose of it. I didn't get the feel of it. I strongly believe in

you do not really understand something unless you can explain it to your grandmother -- Albert Einstein

Well, I can't explain these concepts to a layman or grandma.

1. Why PCA, eigenvectors & eigenvalues? What was the need for these concepts?
2. How would you explain these to a layman?
-

14
Good question. I agree with the quote as well. I believe there are many people in statistics and mathematics who are highly intelligent, and can get very deep into their work, but don't deeply understand what they are working on. Or they do, but are incapable of explaining it to others.I go out of my way to provide answers here in plain English, and ask questions demanding plan English answers. – Neil McGuigan Sep 15 '10 at 21:43
– G. Jay Kerns Sep 16 '10 at 2:18

2
– whuber♦ Sep 16 '10 at 5:03
Similar to explanation by Zuur et al in Analyzing ecological data where they talk about projecting your hand on an overhead projector. You keep rotating your hand so that the projection on the wall looks pretty similar to what you think a hand should look like. – Roman Luštrik Sep 16 '10 at 9:00
– vonjd Oct 26 '10 at 6:25
show 2 more comments

## 18 Answers

This manuscript really helped me grok PCA. I think it's still too complex for explaining to your grandmother, but it's not bad. You should skip first few bits on calculating eigens, etc. Jump down to the example in chapter 3 and look at the graphs.

I have some examples where I worked through some toy examples so I could understand PCA vs. OLS linear regression. I'll try to dig those up and post them as well.

edit: You didn't really ask about the difference between Ordinary Least Squares (OLS) and PCA but since I dug up my notes I did a blog post about it. The very short version is OLS of y ~ x minimizes error perpendicular to the independent axis like this (yellow lines are examples of two errors):

If you were to regress x ~ y (as opposed to y ~ x in the first example) it would minimize error like this:

and PCA effectively minimizes error orthogonal to the model itself, like so:

More importantly, as others have said, in a situation where you have a WHOLE BUNCH of independent variables, PCA helps you figure out which ones matter the most. The examples above just help visualize what the first principal component looks like in a really simple case.

In my blog post I have the R code for creating the above graphs and for calculating the first principal component. It might be worth playing with to build your intuition around PCA. I tend to now really own something until I write code that reproduces it.

-

1
+1 for really caring to answer. – claws Sep 16 '10 at 18:55

1
The tutorial was really great. Could you suggest any further tutorials as a follow-up? – Edward Sep 21 '10 at 9:54

1
Good call on the Lindsay I Smith manuscript - just read it today; very helpful. – Stedy Oct 23 '10 at 5:58
So is PCA equivalent to Total Least Squares if it optimizes orthogonal distances from points to the fit line? – Marcin Apr 16 '11 at 14:25
@Marcin - this is correct. You can re-phrase PCA as finding the best rank $m$ estimate ($1\leq m\leq p$) of the original $p$ variables ($\hat{x}_{ij}\;\;\;\; i=1,\dots,n\;\;\;j=1,\dots,p$), with an objective function of $\sum_{i=1}^{n}\sum_{j=1}^{p}(x_{ij}-\hat{x}_{ij})^{2}$. Choosing the number of PCs is equivalent to choosing the rank of the predictions. – probabilityislogic Sep 5 '11 at 7:24

Let's do (2) first. PCA fits an ellipsoid to the data. An ellipsoid is a multidimensional generalization of distorted spherical shapes like eggs, cigars, and pancakes. These are all neatly described by the directions and lengths of their principal (semi-)axes, such as the axis of the cigar or egg or the plane of the pancake. No matter how the ellipsoid is turned, the eigenvectors point in those principal directions and the eigenvalues give you the lengths. The smallest eigenvalues correspond to the thinnest directions having the least variation, so ignoring them (which collapses them flat) loses relatively little information: that's PCA.

(1) Apart from simplification (above), we have needs for pithy description, visualization, and insight. Being able to reduce dimensions is a good thing: it makes it easier to describe the data and, if we're lucky to reduce them to three or less, lets us draw a picture. Sometimes we can even find useful ways to interpret the combinations of data represented by the coordinates in the picture, which can afford insight into the joint behavior of the variables.

-

1
To add to this, when you have (near-)equal semiaxes (i.e. the ellipsoid has a (near-)circular slice), it indicates that the two pieces of data corresponding to those axes have (near-)dependency; one can talk about principal axes for an ellipse, but circles only have one radius. :) – J. M. Sep 16 '10 at 9:43

1
I would be more cautious here, J.M. First, just to clarify, by "near-dependency" you must mean "nearly independent." This would be true for a multinormal variate, but in many cases PCA is performed with data that are markedly non-normal. Indeed, the clustering analyses that follow some PCA calculations can be viewed as one way to assess a strong form of non-normality. Mathematically, circles do have principal axes, but they are just not uniquely determined: you can choose any orthogonal pair of radii as their principal axes. – whuber♦ Sep 16 '10 at 14:11
Yes, sorry, I suppose "the principal axes of a circle are indeterminate" would have been a better way of putting it. – J. M. Sep 16 '10 at 16:13

2
+1 for the geometric explanation. – vqv Dec 20 '10 at 4:49

Hmm, here goes for a completely non-mathematical take on PCA...

Imagine you have just opened a cider shop. You have 50 varieties of cider and you want to work out how to allocate them onto shelves, so that similar-tasting ciders are put on the same shelf. There are lots of different tastes and textures in cider - sweetness, tartness, bitterness, yeastiness, fruitiness, clarity, fizziness etc etc. So what you need to do to put the bottles into categories is answer two questions:

1) What qualities are most important for identifying groups of ciders? e.g. does classifying based on sweetness make it easier to cluster your ciders into similar-tasting groups than classifying based on fruitiness?

2) Can we reduce our list of variables by combining some of them? e.g. is there actually a variable that is some combination of "yeastiness and clarity and fizziness" and which makes a really good scale for classifying varieties?

This is essentially what PCA does. Principal components are variables that usefully explain variation in a data set - in this case, that usefully differentiate between groups. Each principal component is one of your original explanatory variables, or a combination of some of your original explanatory variables.

-

1
What about the eigenvectors & eigenvalues? – Ηλίας Oct 14 '10 at 8:29
Okay: the Eigenvalue associated with each principal component tells you how much variation in the data set it explains (in my example, how clearly it separates your bottles into groups). They are usually expressed as a percentage of the total variation in the data set. As for the Eigenvectors, well, that's where as claws said I follow the output of an analysis like a machine ;) In my head, they are related to how you rotate Vince's mobile to its 'best' orientation, but this might not be the right way to think of them. – Freya Harrison Oct 27 '10 at 13:07

3
Eigenvectors are just the linear combinations of the original variables (in the simple or rotated factor space); they described how variables "contribute" to each factor axis. Basically, think of PCA as as way to construct new axes that point to the directions of maximal variance (in the original variable space), as expressed by the eigenvalue, and how variables contributions are weighted or linearly transformed in this new space. – chl♦ Nov 23 '10 at 21:46

Alright, I'll give this a try. A few months back I dug through a good amount of literature to find an intuitive explanation I could explain to a non-statistician. I found the derivations that use Lagrange multipliers the most intuitive.

Let's say we have high dimension data - say 30 measurements made on an insect. The bugs have different genotypes and slightly different physical features in some of these dimensions, but with such high dimension data it's hard to tell which insects belong to which group.

PCA is a technique to reduce dimension by:

1. Taking linear combinations of the original variables.
2. Each linear combination explains the most variance in the data it can.
3. Each linear combination is uncorrelated with the others

Or, in mathematical terms:

1. For $Y_j = a_j' x$ (linear combination for jth component)
2. For $k > j$, $V(Y_k) < V(Y_j)$ (first components explain more variation)
3. $a_k' a_j = 0$ (orthogonality)

Finding linear combinations that satisfy these constraints leads us to eigenvalues. Why?

I recommend checking out the book An Introduction to Multivariate Data Analysis for the full derivation (p. 50), but the basic idea is successive optimizations problems (maximizing variance) constrained such that a'a = 1 for coefficients a (to prevent the case when variance could be infinite) and constrained to make sure the coefficients are orthogonal.

This leads to optimization with Lagrange multipliers, which in turn reveals why eigenvalues are used. I am too lazy to type it out (sorry!) but, this PDF goes through the proof pretty well from this point.

I would never try to explain this to my grandmother, but if I had to talk generally about dimension reduction techniques, I'd point to this trivial projection example (not PCA). Suppose you have a Calder mobile that is very complex. Some points in 3-d space close to each other, others aren't. If we hung this mobile from the ceiling and shined light on it from one angle, we get a projection onto a lower dimension plane (a 2-d wall). Now, if this mobile is mainly wide in one direction, but skinny in the other direction, we can rotate it to get projections that differ in usefulness. Intuitively, a skinny shape in one dimension projected on a wall is less useful - all the shadows overlap and don't give us much information. However, if we rotate it so the light shines on the wide side, we get a better picture of the reduced dimension data - points are more spread out. This is often what we want. I think my grandmother could understand that :-)

-

4
That's very layman ;-) – mbq♦ Sep 15 '10 at 21:24

1
It's a little mathy, but the best way to understand something is to derive it. – Vince Sep 16 '10 at 1:08

8
You have an exceptionally well-educated grandmother :-). – whuber♦ Sep 16 '10 at 1:44

1
i like the explanation with the light shining on a 3-d structure – Neil McGuigan Jun 7 '11 at 18:40

OK, a totally non-math answer:

If you have a bunch of variables on a bunch of subjects and you want to reduce it to a smaller number of variables on those same subjects, while losing as little information as possible, then PCA is one tool to do this.

It differs from factor analysis, although they often give similar results, in that FA tries to recover a small number of latent variables from a larger number of observed variables that are believed to be related to the latent variables.

-
Hey Peter! Good to see you here. This is a really good, simple, no math answer. – JD Long Sep 16 '10 at 19:40

1
+1 for mentioning FA, which no one else seems to discuss, and which some people's explanations seem to blend towards. – gung Jan 31 '12 at 3:52

I'd answer in "layman's terms" by saying that PCA aims to fit straight lines to the data points (everyone knows what a straight line is). We call these straight lines "principal components". There are as many principal components as there are variables. The first principal component is the best straight line you can fit to the data. The second principal component is the best straight line you can fit to the errors from the first principal component. The third principal component is the best straight line you can fit to the errors from the first and second principal components, etc., etc.

If someone asks what you mean by "best" or "errors", then this tells you they are not a "layman", so can go into a bit more technical details such as perpendicular errors, don't know where the error is in x- or y- direction, more than 2 or 3 dimensions, etc. Further if you avoid making reference to OLS regression (which the "layman" probably won't understand either) the explanation is easier.

The eigenvectors and eigenvalues are not needed concepts per se, rather they happened to be mathematical concepts that already existed. When you solve the mathematical problem of PCA, it ends up being equivalent to finding the eigenvalues and eigenvectors of the covariance matrix.

-

+1, this is truly in "layman's terms", and I know you could derive it very rigorously if you wanted to! – gung Jan 31 '12 at 3:53

Trying to be non-technical... Imagine you have a multivariate data, a multidimensional cloud of points. When you compute covariance matrix of those you actually (a) center the cloud, i.e. put the origin it the multidimensional mean, the coordinate system axes now cross in the centre of the cloud, (b) encrypt the information about the shape of the cloud and how it is oriented in the space by means of variance-covariance entries. So, all important info about the data as a whole - i.e. except information of each individual data point - is stored in the covariance matrix.

Then you do eigen-decomposition of that martrix and obtain the list of eigenvalues and the corresponding number of eigenvectors. Now, the 1st principal component is the new, latent variable which can be displayed as the axis going through the origin and oriented along the direction of the maximal variance (thickness) of the cloud. The variance along this axis, i.e. the variance of the coordinates of all points on it, is the first eigenvalue, and the orientation of the axis in space referenced to the original axes (the variables) is defined by the 1st eigenvector: its entries are the cosines between it and those original axes. The aforementioned coordinates of data points on the 1st component are the 1st pr. component values, or component scores; they are computed as the product of (centered) data matrix and the eigenvector.

"After" the 1st pr. component got measured it is, to say, "removed" from the cloud with all the variance it accounted for, and the dimensionality of the cloud drops by one. Next, everything is repeated with the second eigenvalue and the second eigenvector - the 2nd pr. component is being recorded, and then "removed". Etc.

So, once again: eigenvectors are direction cosines for principal components, while eigenvalues are the magnitude (the variance) in the principal components. Sum of all eigenvalues is equal to the sum of variances which are on the diagonal of the variance-covariance matrix. If you transfer the "magnitudinal" information stored in eigenvalues over to eigenvectors to add it to the "orientational" information stored therein you get what is called principal component loadings; these loadings - because they carry both types of information - are the covariances between the original variables and the principal components.

-

I might be a bad person to answer this because I'm the proverbial grandmother who has had the concept explained to me and not much more, but here goes:

Suppose you have a population. A large portion of the population is dropping dead of heart attacks. You are trying to figure out what causes the heart attacks.

You have two pieces of data: height and weight.

Now, it's clear that there's SOME relationship between weight and heart attacks, but the correlation isn't really strong. There are some heavy people who have a lot of heart attacks, but some don't.

Now, you do a PCA, and it tells you that weight divided by height ('body mass') is a much more likely predictor of heart attacks then either weight or height, because, lo and behold, the "reality" is that it's body mass that causes the heart attacks.

Essentially, you do PCA because you are measuring a bunch of things and you don't really know if those are really the principle components or if there's some deeper underlying component that you didn't measure.

[Please feel free to edit this if it's completely off base. I really don't understand the concept any more deeply than this].

-

1
– Shane Sep 16 '10 at 2:11

From someone who has used PCA a lot (and tried to explain it to a few people as well) here's an example from my own field of neuroscience.

When we're recording from a person's scalp we do it with 64 electrodes. So, in effect we have 64 numbers in a list that represent the voltage given off by the scalp. Now since we record with microsecond precision, if we have a 1-hour experiment (often they are 4 hours) then that gives us 10e6 * 60^3 == 216,000,000,000 time points at which a voltage was recorded at each electrode so that now we have a 216,000,000,000 x 64 matrix. Since a major assumption of PCA is that your variables are correlated, it is a great technique to reduce this ridiculous amount of data to an amount that is tractable. As has been said numerous times already, the eigenvalues represent the amount of variance explained by the variables (columns). In this case an eigenvalue represents the variance in the voltage at a particular point in time contributed by a particular electrode. So now we can say, "Oh, well electrode `x` at time point `y` is what we should focus on for further analysis because that is where the most change is happening". Hope this helps. Loving those regression plots!

-

I can give you my own explanation/proof of the PCA, which I think is really simple and elegant, and doesn't require anything except basic knowledge of linear algebra. It came out pretty lengthy, because I wanted to write in simple accessible language.

Suppose we have some $M$ samples of data from an $n$-dimensional space. Now we want to project this data on a few lines in the $n$-dimensional space, in a way that retains as much variance as possible (that means, the variance of the projected data should be as big compared to the variance of original data as possible).

Now, let's observe that if we translate (move) all the points by some vector $\beta$, the variance will remain the same, since moving all points by $\beta$ will move their arithmetic mean by $\beta$ as well, and variance is lineary proportional to $\sum_{i=1}^M ||x_i - \mu||_2$. Hence we translate all the points by $-\mu$, so that their arithmetic mean is $0$, for computational comfort. Let's denote the translated points as $x_i' = x_i - \mu$. Let's also observe, that the variance can be now expressed simply as $\sum_{i=1}^M ||x_i'||_2$.

Now the choice of the line. We can describe any line as set of points that satisfy the equation $x = \alpha * v + w$, for some vectors $v,w$. Note that if we move the line by some vector $\gamma$ orthogonal to $v$, then all the projections on the line will also be moved by $\gamma$, hence the mean of the projections will be moved by $\gamma$, hence the variance of the projections will remain unchanged. That means we can move the line parallel to itself, and not change the variance of projections on this line. Again for convenience purposes let's limit ourselves to only the lines passing through the zero points (this means lines described by $x = \alpha *v$).

Alright, now suppose we have a vector $v$ that describes the direction of a line that is a possible candidate for the line we search for. We need to calculate variance of the projections on the line $\alpha * v$. What we will need are projection points and their mean. From linear algebra we know that in this simple case the projection of $x_i'$ on $\alpha * v$ is $\frac{<x_i, v>}{||v||_2}$. Let's from now on limit ourselves to only unit vectors v. That means we can write the length of projection of point $x_i'$ on $v$ simply as $<x_i', v>$.

In some of the previous answers someone said that PCA minimizes the sum of squares of distances from the chosen line. We can now see it's true, because sum of squares of projections plus sum of squares of distances from the chosen line is equal to the sum of squares of distances from point $0$. By maximizing the sum of squares of projections, we minimize the sum of squares of distances and vice versa, but this was just a thoughtful digression, back to the proof now.

As for the mean of the projections, let's observe that $v$ is part of some orthogonal base of our space, and that if we project our data points on every vector of that base, their sum will cancel out (it's like that because projecting on the vectors from the base is like writing the data points in the new orthogonal base). So the sum of all the projections on the vector $v$ (let's call the sum $S_v$) and the sum of projections on other points from the base (let's call it $S_o$) is 0, because it's the mean of the data points. But $S_v$ is orthogonal to $S_o$! That means $S_o = S_v = 0$.

So the mean of our projections is $0$? Well, that's convenient, because that means the variance is just the sum of squares of lengths of projections, or in symbols $\sum_{i=1}^M (x_i' \cdot v)^2 = \sum_{i=1}^M v^T \cdot x_i'^T \cdot x_i' \cdot v = v^T \cdot (\sum_{i=1}^M x_i'^T \cdot x_i) \cdot v$.

Well well, suddenly the covariance matrix popped out. Let's denote it simply by $X$. It means we are now looking for a unit vector $v$, that maximizes $v^T \cdot X \cdot v$, for a semi-positive definite matrix $X$.

Now, let's take the eigenvectors and eigenvalues of matrix $X$, and denote them by $e_1, e_2, \dots , e_n$ and $\lambda_1 , \dots, \lambda_n$ respectively, such that $\lambda_1 \geq \lambda_2 , \geq \lambda_3 \dots$. If the values $\lambda$ do not duplicate, eigenvectors form an orthonormal base. If they do, we choose the eigenvectors in a way that they form an orthonormal base.

Now let's calculate $v^T \cdot X \cdot v$ for eigenvector $e_i$. We have $e_i^T \cdot X \cdot e_i = e_i^T \cdot (\lambda_i * e_i) = \lambda_i (||e_i||_2)^2 = \lambda_i$.

Pretty good, this gives us $\lambda_1$ for $e_1$. Now let's take an arbitrary vector $v$. Since eigenvectors form an orthonormal base, we can write $v = \sum_{i=1}^n e_i * <v, e_i>$, and we have $\sum_{i=1}^n <v, e_i>^2 = 1$. Let's denote $\beta_i = <v, e_i>$.

Now let's count $v^T \cdot X \cdot v$. We rewrite $v$ as a linear combination of $e_i$, and get : $(\sum_{i=1}^n \beta_i * e_i)^T \cdot X \cdot (\sum_{i=1}^n \beta_i *e_i) = (\sum_{i=1}^n \beta_i * e_i) \cdot (\sum_{i=1}^n \lambda_i * \beta_i * e_i) = \sum_{i=1}^n \lambda_i *(\beta_i)^2 * (||e_i||_2)^2$.

The last equation comes from the fact thet eigenvectors where chosen to be pairwaise orthogonal, so their dot product's are zero. Now, because all eigenvectors are also unit, we can write $v^T \cdot X \cdot v = \sum_{i=1}^n \lambda_i * \beta_i^2$, where $\beta_i ^2$ are all positive, and sum to $1$.

That means, that the variance of the projections is a weighted mean of eigenvalues. Certainly, it is always less then the biggest eigenvalue, which is why it should be our choice of the first vector.

Now suppose we want another vector. We should chose it from a space orthogonal to the already chosen one, that means the subspace $lin(e_2, e_3, \dots , e_n)$. By analogical inference we arrive at the conclusion, that the best vector to project on is $e_2$. And so on, and so on...

Btw. it should be now clear, why the variance retained can be expresed by $\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^n \lambda_i}$.

Hope this helps and I didn't make any big mistakes.

We should also justify the greedy choice of vectors. When we want too choose $k$ vectors to project onto, it might not be the best idea to first choose the best vector, then the best from what reamains and so on. I'd like to argue that in this case it is justified and makes no difference. Lets denote the $k$ vector we wish to project onto by $v_1, \dots , v_k$. Also, let's assume the vectors are pairwaise orthogonal. As we already know, the total variance of the projections on those vectors can be expressed by : $\sum_{j=1}^k \sum_{i=1}^n \lambda_i * \beta_{ij}^2 = \sum_{i=1}^n \lambda_i * \gamma_i$ where $\gamma_i = \sum_{j=1}^k \beta_{ij}^2$.

Now, let's write e_i in some orthonormal base that includes $v_1, \dots , v_k$. Let's denote the rest of the base as $u_1, \dots, u_{n-k}$. We can see that $e_i = \sum_{j=1}^k \beta_{ij} v_j + \sum_{j=1}^{n-k} \theta_j * <e_i, u_j>$. Because $||e_i||_2 = 1$, we have $\sum_{j=1}^k \beta_{ij}^2 + \sum_{j=1}^{n-k} \theta_j^2 = 1$, and hence $\gamma_i \leq 1$ for all $i$.

Now we have a similar case to one vector only, we now know that the total variance of projections is $\sum_{i=1}^n \lambda_i * \gamma_i$ with $\gamma_i \leq 1$ and $\sum_{i=1}^n \gamma_i = k$. This is yet another weighted mean, and is certainly no more than $\sum_{i=1}^k \lambda_i$ which corresponds to projecting on $k$ eigenvectors corresponding to biggest eigenvalues.

-

yes very simple:) – berkay Dec 15 '12 at 23:34
awesome and elegant – Vass Mar 11 at 12:07

Eigens are mathematical concept used for implementing PCA, so they are out of discussion. PCA is based on the idea that the multivariate data is so hard to interpret not because the reality is complex, but because we have measured wrong variables, and the reality is trivial; namely there are only linear relations involved.
To this end we use some math-magick to expose this structure, and voilà.

This is pure wishful thinking, but it often works because of the Taylor expansion.

-

I view PCA as a geometric tool. If you are given a bunch of points in 3-space which are pretty much all on a straight line, and you want to figure out the equation of that line, you get it via PCA (take the first component). If you have a bunch of points in 3-space which are mostly planar, and want to discover the equation of that plane, do it via PCA (take the least significant component vector and that should be normal to the plane).

-

Some time back I tried to understand this PCA algorithm and I wanted to make a note about eigen vectors and eigen values. That document stated that the purpose of EVs is to convert a model of the large sized model to a very small sized model.

For example, instead of constructing first the full sized bridge and then carrying out experiments and tests on it, it is possible to use EVs to create a very small sized bridge where all the factors/quantities will be reduced by the same margin and moreover the actual result of tests and stress related tests carried out on it can be calculated and enlarged appropriately as needed for the original model. In a way EVs help to create abstracts of the original.

To me, this explaination had profound meaning to what I was trying to do! Hope it helps you too!

-

The way I understand principle components is this: Data with multiple variables (height, weight, age, temperature, wavelength, percent survival, etc) can be presented in three dimensions to plot relatedness.

Now if you wanted to somehow make sense of "3D data", you might want to know which 2D planes (cross-sections) of this 3D data contain the most information for a given suite of variables. These 2D planes are the principle components, which contain a proportion of each variable.

Think of principal components as variables themselves, with composite characteristics from the original variables (this new variable could be described as being part weight, part height, part age, etc). When you plot one principle component (X) against another (Y), what you're doing is building a 2D map that can geometrically describe correlations between original variables. Now the useful part: since each subject (observation) being compared is associated with values for each variable, the subjects (observations) are also found somewhere on this X Y map. Their location is based on the relative contributions of each underlying variable (i.e. one observation may be heavily affected by age and temperature, while another one may be more affected by height and weight). This map graphically shows us the similarities and differences between subjects and explains these similarities/differences in terms of which variables are characterizing them the most.

-

Although there are myriad examples given to provide an intuitive understanding of PCA, that fact can almost make it more difficult to grasp at the outset, at least it was for me.

"What was the one thing about PCA that all these different examples from different disciplines have in common??"

What helped me intuitively understand were a couple of math parallels I found, since it's apparent the maths is the easy part for you, although this doesn't help explain it to your grandmother...

Think of a regularization problem, trying to get

$$|| XB - Y || = 0$$

Or in English, break down your data $Y$ into two other matrices which will somehow shed light on the data? If those two matrices work well, then the error between them and $Y$ shouldn't be too much.

PCA gives you a useful factorizaton of $Y$, for all the reasons other people have said. It breaks the matrix of data you have, $Y$, down into two other useful matrices. In this case, $X$ would be a matrix where the columns are first $k$ PCs you kept, and $B$ is a matrix giving you a recipe to reconstruct the columns of matrix $Y$ using the columns of $X$. $B$ is the first $k$ rows of $S$, and all of $V$ transpose. The eigenvalues on the diagonal of $S$ basically weights which PCs are most important. That is how the math explicitly tells you which PCs are the most important: they are each weighted by their eigenvalues. Then, the matrix $V^\mathrm{T}$ tells the PCs how to combine.

I think people gave many intuitive examples, so I just wanted to share that. Seeing that helped me understand how it works. There are a world of interesting algorithms and methods which do similar things as PCA. Sparse coding is a subfield of machine learning which is all about factoring matrix $A$ into two other useful and interesting ones that reflect patterns in $A$.

Anyhow have fun!

-

Basically PCA finds new variables which are linear combinations of the original variables such that in the new space, the data has fewer dimensions. Think of a data set consisting of the points in 3 dimensions on the surface of a flat plate held up at an angle. In the original x, y, z axes you need 3 dimensions to represent the data, but with the right linear transformation, you only need 2.

Basically what @Joel said, but only linear combinations of the input variables.

-

Why so eigenvalues/eigenvectors ?

When doing PCA, you want to compute some orthogonal basis by maximizing the projected variance on each basis vector.

Having computed previous basis vectors, you want the next one to be:

• orthogonal to the previous
• norm 1
• maximizing projected variance, i.e with maximal covariance norm

This is a constrained optimization problem, and the Lagrange multipliers (here's for the geometric intuition, see wikipedia page) tell you that the gradients of the objective (projected variance) and the constraint (unit norm) should be "parallel" at the optimium.

This is the same as saying that the next basis vector should be an eigenvector of the covariance matrix. The best choice at each step is to pick the one with the largest eigenvalue among the remaining ones.

-

2
Definitely not an explanation to a layman - orthogonal basis vectors? maximising projection variance? constrained optimisation problem? Lagrange multiplier? These are highly "jargonised" terms. Show a layman who understands what these mean and i'll show you a mathematician/statistician – probabilityislogic Sep 4 '11 at 22:55

Here is a math answer: the first principal component is the longest dimension of the data. Look at it and ask: where is the data widest? That's the first component. The next component is the perpendicular. So a cigar of data has a length and a width. It makes sense for anything that is sort of oblong.

-

3
Unfortunately, the correctness of this answer depends on how the vague expression "longest" is interpreted. Many natural and relevant interpretations, such as the diameter, would be wrong. – whuber♦ Mar 20 at 21:56
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 107, "mathjax_display_tex": 1, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9420769810676575, "perplexity_flag": "middle"}