Principal Component Analysis Revisited
Principal component analysis, or PCA, is the most widely used linear method for dimensionality reduction. It rotates the data onto new axes that capture as much variance as possible.
What the components are
The new axes are the principal components. They are orthogonal directions ordered by how much variance they explain.
- The first component points along the direction of greatest spread.
- The second component is perpendicular to the first and captures the next most variance.
- Later components capture progressively less.
Keeping only the top few components compresses the data while losing little information.
How it is computed
PCA centers the data, then finds the eigenvectors of the covariance matrix. Those eigenvectors are the components, and their eigenvalues give the variance along each. Projecting the data onto the top components yields the reduced representation.
Practical points
- Always standardize features first when they have different units, or large scale features dominate.
- The explained variance ratio tells you how much information each component retains.
- PCA is linear, so it cannot unfold curved manifolds the way nonlinear methods can.
Key idea
PCA rotates data onto orthogonal axes of maximal variance, letting you keep the top components to compress while preserving most information.