Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms a dataset into a lower-dimensional space while retaining as much variance as possible. Instead of selecting or eliminating features directly, PCA creates new features — Principal Components — that are linear combinations of the original features.

What is a linear combination?

A linear combination means multiplying each original column by a weight and summing them to produce a brand-new column. That new column is your Principal Component.

Key ideas:

PCA finds the directions of maximum variance in the data and re-expresses the data along those directions.
The $1^{s t}$ PC captures the most variance; each subsequent PC captures the most remaining variance while being orthogonal (uncorrelated) to all previous ones.
The result is a coordinate system perfectly aligned to the structure of your data.

I. Intuition: What is a Principal Component?

The scenario below builds intuition for what PCs are and what the weights (loadings) mean — before we touch any math.

Imagine a dataset of students with three features: $X_{1}$ (Math), $X_{2}$ (Physics), $X_{3}$ (Literature).

The First Principal Component ( $P C_{1}$ )

PCA finds that Math and Physics move together (students good at one tend to be good at the other). It captures this as:

P C_{1} = (0.70 \times X_{1}) + (0.70 \times X_{2}) + (0.05 \times X_{3})

A student scoring 90 in Math, 95 in Physics, 60 in Literature gets: $(0.70 \times 90) + (0.70 \times 95) + (0.05 \times 60) = 132.5$
The high weights on Math and Physics make $P C_{1}$ a natural index of "STEM Aptitude" — an emergent interpretation, not something we assigned.

The Second Principal Component ( $P C_{2}$ )

PCA then finds the biggest remaining pattern orthogonal to $P C_{1}$ :

P C_{2} = (- 0.10 \times X_{1}) + (- 0.10 \times X_{2}) + (0.95 \times X_{3})

Literature gets weight $0.95$ — $P C_{2}$ naturally becomes an index of "Humanities Aptitude".
A student brilliant at Literature but weak in STEM will score high here and low on $P C_{1}$ .

What does "lower-dimensional space" mean?

We went from 3 original columns ( $X_{1}, X_{2}, X_{3}$ ) to 2 PCs ( $P C_{1}, P C_{2}$ ) that together capture most of the variance. Each student is now described by 2 numbers instead of 3 — with minimal information loss.

II. The Algorithm: Step-by-Step

Step	Operation	Purpose
1	Standardize	If you have data measuring a house's square footage (in the thousands) and number of bedrooms (1 to 5), the math will naturally favor the larger numbers. We standardize the data so every feature has a mean of 0 and an equal scale. ➛ Remove scale bias ( $μ = 0, σ = 1$ )
2	Covariance Matrix	The algorithm checks how all the variables relate to each other. Do they increase together? Move in opposite directions? This matrix acts as a map of the relationships between all your features.
3	Eigen-Decomposition	From the covariance matrix, the math calculates two things: - Eigenvectors: (directions) The actual directions or "angles" of the new axes (like the solid black line in the image above). - Eigenvalues (magnitude of variance): A number attached to each Eigenvector that tells us exactly how much variance (information) that specific line captures.
4	Sort & Select top $k$	The algorithm sorts the Principal Components by their Eigenvalues from highest to lowest. The first component always captures the most information. - Feature Selection: You then decide how many components to keep (e.g., keeping the top 2 out of 50) and discard the rest. - Computing new features: Keep only the components that explain ~90–95% of variance
5	Project	Multiply standardized data by the $k$ eigenvectors to get PC scores. You now have a smaller, compressed dataset!

How to choose $k$ ?

Scree Plot: Look for the "elbow" where variance gain drops sharply.
Cumulative Explained Variance: Choose $k$ such that $\approx 90$ – $95 %$ of total variance is retained.

Always project onto standardized data

The eigenvectors are computed from the standardized data, so they are "anchored" at the origin of that scaled space. Projecting raw (unscaled) data onto them produces meaningless scores. Every data point — including future unseen data in a pipeline — must be standardized with the same mean and std before projection.

III. Worked Example — Full Scale (167 countries, 8 features → 5 PCs)

A complete walkthrough from raw data to the final reduced feature matrix.

Dataset: 167 countries described by 8 socioeconomic indicators. Goal: reduce to 5 PCs retaining ~95% of variance.

#	child_mort	exports	health	imports	income	inflation	life_expec	total_fer
0	90.2	10.0	7.58	44.9	1610	9.44	56.2	5.82
1	16.6	28.0	6.55	48.6	9930	4.49	76.3	1.65
2	27.3	38.4	4.17	31.4	12900	16.10	76.5	2.89
3	119.0	62.3	2.85	42.9	5900	22.40	60.1	6.16
4	10.3	45.5	6.03	58.9	19100	1.44	76.8	2.13
...	...	...	...	...	...	...	...	...

Step 1 — Standardize

z = \frac{x - μ}{σ}

Features like income (in thousands) would otherwise dominate health (single digits). After standardization every feature has mean $0$ , std $1$ :

#	child_mort	exports	health	imports	income	inflation	life_expec	total_fer
0	1.29	-1.14	0.28	-0.08	-0.81	0.16	-1.62	1.90
1	-0.54	-0.48	-0.10	0.07	-0.38	-0.31	0.65	-0.86
2	-0.27	-0.10	-0.97	-0.64	-0.22	0.79	0.67	-0.04
3	2.01	0.78	-1.45	-0.17	-0.59	1.39	-1.18	2.13
4	-0.70	0.16	-0.29	0.50	0.10	-0.60	0.70	-0.54
...	...	...	...	...	...	...	...	...

Step 2 — Covariance Matrix

Σ_{i j} = \frac{1}{n - 1} \sum_{k = 1}^{n} z_{k i} \cdot z_{k j}

Columns/rows order: [child_mort, exports, health, imports, income, inflation, life_expec, total_fer]

	0	1	2	3	4	5	6	7
0	1.01	-0.32	-0.20	-0.13	-0.53	0.29	-0.89	0.85
1	-0.32	1.01	-0.12	0.74	0.52	-0.11	0.32	-0.32
2	-0.20	-0.12	1.01	0.10	0.13	-0.26	0.21	-0.20
3	-0.13	0.74	0.10	1.01	0.12	-0.25	0.05	-0.16
4	-0.53	0.52	0.13	0.12	1.01	-0.15	0.62	-0.50
5	0.29	-0.11	-0.26	-0.25	-0.15	1.01	-0.24	0.32
6	-0.89	0.32	0.21	0.05	0.62	-0.24	1.01	-0.77
7	0.85	-0.32	-0.20	-0.16	-0.50	0.32	-0.77	1.01

Reading the covariance matrix

child_mort ↔ life_expec = $- 0.89$ : high child mortality → shorter life expectancy.
child_mort ↔ total_fer = $+ 0.85$ : high-mortality countries also have higher fertility.
exports ↔ imports = $+ 0.74$ : countries that trade more, do so in both directions.

Step 3 — Eigen-Decomposition

Solve $Σ v = λ v$ to get 8 eigenvalue–eigenvector pairs.

3a. Eigenvalues — how much variance each PC captures:

PC	Eigenvalue $λ$	Explained Variance	Cumulative
1	3.5962	44.68%	44.68%
2	1.5532	19.30%	63.98%
3	1.1704	14.54%	78.52%
4	0.7432	9.23%	87.76%
5	0.5656	7.03%	94.79% ✂ keep
6	0.2248	2.79%	97.58%
7	0.1092	1.36%	98.94%
8	0.0856	1.06%	100%

The first 5 PCs reach ~95% — a standard threshold. PCs 6–8 add noise more than signal.

3b. Eigenvectors (Loadings Matrix) — the weights that define each PC:

Each column is one eigenvector. Each entry is that feature's loading (contribution weight) for that PC.

Feature	PC1	PC2	PC3	PC4	PC5
child_mort	-0.47	0.21	-0.10	-0.12	0.30
exports	0.31	0.61	0.15	-0.10	0.06
health	0.14	-0.24	-0.65	-0.68	-0.06
imports	0.19	0.66	-0.29	-0.06	-0.32
income	0.39	0.03	0.25	-0.32	0.73
inflation	-0.22	0.01	0.62	-0.62	-0.42
life_expec	0.46	-0.24	0.16	-0.00	-0.09
total_fer	-0.46	0.18	-0.05	-0.16	0.30

Interpreting the loadings

PC1 (45%): child_mort (−0.47), life_expec (+0.46), total_fer (−0.46), income (+0.39) — this axis separates wealthy, healthy countries from poor, high-mortality ones.
PC2 (19%): exports (+0.61), imports (+0.66) — this axis captures trade openness.
PC3 (15%): inflation (+0.62), health (−0.65) — an inflation vs. healthcare spending tension axis.
PC5 (7%): income (+0.73) — picks up residual income variation not captured by PC1.

Step 4 — Project onto the Top 5 PCs (Derive New Features)

Stack the 5 selected eigenvectors into a projection matrix $W$ of shape $(8 \times 5)$ .

\underset{n \times 5}{\underset{⏟}{Z_{new}}} = \underset{n \times 8}{\underset{⏟}{Z_{scaled}}} \cdot \underset{8 \times 5}{\underset{⏟}{W}}

Worked calculation for country 0 (standardized row: $[1.29, - 1.14, 0.28, - 0.08, - 0.81, 0.16, - 1.62, 1.90]$ , PC1 loadings: $[- 0.47, 0.31, 0.14, 0.19, 0.39, - 0.22, 0.46, - 0.46]$ ):

\begin{aligned} P C 1_{0} & = \underset{child_mort}{\underset{⏟}{(1.29 \times - 0.47)}} + \underset{exports}{\underset{⏟}{(- 1.14 \times 0.31)}} + \underset{health}{\underset{⏟}{(0.28 \times 0.14)}} + \underset{imports}{\underset{⏟}{(- 0.08 \times 0.19)}} + \underset{income}{\underset{⏟}{(- 0.81 \times 0.39)}} + \underset{inflation}{\underset{⏟}{(0.16 \times - 0.22)}} + \underset{life_expec}{\underset{⏟}{(- 1.62 \times 0.46)}} + \underset{total_fer}{\underset{⏟}{(1.90 \times - 0.46)}} \\ = - 0.607 + (- 0.353) + 0.039 + (- 0.015) + (- 0.316) + (- 0.035) + (- 0.745) + (- 0.874) = - 2.91 \end{aligned}

Repeat for all 5 PCs and all 167 rows:

Country	PC1	PC2	PC3	PC4	PC5
0	-2.91	0.16	-0.91	-0.36	0.40
1	0.72	-0.66	-0.11	0.62	-0.67
2	-0.10	-0.48	1.36	0.32	-0.39
3	-3.00	1.79	1.31	-0.33	0.53
4	1.18	0.08	-0.07	0.66	-0.24
...	...	...	...	...	...

Reading the PC scores

Countries 0 & 3 (PC1 ≈ −3): sit at the "poor, high-mortality" pole of PC1.
Country 4 (PC1 = +1.18): wealthier, healthier.
Country 3 (PC2 = +1.79): relatively high trade openness compared to its PC1 peers.
These 5 columns replace the original 8 and can be fed directly into clustering, regression, or classification models.

IV. Advantages & Limitations

Why do we use PCA?

To visualize data: We can't see in 10 dimensions. If you have 10 features, PCA can reduce them to 2 or 3 so you can graph them on a scatter plot and look for clusters.
To speed up Machine Learning: Training a model on 1,000 features takes a long time and can lead to overfitting. PCA can compress those down to the 50 most important features, making models faster and sometimes more accurate.

Advantages:

✅ Efficient for large, high-dimensional datasets
✅ Removes noise and multicollinearity between features
✅ Improves downstream model performance and training speed

Limitations:

❌ Linearity — assumes linear relationships between features; won't capture non-linear structure (use Kernel PCA or t-SNE instead).
❌ Interpretability — PC columns are blended combinations; the original feature names are lost.
❌ Scale Sensitive — standardization is mandatory, not optional.
❌ Outliers — variance-driven, so outliers can distort principal component directions.

V. Extension: Using PCA for Feature Selection

Feature Extraction vs. Feature Selection

Everything above is feature extraction — PCA creates brand-new PC columns. But PCA can also guide feature selection — keeping a subset of the original named features.

	Feature Extraction (default)	Feature Selection (this section)
Output	New PC columns	Original columns (subset)
Interpretability	Lost	Preserved
How PCA helps	Directly projects data	Uses loadings to rank/prune originals

When to use it: When you must keep original, interpretable features (e.g., a clinician needs "blood pressure", not "PC3"), but still want to prune redundant ones.

The Idea: Read the Loadings

Loadings (eigenvector entries) tell you how much each original feature contributes to each PC. Two strategies:

Drop redundant features: If two features have nearly identical loadings across the top PCs, they carry the same information — keep one, drop the other.
Rank by importance: Score each feature by the magnitude of its loadings on the high-variance PCs; keep the top-ranked originals.

Worked Example — 4 Features → Keep 2 Originals

	Sample 1	Sample 2	Sample 3	Sample 4	Sample 5	Sample 6
F1	10	11	8	3	2	1
F2	6	4	5	3	2.8	1
F3	12	9	10	2.5	1.3	2
F4	5	7	6	2	4	7

Step 1 — Correlation matrix (after standardizing):

R = [\begin{matrix} 1.00 & 0.84 & 0.94 & 0.36 \\ 0.84 & 1.00 & 0.90 & - 0.03 \\ 0.94 & 0.90 & 1.00 & 0.37 \\ 0.36 & - 0.03 & 0.37 & 1.00 \end{matrix}]

$F_{1}, F_{2}, F_{3}$ are all strongly correlated ( $0.84$ – $0.94$ ) — they rise and fall together. $F_{4}$ is nearly uncorrelated (even $- 0.03$ vs. $F_{2}$ ) — it carries independent information.

Step 2 — Eigenvalues:

PC	$λ$	Variance	Cumulative
$P C_{1}$	2.879	71.96%	71.96%
$P C_{2}$	1.018	25.46%	97.42%
$P C_{3}$	0.085	2.14%	99.56%
$P C_{4}$	0.018	0.44%	100%

Only 2 real directions exist — the 4 features live in a 2-D world.

Step 3 — Loadings:

Feature	$P C_{1}$	$P C_{2}$
$F_{1}$	0.572	−0.014
$F_{2}$	0.533	0.396
$F_{3}$	0.584	−0.005
$F_{4}$	0.218	−0.918

PC1 is driven equally by $F_{1}, F_{2}, F_{3}$ — they are redundant; keep only the strongest ( $F_{3}$ ).
PC2 is dominated by $F_{4}$ — it provides unique, independent information.

Step 4 — Select features to keep:

PC	Driving features	Keep	Drop
$P C_{1}$ (72%)	$F_{1}, F_{2}, F_{3}$ (redundant)	$F_{3}$ — highest loading 0.584	$F_{1}$ , $F_{2}$
$P C_{2}$ (25%)	$F_{4}$ (unique)	$F_{4}$ — dominant loading 0.918	—

Result: 4 features → 2 original features, retaining ~97% of information. The surviving columns are real, named, interpretable — not blended PCs.

Rule of thumb

Use feature extraction (project onto PCs) when model accuracy is the only goal.
Use feature selection via loadings when you need original, interpretable features but want to prune redundant/low-variance ones.

Related method: Loading-based elimination

Iteratively remove the original feature with the largest loading on the smallest-eigenvalue component (the least-variance, most-redundant direction) until the desired feature count is reached.