R-Squared (The "Accuracy Score" of your Model)

🎯 Part 1: The Setup - What Are We Trying to Measure?

Imagine you are trying to predict something, like how well you'll do on a test based on how many hours you studied. R-Squared is the grade we give to our prediction "rule" to see how much of the story it actually tells.

• $X$ (Independent Variable): Hours studied
• $Y$ (Dependent Variable): Exam score

You collect data from 50 students and fit a regression line:

Where:

• $m$ = slope = $r\cdot \frac{{\sigma }_y}{{\sigma }_x}$
• $b$ = intercept = $\overline{y}-m\overline{x}$
• $\hat{y}$ = predicted score

🧩 Part 2: The Intuition - Two Competing Model

Think of it as a competition between two models:

★ Model 1: "The Lazy Guesser" (The Mean)

• Predicting everything using just the average (${\mu }_y$). This is our baseline.
• Any "mistake" here is called Total Variation (SST) .

★ Model 2: "The Smart Predictor" (The Regression Line)**

• Predicting using our $(\hat{y}=mx+b)$ formula.
• Any "mistake" left over here is called Unexplained Variation (SSE).

🧱 Part 3: The Building Blocks

★ Block 1: Understand mean score

Imagine ignoring study hours completely. The best guess for any student’s score would be:

This is called as the mean score ${\mu }_y$

★ Block 2: The "Baseline" Model ➛ SST (Total Sum of Squares)

This measures total variability if you ignored $X$ completely and just guessed ${\mu }_y$ for everyone.

Concept: Imagine you had no model at all; you would simply guess the average value ${\mu }_y$ for every prediction. SST represents the total error of those "average" guesses.

What it represents:

• The total "mystery" or uncertainty in your data
• How spread out the scores are from the mean
• The maximum possible error if you had no model

★ Block 3: The "Smart" Model ➛ SSR (Regression Sum of Squares)

This measures how much variability your regression line successfully explained by our model.

Concept: SSR represents how much "better" your regression line is at predicting the data compared to just guessing the average.

• The amount of "total mistakes" that our smart line successfully explained. i.e quantifies how much the data points ($y_i$), vary around the estimated regression line( $\widehat{y_i}$ ).

What it represents:

• The vertical distance between actual points and your predictions
• The "noise" or factors you missed
• The unexplained variance

★ Block 4: The "Leftover Mistakes" ➛ SSE (Sum of Squared Errors)

This measures the variability that your model failed to explain (the residuals).

Concept: These are the "residuals." It represents the noise or factors that your features failed to capture.

• The "mistakes" that are still there even after using our smart line. i.e quantifies how far the estimated sloped regression line, $\widehat{y_i}$, is from the horizontal "no relationship line," the sample mean or ${\mu }_y$.

What it represents:

• The vertical distance between actual points and your predictions
• The "noise" or factors you missed
• The unexplained variance

★ The Golden Equation

These three pieces always follow this relationship:

In words: $$\boxed{\Large \text{Total Variation} = \text{Explained Variation} + \text{Unexplained Variation}}$$

🧮 Part 4: The $R^2$ Formula

Now we can define R-squared in two equivalent ways:

★ Method 1: The "Success Ratio"

Interpretation: "What fraction of the total mystery did we solve?"

★ Method 2: The "Mistake Reduction Ratio"

Interpretation: "If we start with 100% mystery, how much is left over?"

📊📉 Part 5: Visual Summary Table

🔢 Part 6: Worked Example

★ Given Data:

• Total Variation: $SST=200$
• After fitting your model, the remaining error: $SSE=50$

★ Calculation:

🎓 Part 7: How to Interpret $R^2$ Values

The value of $R^2$ is always between 0 and 1

• If $\mathbf{SSR}$ is small, line is a good fit.
• If points lie perfectly on a line: $R^2=1$ ➛ The model explains all of the variability in the dependent variable.
• If points are completely random:$R^2\approx 0$ ➛ The model explains none of the variability in the dependent variable.  
  $R^2$ measures how tightly data hugs the regression line.

When is Low R² Acceptable?

• Social sciences: Human behavior is complex; $R^2=0.30$ can be useful
• Stock markets: Inherently random; even $R^2=0.10$ provides value
• Medical research: Many factors affect outcomes; $R^2=0.40$ can save lives

🔗 Part 8: The Connection to Correlation ($r$)

For simple linear regression (one $X$, one $Y$):

Where $r$ is the Pearson correlation coefficient between $X$ and $Y$.

To recover $r$ from $R^2$:

The sign rule:

• If slope ($m$) is positive → $r$ is positive
• If slope ($m$) is negative → $r$ is negative

⚠️ Part 9: Important Limitations

🎯 Part 10: The Problem with $R^2$ ➛ Enter "$\text{Adjusted}{R}^2$"

⚠️ The Flaw in $R^2$

Problem: $R^2$ will always increase (or stay the same) when you add more features, even if those features are pure random noise.

Why? The mathematical definition of $R^2$ is designed to reward any reduction in $SSE$, no matter how tiny.

Example of the Problem
You have a model predicting house prices with:

• Feature 1: Square footage ($R^2=0.70$)
• Feature 2: Number of bedrooms ($R^2=0.75$) ✅ Improvement!
• Feature 3: Homeowner's favorite color ($R^2=0.751$) 🤔 Wait...

Adding "favorite color" increased $R^2$ by 0.001, but it's obviously meaningless!

✅ The Solution: Adjusted $R^2$

Adjusted $R^2$ penalizes you for adding features that don't pull their weight.

• How it works: It only increases if the new feature improves the model's predictive power significantly more than what would be expected by random chance.
• The "Drop" Mechanism: If you add a useless feature, the $R^2$ might go up by $0.0001$, but the penalty for adding a new variable will be larger than that tiny gain. The result? The Adjusted $R^2$ will actually go down.

🤔 Why Adjusted $R^2$ is Better for Your Model?

➛ in case of Multiple linear Regression

A. It Fights Overfitting

Overfitting happens when your model learns the "noise" in your data rather than the actual "signal." By using Adjusted $R^2$, you are visually alerted when your model is getting too complex without adding real value.

B. It Guides Feature Selection

When you are deciding which features to keep:

1. Add a feature.
2. The Penalty: It adds a "penalty" for every new feature you add.
3. If Adjusted $R^2$ increases, the feature is adding value.
4. If Adjusted $R^2$ decreases, the feature is likely noise and should be removed.

C. It Accounts for Sample Size ($n$)

The formula for Adjusted $R^2$ includes the number of data points ($n$). This is crucial because it’s much easier to "fake" a high $R^2$ with a small dataset (e.g., 5 points and 4 features) than with a large one. Adjusted $R^2$ corrects for this bias.

📝 The Formula:

Where:

• $n$ = number of data points
• $k$ = number of features (predictors) in your model

📝 Alternative Form:

📈📉 The "Gap" Test

As a rule of thumb in my lab, I always look at the gap between the two:

• Small Gap: Your features are high-quality and relevant.
• Large Gap: You have "junk" features that are inflating your $R^2$ without providing real predictive power.

★ Visual Comparison:

📚 Part 11: Quick Summary