Statistical/Probabilistic Distances

1. Pearson Correlation Coefficient

The Pearson Correlation measures the linear relationship between two variables. It's similar to cosine similarity but works with centered data (mean-subtracted), making it useful for finding linear patterns.

Formula

For two vectors $X = (x_{1}, x_{2}, . . ., x_{n})$ and $Y = (y_{1}, y_{2}, . . ., y_{n})$ :

r = \frac{\sum_{i = 1}^{n} (x_{i} - \bar{x}) (y_{i} - \bar{y})}{\sqrt{\sum_{i = 1}^{n} (x_{i} - \bar{x})^{2}} \sqrt{\sum_{i = 1}^{n} (y_{i} - \bar{y})^{2}}}

Where:

$\bar{x}$ and $\bar{y}$ are the means of $X$ and $Y$
$r$ ranges from -1 to +1

Interpretation:

+1: Perfect positive linear relationship
0: No linear relationship
-1: Perfect negative linear relationship

Relationship to Cosine Similarity:

Pearson correlation = Cosine similarity of mean-centered vectors
Both measure directional alignment, but Pearson removes mean effects

When to Use:

Detecting linear correlations between features
Collaborative filtering with user ratings
Feature selection (identifying correlated features)
Time series analysis

Advantages:

✅ Captures linear relationships
✅ Invariant to linear transformations
✅ Standard statistical measure

Disadvantages:

❌ Only captures linear relationships
❌ Sensitive to outliers
❌ Requires continuous data

2. Mahalanobis Distance

Mahalanobis Distance (correlated features): Measures the distance between a point and a distribution, taking into account the covariance of the data, making it useful for anomaly detection, clustering, and classification.

Key Properties

Scale-invariant: It is unaffected by changes in the measurement scales of your variables.

Formula:

d (x, μ) = \sqrt{(x - μ)^{T} Σ^{- 1} (x - μ)}

Where:

𝑥: is the multivariate data point (vector).
𝜇: The centroid (mean) vector of the cluster.
$\sum$ : is the covariance matrix of the variables.
$\sum^{- 1}$ : is the inverse of the covariance matrix.
𝑇: denotes the transpose of the vector.

How Mahalanobis Distance Accounts for Correlation?

By incorporating the covariance matrix ( $Σ$ ), Mahalanobis distance rescales the feature space to account for correlations among features.
It effectively stretches or compresses the space along the axes of variability.
- Points that align with the natural shape of the cluster will have smaller Mahalanobis distances.
- Points that deviate from this shape will have larger Mahalanobis distances, even if their Euclidean distances are the same.

Why Mahalanobis Distance is Better for Clustering in Correlated Data

Handles Correlations:
- By taking the covariance matrix into account, Mahalanobis distance adjusts the distance based on the interdependencies between features.
Outlier Detection:
- Mahalanobis distance better identifies outliers, as it recognizes that deviations from the natural direction of the data distribution should result in larger distances.
Elliptical Clusters:
- Mahalanobis Distance performs well for elliptical clusters, where the data's variance differs along various principal axes.
Density-based Analysis:
- It detects whether a point lies inside or outside the natural shape of the data distribution, making distances more meaningful.