CatBoost (Categorical Boosting)

CatBoost is a state-of-the-art open-source machine learning algorithm based on gradient boosting over decision trees, designed specifically for categorical features, with clever techniques to prevent overfitting and target leakage.
CatBoost delivers high accuracy with little to no hyperparameter tuning, and offering blazing-fast prediction speeds

While Boosting - XGBoost optimized for speed and LightGBM for scale. CatBoost is built to solve a different problem: how to handle categorical features properly without data leakage.

The Categorical Feature Challenge

Real-world datasets are full of categorical features — user IDs, product categories, locations, brands, etc. Before a gradient-boosted tree can split on them, they must be turned into numbers, and every traditional way of doing so has a serious flaw:

One-Hot Encoding:

• One binary column per category.
• Issues
  • Dimension explosion: 100 unique categories → 100 columns (dimension explosion!)
  • Sparse data, memory-intensive
  • Loses information — every category is treated as equally distant

Label Encoding:

• Assigns an arbitrary integer to each category (Red ➛ 0, Green ➛ 1, Blue ➛ 2)
• Issues
  • Fake ordinal relationship: The model reads "2 > 1" as if categories were ordered, which is usually meaningless the tree may split as if Blue (2) > Green (1) > Red (0)

Target Encoding (see Target Encoding):

• Replace each category with the average target value for that category
• Issues:
  • Target is baked into the feature, so the model memorizes answers it won't have at inference. Using the target to build the feature lets the model "see" answers it won't have on new data (see the leakage problem)
  • Model learns patterns that won't exist in production → optimistic offline scores, poor generalization

Key Features

1. Native Categorical Support

Eliminates the need for extensive data preprocessing like one-hot encoding. It converts categories into numerical values using innovative, permutation-driven target statistics, preventing data leakage.

CatBoost introduces the concept of "artificial time."

• It randomly shuffles the training dataset.
• For any given row (let's call it Row 10), CatBoost calculates the target encoding for its category using only the rows that came before it (Rows 1 through 9).
• It does not look at Row 10's own target, nor does it look into the "future" at Rows 11 onwards.

By relying only on the "past," CatBoost completely eliminates target leakage while generating highly accurate encodings on the fly.

2. Ordered Boosting

One of the most innovative features of CatBoost is Ordered Boosting, which addresses the issue of prediction bias during training.

How Standard Gradient Boosting Works:

• During training, tree predictions are made based on the entire dataset.
• This can create bias because the model uses labels from the same dataset it is working to predict.

For Category "A":
- Calculate: Average of ALL samples with Category "A"
- Use this average for EVERY sample with "A"
Problem: Future data used to encode past data!

Ordered Boosting Process:

• A specialized technique that builds each tree iteration using only the data available prior to the current observation, which significantly reduces overfitting.

For each sample i with Category "A":
- Calculate: Average of samples 1 to i-1 with Category "A"
- Use this for sample i
- Next sample sees average including sample i
Result: Only past information used—no leakage!

Example: Encoding "City" feature

Benefits:

• Reduces overfitting.
• Eliminating bias in gradient calculation during model training.

3. Oblivious (Symmetric) Trees

Uses symmetric decision trees, meaning the same splitting criterion is used across all nodes at a specific depth.

Standard Trees:

• Each node can split on any feature
• Each path through tree is unique
• Complex, potentially overfit

CatBoost's Oblivious Trees:

• Same splitting criteria used for entire level
• If level 1 splits on "Age", every node at level 1 splits on "Age"
• Tree is perfectly symmetric (balanced)

Visualization:

Standard Tree:           Oblivious Tree:
     Age                      Age
    /   \                    /   \
  Sex   Income          Income Income
  / \    / \              / \     / \
 A  B   C  D             A  B    C  D

Benefits:

• Faster prediction: Can use bitwise operations (very efficient). This structure enables incredibly fast scoring and highly parallelized CPU/GPU training.
• Better regularization: Symmetry prevents overfitting
• Easier to tune: Fewer hyperparameters
• More interpretable: Simpler structure

Trade-off: Slightly less flexible than asymmetric trees, but the regularization benefit usually outweighs this.

4. Built-In Missing Value Handling:

Automatically processes missing values during both training and inference without requiring prior imputation.

5. The Shuffle

If you look at an entire dataset all at once, it is easy for a model to accidentally peek at the answers. To stop this, CatBoost takes your training data and randomly shuffles the rows into a sequence. The algorithm then pretends that this data is arriving sequentially over time, one row after another.

CatBoost goes even further: it uses different random permutations of data for:

• Computing residuals (what we got wrong)
• Training trees (learning from mistakes)

This creates additional protection against overfitting—the same data points don't repeatedly influence both residual calculation and tree building.

Key Components in CatBoost

1. Loss Functions:
   • Supports a variety of objective functions like log loss, MAE, RMSE, and cross-entropy, making it versatile across tasks (classification, regression, ranking).
2. Feature Combinations:
   • CatBoost creates combined feature interactions during training automatically.
3. Hyperparameter Optimization:
   • Includes parameters such as learning rate, number of iterations, tree depth, and regularization factors.
4. GPU and Distributed Training:
   • CatBoost’s support for GPU and distributed systems ensures scalability for massive datasets.

Training Process

• Good Example 👉 CatBoost Part 2: Building and Using Trees - (StatQuest with Josh Starmer)

How to Use CatBoost with Categorical Features

Super Simple:

# No preprocessing needed!
import catboost as cb

# Just specify which columns are categorical
cat_features = ['City', 'Category', 'Brand', 'UserID']

model = cb.CatBoostClassifier(
    cat_features=cat_features,  # That's it!
    iterations=1000,
    learning_rate=0.1
)

model.fit(X_train, y_train)

CatBoost Characteristics

Strengths:

• Best categorical handling: Directly encodes categorical features (e.g., one-hot encoding). No dimension explosion
• No preprocessing: Use raw categorical data directly—No manual encoding needed
• Handles Target Leakage: Prevents data leakage using ordered target encoding.
• Robust to Overfitting: Uses ordered boosting and regularization techniques to prevent overfitting.
• Fast Training and Inference: Optimized for both CPU and GPU systems.
• Fast prediction: Symmetric trees enable efficient inference
• Cross-Platform Operations: Works seamlessly for classification, regression, and ranking tasks.
• Robust defaults: Excellent performance without tuning
• Built-in evaluation: Comprehensive metrics and visualization

Weaknesses:

• High Computational Cost: Slightly slower than LightGBM and XGBoost on purely numerical datasets.
• Slower training: Ordered target statistics add overhead (especially with many categoricals)
• Limited Interpretability for Trees: Symmetric trees are harder to interpret.
• Dependency on Target Encoding: Relies heavily on accurate target encoding for categorical data.
• Requires Careful Hyperparameter Tuning: Needs tuning for optimal performance on large datasets.
• Memory with categoricals: High-cardinality features can use significant memory
• Large datasets: Not as fast as LightGBM for massive datasets

When to Choose CatBoost:

• Many categorical features: User IDs, product IDs, categories (this is its superpower)
• Need good defaults: Don't have time for extensive hyperparameter tuning
• Worried about leakage: Want built-in protection against target leakage
• Classification focus: Especially strong for classification problems
• Interpretability matters: Symmetric trees easier to understand
• Small to medium data: Best suited for datasets under 10M rows