Feature Scaling Part 1: When (and Why) to Scale Data in Machine Learning

Learn why feature scaling is a critical preprocessing step in machine learning. This guide explains which models are sensitive to data scale and provides a clear visual demonstration.
Fundamentals
Data Preprocessing
Scaling
Series: Feature Scaling
Author

January Tabaka

Published

July 30, 2025

The Unseen Bias in Data

When you first load a dataset for a real-world problem, you’ll immediately notice that the features come in all shapes and sizes.

  • For a financial fraud detection model, you might have:
    • a user’s transaction amount in (e.g., 5.00 – 20,000.00).
    • the number of transactions they’ve made in the last 24 hours (e.g., 1 – 50).
    • the time since their last login, measured in seconds (e.g., 300 – 86,400).
  • In E-commerce Analytics
    • a column might represent the number of items in a user’s shopping cart (e.g., 1-20),
    • another the total purchase value in cents (e.g., 500-50,000),
    • a third the user’s lifetime customer score (e.g., 0-1)
  • For SaaS User Engagement & Churn Prediction
    • a feature could represent a user’s number of logins this month (e.g., 5-100),
    • another the total minutes they spent on the platform (e.g., 60-10,000),
    • and a third their calculated churn probability (e.g., 0.05-0.95).
  • In an industrial Predictive Maintenance scenario
    • one feature could be the temperature of a machine in Celsius (e.g., 40-90) as one feature,
    • another its vibrations in Hz (e.g., 10-2000),
    • and yet another its total operational hours (e.g., 5,000-50,000)

In this scenario, the features exist on wildly different scales. You might be tempted to feed this data directly into a machine learning model, but this can be a critical mistake. Many of the most powerful and common algorithms are sensitive to the scale of their input features. Without a preprocessing step, the model might be disproportionately influenced by high-magnitude features, reducing model accuracy. That step is feature scaling.

Why Scaling is So Important: A Visual Analogy

To see why scaling matters, let’s look at some data.

We’ll use the well-known California Housing dataset, available directly through scikit-learn.

It’s a powerful way to see the two key steps of the standardization process.

Here’s what each stage of visualization shows:

  • 1. Raw Data (Left): This is the original, un-scaled data from the sample of the California Housing dataset. The plot is dramatically distorted into a thin line. Because the Population feature has a much larger range and variance than the HouseAge, the true relationship between the features is completely hidden.

  • 2. After Centering (Middle): This plot shows the data after subtracting the mean from each feature. The data has shifted to be centered around the origin (0,0), but the fundamental problem remains. The plot is still a distorted line, and the Population feature continues to dominate the x-axis.

  • 3. After Scaling (Right): This is the final transformation. After dividing by the standard deviation, the variance of both features becomes equal. The data is now a cloud of points. The distortion is gone, and the true underlying structure of the data is revealed and ready for a machine learning model.

The distribution of housing features. at three stages: raw, centered, and fully scaled. Notice how centering shifts the data, but only the final scaling step corrects the distortion.

Models that rely on calculating distances (like K-Nearest Neighbors) or use gradient descent for optimization (like Linear Regression and Neural Networks) are the most affected by the scale of their features.

If you’re curious how I generated the plots above, here is the Python code that loads the data and creates the visualizations.

It is inspired by the “Feature Scaling and Learning Rate (Multi-variable)” lab found in the Machine Learning Specialization course Supervised Machine Learning: Regression and Classification by the deeplearning.ai, taught by professor Andrew Ng.

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing

# Load the full dataset from scikit-learn
housing = fetch_california_housing()

# Convert the data to a pandas DataFrame for easier handling
df = pd.DataFrame(housing.data, columns=housing.feature_names)

# Select only the two features needed for visualization
X = df[['Population', 'HouseAge']]

# Create a smaller, random sample of 100 data points for a clean plot
X_train = X.sample(n=100, random_state=42)

# Calculate the mean and standard deviation for each column
mu = X_train.mean()
sigma = X_train.std()

# Center the data by subtracting the mean
X_centered = X_train - mu

# Scale the data by dividing by the standard deviation
X_scaled = (X_train - mu) / sigma

# Create the Three Plots
fig, ax = plt.subplots(1, 3, figsize=(8, 2.5))

# Plot 1: Raw Data
ax[0].scatter(X_train['Population'], X_train['HouseAge'])
ax[0].set_xlabel("Population")
ax[0].set_ylabel("House Age")
ax[0].set_title("1. Raw Data")
ax[0].axis('equal')
ax[0].grid(True)

# Plot 2: Centered Data
ax[1].scatter(X_centered['Population'], X_centered['HouseAge'])
ax[1].set_xlabel("Population")
ax[1].set_ylabel("House Age")
ax[1].set_title("2. After Centering (X - μ)")
ax[1].axis('equal')
ax[1].grid(True)

# Plot 3: Scaled Data
ax[2].scatter(X_scaled['Population'], X_scaled['HouseAge'])
ax[2].set_xlabel("Population")
ax[2].set_ylabel("House Age")
ax[2].set_title("3. After Scaling ((X - μ) / σ)")
ax[2].axis('equal')
ax[2].grid(True)

plt.tight_layout(rect=[0, 0.03, 1, 0.95])
fig.suptitle("The Step-by-Step Effect of Scaling")
plt.show()

Beyond the visual intuition, I think it’s important to understand the two key technical reasons why scaling is so critical for certain models.

Linear Regression: Optimization vs. Interpretability

Linear Regression benefits from scaling, but the “why” depends on how the model is trained.

  • With Gradient Descent, scaling is essential. Unscaled features create a skewed, elliptical loss surface, causing the optimizer to take a slow, zigzagging path toward the minimum. Scaling reshapes the surface into a more uniform shape, enabling faster and more stable convergence.

  • With the Normal Equation, (closed-form solution), scaling isn’t required for mathematical correctness—but it’s still highly recommended for interpretability. Without scaling, the magnitudes of the coefficients reflect the feature units and aren’t directly comparable. Standardizing features allows for meaningful comparison of coefficient sizes.

Regularization: Ensuring a Fair Penalty

Scaling becomes mandatory when using regularization (like L1/Lasso or L2/Ridge). These techniques penalize large coefficients to prevent overfitting. However, the penalty is applied uniformly, without considering the scale of the underlying features.

If features vary in scale, those with larger numeric ranges will be penalized more harshly, regardless of their actual importance. This can lead to incorrect shrinkage of informative features.

By scaling the data, you ensure that regularization treats all features equally, allowing the model to make a more accurate judgment about the true importance of each feature.

So, When Should You Scale Data?

This brings us to the crux of the matter. While scaling is critical for the majority of algorithms, it is not a universal requirement.

Some exceptions to the rule are tree-based models. Algorithms like Decision Trees, Random Forests, and Gradient Boosting models (like XGBoost) are generally insensitive to the scale of the features.

This is because these models work by making a series of splits on individual features. A decision tree asks questions like, “Is the income greater than $100,000?” or “Is the age less than 40?”. The answer to these questions is the same whether the income is measured in dollars or thousands of dollars. The scale of other features has no bearing on the split that is chosen for a particular feature.

So what’s the takeaway? While tree-based models give you a pass, for almost everything else, I strongly recommend making feature scaling a critical, non-negotiable step in your modeling pipeline.

What’s Next?

Now that we’ve established why scaling is so critical, I hope you’re ready to get hands-on with the how. In the next post we’ll explore three of the most widely used scaling techniques: Standardization, Normalization, and Robust Scaling, with code in both Python and R.