Learn & Understand

L1 vs L2: Why Manhattan Geometry Produces Sparse Models

The companion calculator computes Manhattan distance, the sum of absolute differences, named for a taxi navigating a city grid. That same absolute-value geometry underlies one of the most useful tools in machine learning: L1 regularization, or Lasso, which shrinks a model's coefficients and, remarkably, drives many of them exactly to zero, performing automatic feature selection. Its counterpart, L2 regularization based on Euclidean geometry, only shrinks coefficients toward zero without eliminating them. Understanding why the Manhattan versus Euclidean distinction produces this difference is a beautiful piece of geometric intuition.

Regularization: Penalizing Complexity

Regularization combats overfitting by adding a penalty on the size of a model's coefficients to its training objective, discouraging the model from relying too heavily on any feature and thereby keeping it simpler and more generalizable. The two most common penalties measure coefficient size differently: L1 uses the sum of absolute values, exactly Manhattan distance from zero, while L2 uses the sum of squares, related to Euclidean distance. This is not a superficial resemblance, the geometry of each distance is precisely what gives the two methods their distinctive behavior.

The Key Difference: Sparsity

The practical consequence is that L1 produces sparse models, many coefficients become exactly zero, which effectively removes those features from the model, a built-in feature selection. L2 keeps all features but with small coefficients. When you want a simpler, more interpretable model that uses only the most important features, L1's sparsity is invaluable; when you want to retain all features while damping them, L2 is the choice.

The Geometric Intuition

Why does the absolute-value penalty zero out coefficients while the squared penalty does not? The answer is the shape of the constraint region each penalty defines. The L1 penalty, being based on absolute values, defines a region shaped like a diamond, with sharp corners lying exactly on the axes, where one coordinate is zero. The L2 penalty defines a smooth, round region, a circle, with no corners. When the optimization seeks the best solution within the penalty budget, it tends to touch the constraint region at a point, and the L1 diamond's corners, which sit on the axes, are precisely the points where some coefficients are zero. The round L2 region has no such corners, so its optimal points rarely land exactly on an axis. The pointy geometry of Manhattan distance is literally why L1 creates sparsity, and the smooth geometry of Euclidean distance is why L2 does not.

Choosing Between Them

The choice reflects what you want from the model. L1 (Lasso) is attractive when you suspect many features are irrelevant and want the model to identify and discard them, yielding a sparse, interpretable result. L2 (Ridge) is preferred when you believe most features contribute a little and you want to shrink them all without discarding any, which also handles correlated features more gracefully. There is even a combined approach that blends both penalties to get some of each. The Manhattan-versus-Euclidean distinction, so simple in the distance calculator, thus reaches deep into how models are built.

From Distance to Sparse Models

Use the calculator to compute Manhattan distance, and recognize its deeper role: as the L1 penalty, its absolute-value, diamond-shaped geometry is what makes Lasso regularization drive coefficients to exactly zero and perform feature selection, while the round geometry of Euclidean-based L2 only shrinks them. The calculation sums absolute differences; understanding that geometry is what explains why L1 produces sparse models and L2 does not.