Feature Selection

The Problem with More Features

A hospital informatics team spent months building a readmission prediction model. They had access to 200 fields per patient: lab values, diagnostic codes, vital sign histories, medication records, scheduling metadata. They fed all 200 into a gradient-boosted classifier and achieved reasonable training accuracy. Then they tried a second version, trained on just 40 features selected by a domain expert. The 40-feature model was more accurate on held-out patients, faster to run, and far easier for clinical staff to audit.

The result confused the team at first. More data is almost always better — that much is well established. But more features is not the same thing as more data. The number of patients was the same in both experiments. What changed was the dimensionality of the input space.

Irrelevant features introduce noise. A feature with no statistical relationship to readmission risk does not contribute zero to the model — it contributes random patterns. On any finite training set, an irrelevant feature will appear to correlate with the outcome by chance in some minor way. The model fits that spurious correlation. When the model encounters new patients, the spurious pattern vanishes and accuracy drops. This is one pathway to overfitting that does not require high model complexity: it only requires noisy inputs.

Redundant features create a different problem. Two features that track the same underlying variable both appear informative when scored individually. But once the model has learned from one, the other adds relatively little new signal. Including it anyway bloats the model, destabilises coefficient estimates, and can cause the learning algorithm to split its attention between two descriptions of the same phenomenon.

Irrelevance and Redundancy — Two Distinct Problems

The distinction between irrelevance and redundancy matters because the solutions differ. Irrelevance is about the absence of signal. Redundancy is about the duplication of signal that already exists elsewhere in the feature set.

Irrelevant features have no statistical relationship to the target. In the hospital dataset, the sum of the digits in a patient’s administrative ID number is an example: it derives from an arbitrary numbering system with no connection to clinical outcomes. Any pattern a model discovers involving this feature in the training data is an artefact of the specific ID assignments in that dataset. Those patterns will not generalise to patients registered in a different year or transferred from another hospital system.

Redundant features do carry genuine signal. In the hospital dataset, the number of prior admissions and the number of medications on the discharge prescription both predict readmission — patients with complex chronic diseases accumulate both prior episodes and extensive medication regimens. Both features earn high correlation scores with the outcome. But they also correlate strongly with each other. Once a model has learned the pattern from prior admissions, medications adds less new information than its individual score suggests. In a linear model trained on both, the coefficient estimates can become unstable: small changes in the training sample shift credit back and forth between the two correlated predictors, even though the model’s overall prediction does not change much. The practical effect is an unreliable model that is hard to audit.

Filter Methods — Score First, Model Later

Filter methods rank features by their statistical relationship to the target. No model is trained. The scores are computed directly from the data, then features below a threshold are removed before any learning begins.

Pearson correlation measures the linear relationship between a continuous feature and the target. For a feature $X$ and target $Y$ across $n$ samples:

The result lies in $[-1,1]$. For feature selection, the absolute value is used: features are ranked by $|r|$. A score near 1 means the feature moves in lockstep with the target. A score near 0 means they are linearly unrelated.

Pearson correlationpearson-correlation is fast and easy to interpret, but it only detects linear relationships. A feature with a strong U-shaped relationship to the outcome will receive a low Pearson score and risk being discarded.

Mutual information removes the linearity constraint. It measures any statistical dependency between the feature and the target, regardless of its shape.

The intuition: if knowing $X$ tells you nothing about $Y$, the joint probability $p(x,y)$ equals the product of the marginals $p(x)\cdot p(y)$ everywhere, making every log term zero. Any statistical dependency — linear or not — produces a positive score. Mutual informationmutual-information can detect non-linear patterns that Pearson misses, but estimating the joint distribution requires more data and is noisier on small samples.

import numpy as np
from sklearn.feature_selection import mutual_info_classif

scores = mutual_info_classif(X, y, random_state=42)
ranked_features = np.argsort(scores)[::-1]

Wrapper Methods — Let the Model Decide

Wrapper methods use a model’s validation performance as the selection criterion. Rather than scoring features statistically, they ask: does adding this feature to the current selected set improve the model on held-out data?

Forward selection starts with an empty set and adds features greedily. At each step, every remaining candidate is tested by training the model with the current selected set plus that one feature. The candidate that produces the best validation score is added permanently. The process continues until adding any remaining feature fails to improve performance.

Backward elimination reverses the process: start with the full feature set and remove the feature whose loss causes the smallest drop in performance, repeating until further removal hurts.

Both approaches require roughly $O(p^2)$ model training runs for $p$ features — expensive for large feature sets, but powerful because the model itself evaluates whether each feature adds genuine signal given what is already included. The results are also model-specific: switching from logistic regression to a random forest may produce an entirely different selected set, because the two models extract different structure from the same features.

Embedded Methods — Selection During Training

Embedded methods weave feature selection into the training process itself. The model learns which features are useful and which are not simultaneously with learning the prediction task.

Lasso regularisation adds an L1 penalty to the training loss:

The term $\lambda \Vert \mathbf{w}{\Vert }_1$ is proportional to the sum of the absolute values of the weights. The key property of the L1 norm is geometric: its feasible region (the set of weight vectors the penalty allows) has corners on the coordinate axes. When gradient descent minimises the sum of the prediction loss and this penalty, the solution tends to land on a corner — meaning some weights are driven to exactly zero, not merely small. Features with zeroed weights are eliminated entirely. Increasing $\lambda$ tightens the penalty, zeroing out more weights and producing a sparser model. Lassolasso is computationally efficient: one training run selects features and fits the model simultaneously.

Decision tree feature importance takes a different approach. Trees split on whichever feature reduces impurity (Gini or entropy) most at each node. After training, each feature’s importance is the total weighted reduction in impurity it contributes across all its splits in all trees. Features that drive large, clean splits early in many trees score highest. Features that are never selected, or that only appear in noisy shallow splits on small data subsets, score near zero. Ensemble methods like random forests and gradient boosting expose this score directly.

Choosing a Method

No single method is always best. The practical choice depends on the number of features, the available compute, and how much model-specific guidance is needed.

When a dataset has more than 100 candidate features and the relationship between features and target is unknown, filter methods are the right starting point: they are cheap, model-agnostic, and eliminate the most obviously uninformative features quickly. This cuts the search space before any expensive model training begins.

When the feature count is manageable (roughly 10–50) and the model type is already decided, embedded methods like Lasso or tree importance give a principled answer in a single training run. They account for some redundancy: Lasso penalises having two highly correlated features with non-zero weights; tree importance assigns low scores to features made redundant by another that splits earlier.

Wrapper methods are reserved for smaller, performance-critical problems where the cost of many training runs is acceptable and detecting feature interactions is important.

In production, the most common pattern chains the approaches: a filter pass removes obviously irrelevant features, then an embedded or wrapper method handles redundancy among the survivors.