Feature selection
The classes in the sklearn.feature_selection
module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very highdimensional datasets.
Removing features with low variance
VarianceThreshold
is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zerovariance features, i.e. features that have the same value in all samples.
As an example, suppose that we have a dataset with boolean features, and we want to remove all features that are either one or zero (on or off) in more than 80% of the samples. Boolean features are Bernoulli random variables, and the variance of such variables is given by
so we can select using the threshold .8 * (1  .8)
:
>>> from sklearn.feature_selection import VarianceThreshold
>>> X = [ [0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1] ]
>>> sel = VarianceThreshold(threshold=(.8 * (1  .8)))
>>> sel.fit_transform(X)
array([ [0, 1],
[1, 0],
[0, 0],
[1, 1],
[1, 0],
[1, 1] ])
As expected, VarianceThreshold has removed the first column, which has a probability of containing a zero.
Univariate feature selection
Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikitlearn exposes feature selection routines as objects that implement the transform
method:

SelectKBest
removes all but the highest scoring features 
SelectPercentile
removes all but a userspecified highest scoring percentage of features 
using common univariate statistical tests for each feature: false positive rate
SelectFpr
, false discovery rateSelectFdr
, or family wise errorSelectFwe
. 
GenericUnivariateSelect
allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyperparameter search estimator.
For instance, we can perform a test to the samples to retrieve only the two best features as follows:
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectKBest
>>> from sklearn.feature_selection import chi2
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X.shape
(150, 4)
>>> X_new = SelectKBest(chi2, k=2).fit_transform(X, y)
>>> X_new.shape
(150, 2)
These objects take as input a scoring function that returns univariate scores and pvalues (or only scores for SelectKBest
and SelectPercentile
):

For regression:
f_regression, mutual_info_regression

For classification:
chi2, f_classif, mutual_info_classif
The methods based on Ftest estimate the degree of linear dependency between two random variables. On the other hand, mutual information methods can capture any kind of statistical dependency, but being nonparametric, they require more samples for accurate estimation.
Feature selection with sparse data
If you use sparse data (i.e. data represented as sparse matrices),
chi2, mutual_info_regression, mutual_info_classif
will deal with the data without making it dense.
Warning: Beware not to use a regression scoring function with a classification problem, you will get useless results.
Recursive feature elimination
Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE
) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_
attribute or through a feature_importances_ attribute
. Then, the least important features are pruned from current set of features.That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.
RFECV
performs RFE
in a crossvalidation loop to find the optimal number of features.
Feature selection using SelectFromModel
SelectFromMode
l is a metatransformer that can be used along with any estimator that has a coef_
or feature_importances_ attribute
after fitting. The features are considered unimportant and removed, if the corresponding coef_ or feature_importances_
values are below the provided threshold
parameter. Apart from specifying the threshold numerically, there are builtin heuristics for finding a threshold using a string argument. Available heuristics are “mean”, “median” and float multiples of these like “0.1*mean”.
For examples on how it is to be used refer to the sections below.
L1based feature selection
Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero. When the goal is to reduce the dimensionality of the data to use with another classifier, they can be used along with feature_selection.SelectFromModel
to select the nonzero coefficients. In particular, sparse estimators useful for this purpose are the linear_model.Lasso
for regression, and of linear_model.LogisticRegression
and svm.LinearSVC
for classification:
>>> from sklearn.svm import LinearSVC
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X.shape
(150, 4)
>>> lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
>>> model = SelectFromModel(lsvc, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape
(150, 3)
With SVMs and logisticregression, the parameter C controls the sparsity: the smaller C the fewer features selected. With Lasso, the higher the alpha parameter, the fewer features selected.
L1recovery and compressive sensing
For a good choice of alpha, the Lasso can fully recover the exact set of nonzero variables using only few observations, provided certain specific conditions are met. In particular, the number of samples should be “sufficiently large”, or L1 models will perform at random, where “sufficiently large” depends on the number of nonzero coefficients, the logarithm of the number of features, the amount of noise, the smallest absolute value of nonzero coefficients, and the structure of the design matrix X. In addition, the design matrix must display certain specific properties, such as not being too correlated.
There is no general rule to select an alpha parameter for recovery of nonzero coefficients. It can by set by crossvalidation (LassoCV or LassoLarsCV), though this may lead to underpenalized models: including a small number of nonrelevant variables is not detrimental to prediction score. BIC (LassoLarsIC) tends, on the opposite, to set high values of alpha.
Treebased feature selection
Treebased estimators (see the sklearn.tree
module and forest of trees in the sklearn.ensemble
module) can be used to compute feature importances, which in turn can be used to discard irrelevant features (when coupled with the sklearn.feature_selection.SelectFromModel
metatransformer):
>>> from sklearn.ensemble import ExtraTreesClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X.shape
(150, 4)
>>> clf = ExtraTreesClassifier(n_estimators=50)
>>> clf = clf.fit(X, y)
>>> clf.feature_importances_
array([ 0.04..., 0.05..., 0.4..., 0.4...])
>>> model = SelectFromModel(clf, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape
(150, 2)
Feature selection as part of a pipeline
Feature selection is usually used as a preprocessing step before doing the actual learning. The recommended way to do this in scikitlearn is to use a sklearn.pipeline.Pipeline:
clf = Pipeline([('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
('classification', RandomForestClassifier())])
clf.fit(X, y)
In this snippet we make use of a sklearn.svm.LinearSVC
coupled with sklearn.feature_selection.SelectFromModel
to evaluate feature importances and select the most relevant features. Then, a sklearn.ensemble.RandomForestClassifier
is trained on the transformed output, i.e. using only relevant features. You can perform similar operations with the other feature selection methods and also classifiers that provide a way to evaluate feature importances of course. See the sklearn.pipeline.Pipeline
examples for more details.