3.2.3.3.5. sklearn.ensemble.GradientBoostingClassifier¶
- class sklearn.ensemble.GradientBoostingClassifier(loss='deviance', learning_rate=0.1, n_estimators=100, subsample=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=3, init=None, random_state=None, max_features=None, verbose=0)¶
Gradient Boosting for classification.
GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage n_classes_ regression trees are fit on the negative gradient of the binomial or multinomial deviance loss function. Binary classification is a special case where only a single regression tree is induced.
Parameters : loss : {‘deviance’}, optional (default=’deviance’)
loss function to be optimized. ‘deviance’ refers to deviance (= logistic regression) for classification with probabilistic outputs.
learning_rate : float, optional (default=0.1)
learning rate shrinks the contribution of each tree by learning_rate. There is a trade-off between learning_rate and n_estimators.
n_estimators : int (default=100)
The number of boosting stages to perform. Gradient boosting is fairly robust to over-fitting so a large number usually results in better performance.
max_depth : integer, optional (default=3)
maximum depth of the individual regression estimators. The maximum depth limits the number of nodes in the tree. Tune this parameter for best performance; the best value depends on the interaction of the input variables.
min_samples_split : integer, optional (default=2)
The minimum number of samples required to split an internal node.
min_samples_leaf : integer, optional (default=1)
The minimum number of samples required to be at a leaf node.
subsample : float, optional (default=1.0)
The fraction of samples to be used for fitting the individual base learners. If smaller than 1.0 this results in Stochastic Gradient Boosting. subsample interacts with the parameter n_estimators. Choosing subsample < 1.0 leads to a reduction of variance and an increase in bias.
max_features : int, float, string or None, optional (default=”auto”)
- The number of features to consider when looking for the best split:
- If int, then consider max_features features at each split.
- If float, then max_features is a percentage and int(max_features * n_features) features are considered at each split.
- If “auto”, then max_features=sqrt(n_features).
- If “sqrt”, then max_features=sqrt(n_features).
- If “log2”, then max_features=log2(n_features).
- If None, then max_features=n_features.
Choosing max_features < n_features leads to a reduction of variance and an increase in bias.
init : BaseEstimator, None, optional (default=None)
An estimator object that is used to compute the initial predictions. init has to provide fit and predict. If None it uses loss.init_estimator.
verbose : int, default: 0
Enable verbose output. If 1 then it prints progress and performance once in a while (the more trees the lower the frequency). If greater than 1 then it prints progress and performance for every tree.
References
J. Friedman, Greedy Function Approximation: A Gradient Boosting Machine, The Annals of Statistics, Vol. 29, No. 5, 2001.
- Friedman, Stochastic Gradient Boosting, 1999
T. Hastie, R. Tibshirani and J. Friedman. Elements of Statistical Learning Ed. 2, Springer, 2009.
Attributes
feature_importances_ array, shape = [n_features] The feature importances (the higher, the more important the feature). oob_improvement_ array, shape = [n_estimators] The improvement in loss (= deviance) on the out-of-bag samples relative to the previous iteration. oob_improvement_[0] is the improvement in loss of the first stage over the init estimator. oob_score_ array, shape = [n_estimators] Score of the training dataset obtained using an out-of-bag estimate. The i-th score oob_score_[i] is the deviance (= loss) of the model at iteration i on the out-of-bag sample. Deprecated: use oob_improvement_ instead. train_score_ array, shape = [n_estimators] The i-th score train_score_[i] is the deviance (= loss) of the model at iteration i on the in-bag sample. If subsample == 1 this is the deviance on the training data. loss_ LossFunction The concrete LossFunction object. init BaseEstimator The estimator that provides the initial predictions. Set via the init argument or loss.init_estimator. estimators_: list of DecisionTreeRegressor The collection of fitted sub-estimators. Methods
decision_function(X) Compute the decision function of X. fit(X, y) Fit the gradient boosting model. get_params([deep]) Get parameters for this estimator. predict(X) Predict class for X. predict_proba(X) Predict class probabilities for X. score(X, y) Returns the mean accuracy on the given test data and labels. set_params(**params) Set the parameters of this estimator. staged_decision_function(X) Compute decision function of X for each iteration. staged_predict(X) Predict class probabilities at each stage for X. staged_predict_proba(X) Predict class probabilities at each stage for X. - __init__(loss='deviance', learning_rate=0.1, n_estimators=100, subsample=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=3, init=None, random_state=None, max_features=None, verbose=0)¶
- decision_function(X)¶
Compute the decision function of X.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : score : array, shape = [n_samples, k]
The decision function of the input samples. Classes are ordered by arithmetical order. Regression and binary classification are special cases with k == 1, otherwise k==n_classes.
- feature_importances_¶
- Return the feature importances (the higher, the more important the
- feature).
Returns : feature_importances_ : array, shape = [n_features]
- fit(X, y)¶
Fit the gradient boosting model.
Parameters : X : array-like, shape = [n_samples, n_features]
Training vectors, where n_samples is the number of samples and n_features is the number of features.
y : array-like, shape = [n_samples]
Target values (integers in classification, real numbers in regression) For classification, labels must correspond to classes 0, 1, ..., n_classes_-1
Returns : self : object
Returns self.
- get_params(deep=True)¶
Get parameters for this estimator.
Parameters : deep: boolean, optional :
If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns : params : mapping of string to any
Parameter names mapped to their values.
- predict(X)¶
Predict class for X.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : y : array of shape = [n_samples]
The predicted classes.
- predict_proba(X)¶
Predict class probabilities for X.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : p : array of shape = [n_samples]
The class probabilities of the input samples. Classes are ordered by arithmetical order.
- score(X, y)¶
Returns the mean accuracy on the given test data and labels.
Parameters : X : array-like, shape = (n_samples, n_features)
Test samples.
y : array-like, shape = (n_samples,)
True labels for X.
Returns : score : float
Mean accuracy of self.predict(X) wrt. y.
- set_params(**params)¶
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as pipelines). The former have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.
Returns : self :
- staged_decision_function(X)¶
Compute decision function of X for each iteration.
This method allows monitoring (i.e. determine error on testing set) after each stage.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : score : generator of array, shape = [n_samples, k]
The decision function of the input samples. Classes are ordered by arithmetical order. Regression and binary classification are special cases with k == 1, otherwise k==n_classes.
- staged_predict(X)¶
Predict class probabilities at each stage for X.
This method allows monitoring (i.e. determine error on testing set) after each stage.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : y : array of shape = [n_samples]
The predicted value of the input samples.
- staged_predict_proba(X)¶
Predict class probabilities at each stage for X.
This method allows monitoring (i.e. determine error on testing set) after each stage.
Parameters : X : array-like of shape = [n_samples, n_features]
The input samples.
Returns : y : array of shape = [n_samples]
The predicted value of the input samples.