SGT Regressor
Stochastic Gradient Tree for regression.
Incremental decision tree regressor that minimizes the mean square error to guide its growth.
Stochastic Gradient Trees (SGT) directly minimize a loss function to guide tree growth and update their predictions. Thus, they differ from other incrementally tree learners that do not directly optimize the loss, but a data impurity-related heuristic.
Parameters
-
delta(
float, Default:1e-07) → Define the significance level of the F-tests performed to decide upon creating splits or updating predictions. -
grace_period(
int, Default:200) → Interval between split attempts or prediction updates. -
lambda_(
float, Default:0.1) → Positive float value used to impose a penalty over the tree's predictions and force them to become smaller. The greater the lambda value, the more constrained are the predictions. -
gamma(
float, Default:1.0) → Positive float value used to impose a penalty over the tree's splits and force them to be avoided when possible. The greater the gamma value, the smaller the chance of a split occurring.
Example Usage
We can create an instance of the SGT Regressor model like this.
import turboml as tb
sgt_model = tb.SGTRegressor()This implementation enhances the original proposal1 by using an incremental strategy to discretize numerical features dynamically, rather than relying on a calibration set and parameterized number of bins. The strategy used is an adaptation of the Quantization Observer (QO)2. Different bin size setting policies are available for selection. They directly related to number of split candidates the tree is going to explore, and thus, how accurate its split decisions are going to be. Besides, the number of stored bins per feature is directly related to the tree's memory usage and runtime.
Footnotes
-
Gouk, H., Pfahringer, B., & Frank, E. (2019, October). Stochastic Gradient Trees. In Asian Conference on Machine Learning (pp. 1094-1109). ↩
-
Mastelini, S.M. and de Leon Ferreira, A.C.P., 2021. Using dynamical quantization to perform split attempts in online tree regressors. Pattern Recognition Letters. ↩