Nonparametric Variable Selection with Optimal Decision Stumps
Apr 14, 2021
Speakers
Jason Klusowski
Speaker · 0 followers
Peter M Tian
Speaker · 0 followers
About
Decision trees and their ensembles are endowed with a rich set of diagnostic tools for ranking and extracting relevant input variables in a predictive model. One of the most commonly used in practice is the Mean Decrease in Impurity (MDI), which calculates an importance score for a variable by summing the weighted impurity reductions over all non-terminal nodes split with that variable. Despite the widespread use of tree based variable importance measures such as MDI, pinning down their theoretical properties has been challenging and therefore largely unexplored. To address this gap between theory and practice, we derive rigorous finite sample performance guarantees for variable selection in additive models with MDI for a single-level decision tree (decision stump). We find that the marginal signal strength of each variable can be considerably weaker than those for state-of-the-art nonparametric variable selection methods. We also show that SDI is asymptotically equivalent to Sure Independence Screening when the model is linear with Gaussian variates. Furthermore, unlike previous methods that attempt to directly estimate each marginal regression function via a truncated basis expansion, the fitted model used here is a simple, parsimonious decision stump—eliminating the need for bandwidth calibration. Thus, surprisingly, even though single-level decision trees are highly inaccurate for prediction, they can still be used to perform consistent model selection.Decision trees and their ensembles are endowed with a rich set of diagnostic tools for ranking and extracting relevant input variables in a predictive model. One of the most commonly used in practice is the Mean Decrease in Impurity (MDI), which calculates an importance score for a variable by summing the weighted impurity reductions over all non-terminal nodes split with that variable. Despite the widespread use of tree based variable importance measures such as MDI, pinning down their theoreti…
Organizer
AISTATS 2021
Account · 63 followers
Categories
Mathematics
Category · 2.4k presentations
AI & Data Science
Category · 10.8k presentations
About AISTATS 2021
The 24th International Conference on Artificial Intelligence and Statistics was held virtually from Tuesday, 13 April 2021 to Thursday, 15 April 2021.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Latent Derivative Bayesian Last Layer Networks
Watch later
Joe Watson, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Federated f-Differential Privacy
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Beyond Perturbation Stability: LP Recovery Guarantees for MAP Inference on Noisy Stable Instances
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Minimax optimality of Laplacian smoothing
Watch later
Alden Green, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Principal Subspace Estimation Under Information Diffusion
Watch later
Fan Zhou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bandit algorithms: Letting go of logarithmic regret for statistical robustness
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%