Running cross-validation or the bootstrap on a final model after you’ve eliminated a bunch of variables is missing the point, and will give materially misleading statistics (biased towards things being more “significant” than there really is evidence for). That’s because one of the highest risk parts of the modelling process is that variable selection. In particular, if the strategy involves variable selection (as two of my candidate strategies do), you have to automate that selection process and run it on each different resample. It’s critical that the re-sampling in the process envelopes the entire model-building strategy, not just the final fit. The bootstrap methods can give over-optimistic estimates of model validity compared to cross-validation there are various other methods available to address this issue although none seem to me to provide all-purpose solution. One of the problems with k-fold cross-validation is that it has a high variance ie doing it different times you get different results based on the luck of you k-way split so repeated k-fold cross-validation addresses this by performing the whole process a number of times and taking the average.Īs the sample sizes get bigger relative to the number of variables in the model the methods should converge.
The average of the 10 goodness of fit statistics becomes your estimate of the actual goodness of fit. 10-fold cross-validation involves dividing your data into ten parts, then taking turns to fit the model on 90% of the data and using that model to predict the remaining 10%. There’s a nice step by step explanation by thestatsgeek which I won’t try to improve on. This is a little more involved and is basically a method of estimating the ‘optimism’ of the goodness of fit statistic.
Bootstrap caret full#
Note – Following Efron, Harrell calls this the “simple bootstrap”, but other authors and the useful caret package use “simple bootstrap” to mean the resample model is used to predict the out-of-bag values at each resample point, rather than the full original sample. This involves creating resamples with replacement from the original data, of the same size applying the modelling strategy to the resample using the model to predict the values of the full set of original data and calculating a goodness of fit statistic (eg either R-squared or root mean squared error) comparing the predicted value to the actual value. The three validation methods I’ve used for this post are: There are many methods of validating models, although I think k-fold cross-validation has market dominance (not with Harrell though, who prefers varieties of the bootstrap). Confidence in hypothetical predictions gives us confidence in the insights the model gives into relationships between variables. As there is no possibility of new areas in New Zealand from 2013 that need to have their income predicted, the “prediction” is a thought-exercise which we need to find a plausible way of simulating. The main purpose of the exercise was actually to ensure I had my head around different ways of estimating the validity of a model, loosely definable as how well it would perform at predicting new data. None of these is exactly what I would use for real, but they serve the purpose of setting up a competition of strategies that I can test with a variety of model validation techniques.
The features are all continuous and are variables like “mean number of bedrooms”, “proportion of individuals with no religion” and “proportion of individuals who are smokers”. Using data with 54 variables on 1,785 area units from New Zealand’s 2013 census, I’m looking to predict median income on the basis of the other 53 variables. I wanted to evaluate three simple modelling strategies in dealing with data with many variables. I’ve been re-reading Frank Harrell’s Regression Modelling Strategies, a must read for anyone who ever fits a regression model, although be prepared – depending on your background, you might get 30 pages in and suddenly become convinced you’ve been doing nearly everything wrong before, which can be disturbing.
0 kommentar(er)
