Objective function for voting model selection

<2016-06-14 Tue 11:43>

  • The objective function is to minimise the average cv error
    • This is because we want to minimise MSE (or some other statistic) on an unseen dataset
  • However, mean-CV error is just an estimate of MSE due to the resampling used
  • So, do we believe that argmin mean-CV is the best way to choose λ
    • (or some other model feature)
  • How about if we want the λ that is most popular
    • That is, the one which would be optimal most of the time
    • Do they converge to each other in the limit?
    • Like saying we want the λ that results in the most lowest-MSEs in new, unseen data, not just averages them