Meta-overfitting

Overfitting is a phenomenon in machine learning: very flexible learning methods with enough parameters will end up memorizing the training data; you'll see very low training error, but high error rates on new (test) data.

"Meta-overfitting" is my term to describe the same condition that takes place at a higher level of abstraction. The whole learning system (learning method, error metric, choices for cross-validation, particular training and test sets) produce a learning model. But the resulting model is very often too specific to the circumstances in which it was created, and won't generalize well to out-of-sample predictions. I suspect this happens quite a lot in machine learning contests.

(I first thought of this in late 2011 while playing around with a Kaggle contest. It seems that others were thinking the same thing: here and here.)