Zen of Statistical Modelling

Learning = Representation + Evaluation + Optimization
It’s the generalization that counts
Data alone is not enough
Representable does not imply learnable
Represented in which kinds of knowledge are easily expressed
Feature engineering is the key
Intuition fails in high dimensions
Overfitting has many faces, bias and variance
If A is better than B at learning infinite data, B is often better than A at learning finite data
Ensemble, regularization and early stop are friends
More data and scalability beat a cleverer algorithm
Theoretical guarantees are usually loose and pessimistic
Simplicity does not imply accuracy
Correlation does not imply causation


Bibliography
Pedro Domingos. A Few Useful Things to Know about Machine Learning. Communications of the ACM, 55 (10), 78-87, 2012.