The curve-fitting problem revisited

Abstract

Abstract The curve-fitting problem refers to the task of fitting a mathematical function to given observations. According to the standard analysis of the problem, this task involves a trade-off between simplicity and in-sample accuracy. Furthermore, when the goal is to use the fitted function for making predictions, simplicity and in-sample accuracy have to be balanced to avoid under- and overfitting and, thus, impeding the function’s predictive accuracy. This paper revisits the standard analysis of the curve-fitting problem in light of recent advances in statistics and machine learning. I show that applications of deep neural networks involve the curve-fitting problem for they are also concerned with fitting a mathematical function to given observations. Surprisingly, however, deep neural networks apparently undermine the standard analysis of the curve-fitting problem: they commonly exhibit high predictive accuracy without the need to actively balance simplicity and in-sample accuracy. I explore the implications of this phenomenon and argue that, nevertheless, the standard analysis of the curve-fitting problem can be saved under a suitable definition of simplicity. I also highlight how this renews the challenge to justify simplicity as a methodological norm in curve fitting.