Exploring how iterative methods can make Gaussian processes scalable while properly quantifying computational uncertainty. This deep dive traces the mathematical foundations of GPs from univariate Gaussians to infinite-dimensional distributions, then introduces IterGP as a principled Bayesian approach that treats computation itself as a source of uncertainty when approximating representer weights.