https://kalyaannnn.github.io/tags/bayesian-inference/Recent content in Bayesian Inference on Kalyaan RaoHugo -- 0.160.1en-usSun, 04 Jan 2026 00:00:00 +0000https://kalyaannnn.github.io/posts/rethinking-model-complexity/Sun, 04 Jan 2026 00:00:00 +0000https://kalyaannnn.github.io/posts/rethinking-model-complexity/<h2 id="1-why-model-complexity-is-not-the-number-of-parameters">1. why model complexity is not the number of parameters</h2> <p>model selection criteria often penalize models according to the number of parameters they use. this is reasonable when every parameter direction independently affects predictions. but many modern models violate this assumption: different parameter values can represent the same function. low-rank regression, overcomplete models, and neural networks with symmetries all have redundant coordinates.</p> <p>a simple thought experiment makes this issue clear. consider two models that represent the same family of functions:</p>