[{"content":"1. why model complexity is not the number of parameters model selection criteria often penalize models according to the number of parameters they use. this is reasonable when every parameter direction independently affects predictions. but many modern models violate this assumption: different parameter values can represent the same function. low-rank regression, overcomplete models, and neural networks with symmetries all have redundant coordinates.\na simple thought experiment makes this issue clear. consider two models that represent the same family of functions:\nmodel a uses a minimal coordinate system model b uses an inflated coordinate system with redundant directions functionally, these models are identical. yet parameter-count penalties will prefer model a, appealing to occam\u0026rsquo;s razor. this mismatch motivates a basic question:\nwhat does bayesian evidence actually count when parameters are redundant?\n2. where the bic penalty comes from and where the assumption enters before diving into formulas, let\u0026rsquo;s build intuition for why curvature matters at all. imagine standing in a valley:\nin some directions the ground slopes steeply upward in others it is almost flat if we only know our location up to measurement noise, steep directions pin us down precisely, while flat directions leave us uncertain. bayesian evidence is, at its core, a calculation of how much volume of plausible parameter space remains after seeing data.\ncurvature determines how fast this volume shrinks as we collect more data.\nwe observe a dataset $$ D_n = {(x_i, y_i)}_{i=1}^n $$ of size $n$. a model is parameterized by $\\theta \\in \\mathbb{R}^d$ with prior distribution $\\pi(\\theta)$.\nthe likelihood is $$ p(D_n \\mid \\theta) = \\prod_{i=1}^n p(y_i \\mid x_i, \\theta), $$ and the marginal likelihood (evidence) is $$ p(D_n) = \\int p(D_n \\mid \\theta), \\pi(\\theta), d\\theta. $$\nthroughout this post, we consider the asymptotic regime where the sample size $n$ grows while the model class is fixed.\ngeometrically, it is useful to distinguish two kinds of directions in parameter space:\ncurved directions: small changes in $\\theta$ lead to quadratic changes in the loss or log-likelihood flat directions: changes in $\\theta$ leave the model\u0026rsquo;s predictions unchanged (or only change them at higher order) the central theme is that bayesian evidence is sensitive to curved directions, not the raw number of parameters.\nfor large sample size $n$, the marginal likelihood is approximated using laplace\u0026rsquo;s method. the key steps are:\nthe posterior concentrates near a maximizer $\\hat{\\theta}$ near $\\hat{\\theta}$, the log posterior is approximated by a quadratic integrating a gaussian in $d$ dimensions yields a volume factor writing $H_n$ for the hessian of the negative log posterior at $\\hat{\\theta}$, $$ \\log p(D_n) \\approx \\log p(D_n \\mid \\hat{\\theta}) - \\tfrac{1}{2} \\log \\det H_n + O(1). $$\nat this point, it is useful to introduce the fisher information. it measures how sensitive the likelihood is to small parameter changes:\nlarge fisher information means predictions change rapidly small fisher information means predictions barely change in regular models, the fisher information matrix is full rank. mathematically, $$ H_n \\approx n \\quad I(\\theta^\\star), $$ where $I(\\theta^\\star)$ is the fisher information at a kl minimizer.\nif $I(\\theta^\\star)$ is full rank with dimension $d$, $$ \\log \\det H_n \\sim d \\log n, $$ leading to $$ \\log p(D_n) \\approx \\log p(D_n \\mid \\hat{\\theta}) - \\frac{d}{2} \\log n + O(1), $$ which is the bayesian information criterion (bic).\ncrucially, this derivation assumes every parameter direction contributes curvature. if some directions are flat, this scaling breaks.\n3. what goes wrong in overparameterized models consider fitting a straight line using two parameterizations:\na minimal one with slope and intercept an inflated one using five constrained parameters both describe the same functions, but the second has directions where moving does nothing. these directions have zero fisher information; the data cannot constrain them.\nmore generally, many models have likelihoods that are flat along some directions. laplace\u0026rsquo;s method treats all directions as curved, overestimating posterior contraction and over-penalizing the model.\nthis error scales as $\\log n$, so it diverges with more data. to see this explicitly, we turn to a minimal example.\n4. a minimal example: rank-deficient linear regression consider the linear–gaussian model $$ y_i = x_i^\\top B\\theta + \\varepsilon_i, \\qquad \\varepsilon_i \\sim \\mathcal N(0, \\sigma^2), $$ with prior $\\theta \\sim \\mathcal N(0, \\tau^2 I_d)$.\nhere:\n$x_i \\in \\mathbb{R}^p$ $B \\in \\mathbb{R}^{p \\times d}$ $\\theta \\in \\mathbb{R}^d$ $\\varepsilon_i$ is gaussian noise $I_d$ is the identity matrix $\\tau^2$ controls prior scale if $\\mathrm{rank}(B) = r \u0026lt; d$, only an $r$-dimensional projection of $\\theta$ affects predictions.\n4.1 exact marginal likelihood because the model is gaussian, the marginal likelihood is exact. writing $A_n = X_n B$ and $S_n = A_n^\\top A_n$,\n$$ \\log p(D_n) = -\\tfrac{1}{2}\\Big(n\\log(2\\pi) + n\\log \\sigma^2 + \\log \\det(I + \\alpha S_n) + \\text{data-fit terms}\\Big), $$\nwith $\\alpha = \\tau^2/\\sigma^2$.\nthe spectrum of $S_n$ has:\n$r$ eigenvalues scaling as $n$ $d-r$ eigenvalues remaining $O(1)$ 4.2 effective dimension from eigenvalues $$ \\log \\det(I + \\alpha S_n) = r \\log n + O(1), $$\nso\n$$ \\log p(D_n) = \\log p(D_n \\mid \\theta^\\star) - \\frac{r}{2} \\log n + O(1). $$\n4.3 quantifying the bic error bic predicts a penalty of $\\frac{d}{2} \\log n$. the error is $$ \\Big(\\frac{d}{2} - \\frac{r}{2}\\Big)\\log n + O(1), $$ which diverges as $n$ grows.\nbayesian evidence penalizes the number of directions that actually change the model, not the number of parameters used to describe it.\n5. a primer on singular learning theory (slt) laplace\u0026rsquo;s method and bic assume the model is regular: the fisher information is full rank, the posterior concentrates at a unique point, and every parameter direction contributes curvature. many modern models violate this.\nmixture models, neural networks, and low-rank representations all exhibit redundant parameters and flat directions in the likelihood. in such singular models, the set of kl minimizers $$ {\\theta : \\mathrm{KL}(p^* | p_\\theta) = 0} $$ can have nontrivial geometry—curves, surfaces, or more complex singularities—rather than being isolated points.\nsingular learning theory (slt), developed by watanabe, shows that in these models the marginal likelihood is governed by the real log canonical threshold (rlct) $\\lambda$, a birational invariant that measures the local singularity structure: $$ \\log p(D_n) = \\log p(D_n \\mid \\theta^*) - \\lambda \\log n + (m-1)\\log\\log n + O(1), $$ where $m \\in \\mathbb{N}$ is the multiplicity. for regular models, $\\lambda = d/2$ and $m = 1$, recovering bic. for singular models, typically $\\lambda \u0026lt; d/2$.\nthe key insight is that effective complexity depends on the intrinsic geometry of the likelihood—not on the number of coordinates used to describe it.\n6. effective dimension and the real log canonical threshold (rlct) slt shows that\n$$ \\log p(D_n) = \\log p(D_n \\mid \\theta^\\star) - \\lambda \\log n + O(\\log \\log n). $$\n$\\lambda$, the rlct, is a coordinate-free effective dimension.\nin regular models, $\\lambda = d/2$. in singular models, $\\lambda \u0026lt; d/2$.\nfor rank-deficient regression, $\\lambda = r/2$.\n7. representation invariance rlct depends only on the function class, not the parameterization. bic does not satisfy this invariance.\novercomplete representations introduce near-zero curvature directions without changing the intrinsic subspace. both parameterizations span the same subspace, but the overcomplete model introduces additional near-zero eigenvalues corresponding to redundant coordinates.\n8. why this matters model complexity is a geometric property of the likelihood, not a bookkeeping exercise over parameters. slt provides the correct language for understanding generalization in overparameterized models.\nappendix a: exact marginal likelihood and evidence slopes in linear–gaussian models this appendix records the exact marginal likelihood calculation underlying section 4, and explains why the leading $\\log n$ term depends on intrinsic rank rather than parameter count.\nwe consider the linear–gaussian regression model $$ y_i = x_i^\\top B\\theta + \\varepsilon_i, \\quad \\varepsilon_i \\sim \\mathcal{N}(0, \\sigma^2), $$ with prior $\\theta \\sim \\mathcal{N}(0, \\tau^2 I_d)$. writing $X_n \\in \\mathbb{R}^{n \\times p}$ for the design matrix and $$ A_n := X_n B \\in \\mathbb{R}^{n \\times d}, $$ the likelihood can be written compactly as $$ p(y \\mid \\theta) = \\mathcal{N}(y \\mid A_n \\theta, \\sigma^2 I_n). $$\nbecause both likelihood and prior are gaussian, the marginal likelihood can be computed in closed form by integrating out $\\theta$. a standard gaussian completion of the square yields $$ p(y) = \\mathcal{N}(y \\mid 0, \\sigma^2 I_n + \\tau^2 A_n A_n^\\top). $$\ntaking logarithms, the log evidence is $$ \\log p(D_n) = -\\frac{1}{2}\\Big(n \\log(2\\pi) + n \\log \\sigma^2 + \\log \\det(I_d + \\alpha S_n) + \\text{data-fit terms}\\Big), $$ where $$ S_n := A_n^\\top A_n, \\quad \\alpha := \\tau^2 / \\sigma^2. $$\nthe key object controlling the complexity penalty is the gram matrix $S_n$.\nspectrum and intrinsic rank assume the rows $x_i$ are i.i.d. with nondegenerate covariance $\\Sigma_x$, and that $\\mathrm{rank}(B) = r \u0026lt; d$. then $$ \\frac{1}{n} S_n = \\frac{1}{n} B^\\top X_n^\\top X_n B \\longrightarrow B^\\top \\Sigma_x B \\quad \\text{almost surely}. $$\nas a consequence:\n$r$ eigenvalues of $S_n$ scale linearly with $n$ the remaining $d - r$ eigenvalues remain $O(1)$ taking determinants, $$ \\log \\det(I_d + \\alpha S_n) = \\sum_{j=1}^{r} \\log(\\alpha n \\lambda_j + 1) + O(1) = r \\log n + O(1). $$\nsubstituting back, the leading asymptotic form of the evidence is $$ \\log p(D_n) = \\log p(D_n \\mid \\theta^\\star) - \\frac{r}{2} \\log n + O(1). $$\nthis shows explicitly that the effective dimension of the model is $r$, not the ambient parameter count $d$. in the language of singular learning theory, the real log canonical threshold (rlct) is $$ \\lambda = r/2. $$\nby contrast, laplace\u0026rsquo;s approximation (and bic) always insert $d/2$ in place of $\\lambda$, leading to a systematic error of $$ \\Big(\\frac{d}{2} - \\frac{r}{2}\\Big) \\log n $$ in singular models. this is the precise sense in which bic \u0026ldquo;over-penalizes\u0026rdquo; overparameterized representations.\nfurther reading kalyaan rao, \u0026ldquo;evidence slopes and effective dimension in singular linear models,\u0026rdquo; preprint sumio watanabe, algebraic geometry and statistical learning theory, cambridge university press, 2009 sumio watanabe, mathematical theory of bayesian statistics, chapman and hall/crc, 2018 gideon schwarz, \u0026ldquo;estimating the dimension of a model,\u0026rdquo; the annals of statistics, 1978 ","permalink":"https://kalyaannnn.github.io/posts/rethinking-model-complexity/","summary":"\u003ch2 id=\"1-why-model-complexity-is-not-the-number-of-parameters\"\u003e1. why model complexity is not the number of parameters\u003c/h2\u003e\n\u003cp\u003emodel selection criteria often penalize models according to the number of parameters they use. this is reasonable when every parameter direction independently affects predictions. but many modern models violate this assumption: different parameter values can represent the same function. low-rank regression, overcomplete models, and neural networks with symmetries all have redundant coordinates.\u003c/p\u003e\n\u003cp\u003ea simple thought experiment makes this issue clear. consider two models that represent the same family of functions:\u003c/p\u003e","title":"rethinking model complexity in overparameterized models"},{"content":"stand still. the forest knows where you are.\nyou must let it find you.\n— david wagoner, lost\n","permalink":"https://kalyaannnn.github.io/about/","summary":"\u003cp\u003estand still. the forest knows where you are.\u003cbr\u003e\nyou must let it find you.\u003cbr\u003e\n— david wagoner, \u003cem\u003elost\u003c/em\u003e\u003c/p\u003e","title":"About"}]