Rethinking Weight Decay
From Scalar Shrinkage to Spectral Dynamics

Weight decay has a direction, too

When we tune weight decay, we usually tune its strength: how large should the coefficient be, should it change during training, and should different modules use different values? These questions all assume that the direction of decay is already settled. For matrix weights, it need not be.

AdamW separates the task update from weight shrinkage. The optimizer follows information from gradients; the decay term acts on the current weights. Ordinary weight decay (WD) points along $W$, but a matrix’s singular-value decomposition gives us other directions to consider.

On its own, ordinary WD makes every singular value smaller by the same factor. The matrix shrinks, but its relative spectrum stays the same. Can we instead choose which parts of the spectrum to shrink more strongly?

Spectral-power weight decay, denoted by $\mathcal{S}_p$ WD, makes this choice through a singular-value power. Let $W\in\mathbb R^{m\times n}$ have the compact rank-$r$ SVD $W=U\operatorname{diag}(\sigma_1,\ldots,\sigma_r)V^\top$, where $U,V$ have orthonormal columns and $\sigma_i>0$ are the nonzero singular values. Define

The exponent $p\ge0$ determines how shrinkage varies with mode size. We use $S_p(W)$ for the decay direction and $\mathcal{S}_p$ WD for the method. The associated penalty is

Here $\lVert W\rVert_{S_q}=(\sum_i\sigma_i^q)^{1/q}$ is the Schatten norm. For $p>0$, $S_p=\nabla R_p$; at $p=0$, $S_p$ is a nuclear-norm subgradient (Lewis, 1995; Watson, 1992). Ordinary WD is recovered at $p=1$. Other powers change the relative pressure on the large and small singular values—the head and tail of the spectrum.

The next question is whether changing the weight spectrum also changes the representations the model learns, and whether those changes help prediction. We will follow this progression from a solvable linear model to Transformer experiments.

1. From ordinary weight decay to decoupled optimizers

1.1 When L2 regularization and weight decay coincide

Ridge regression and Tikhonov regularization add a quadratic penalty to control parameter size (Hoerl & Kennard, 1970; Tikhonov & Arsenin, 1977):

Let $L$ be the task loss, $\beta\ge0$ the L2 penalty coefficient, $\alpha>0$ the base step size, and $\eta_t\ge0$ a schedule multiplier. In plain SGD, the update is

Thus, L2 regularization and weight decay give the same plain-SGD update when $\lambda=\alpha\beta$. The penalty controls total squared weight magnitude, which can also be written as

Early analyses of weight decay studied how this norm penalty suppresses irrelevant components and limits the effect of noise in the training targets (Krogh & Hertz, 1991).

1.2 Why the distinction matters beyond plain SGD

Momentum averages gradients over time (Sutskever et al., 2013); AdaGrad and Adam also rescale coordinates using gradient history (Duchi et al., 2011; Kingma & Ba, 2015). Write the parameters as a vector $w_t$, the stochastic task gradient as $g_t$, and the preconditioner as $P_t$. In Adam, the diagonal scaling uses the inverse square root of a second-moment estimate, with a numerical stabilizer. Leaving out momentum averaging for the moment, coupled L2 regularization has the schematic form

Decoupled decay instead acts directly on the parameters:

For plain SGD, $P_t=I$, and matching $\lambda=\alpha\beta$ recovers the equivalence above. In Adam, a coupled L2 term also enters the moment estimates and is transformed by the preconditioner. Moving it outside the optimizer therefore changes the algorithm, not just the notation. This distinction can also affect generalization (Tang et al., 2025; Zou et al., 2023).

1.3 A separate branch for decay

AdamW makes this separation explicit (Loshchilov & Hutter, 2019). Throughout the article, we write the decoupled update as

Here $D_t$ is the task direction computed from the gradient and optimizer state. The base coefficients $\alpha$ and $\lambda$ are fixed; $\eta_t$ schedules both terms. “Decoupled” means that decay bypasses the optimizer’s preconditioning and state updates. It does not require the two terms to have separate schedules.

Much of the work on this branch concerns decay strength. AdaDecay assigns parameter-wise rates using gradient statistics (Nakamura & Hong, 2019). Adaptive WD adjusts coefficients according to the relative strengths of task and L2 updates (Ghiasi et al., 2023). AlphaDecay assigns module-wise coefficients in LLMs to balance heavy-tailed spectral statistics (He et al., 2025).

These methods refine how much decay to apply and where to apply it. Parameter-wise scaling can itself change the update’s geometry, so strength and direction are not completely separate. Our question is more specific: can we prescribe shrinkage in the weight matrix’s singular basis?

1.4 Matrix-aware optimizers make geometry explicit

K-FAC, Shampoo, SOAP, and Muon already use matrix structure to shape task updates (Gupta et al., 2018; Jordan et al., 2024; Martens & Grosse, 2015; Vyas et al., 2025). Muon offers a particularly simple example: its idealized matrix update preserves the singular directions of a momentum matrix while replacing each nonzero singular value by one (Jordan et al., 2024). Polar Express efficiently approximates this polar-factor computation (Amsel et al., 2026).

The same kind of matrix operation can be useful in the decay branch, but it acts on a different object. Muon transforms task momentum; $\mathcal{S}_p$ WD transforms the current weights. One chooses how to follow the task signal, the other how to regularize the parameters.

2. What ordinary weight decay controls

2.1 Isolated decay changes scale, not normalized shape

Set the task update aside. For nonzero $W=U\Sigma V^\top$, with $\Sigma=\operatorname{diag}(\sigma_i)$, an ordinary-WD step gives

where $\eta$ is the current schedule multiplier and $0\le a<1$. It follows that

Both the spectral and Frobenius norms shrink, but all singular-value ratios remain unchanged. So do stable rank and entropy effective rank, or eRank, defined below. This is a statement about an isolated decay step. During training, task updates also change singular values and singular vectors; decay can influence the final spectrum through its interaction with those updates (Loshchilov & Hutter, 2019).

2.2 Norm control can still affect learned structure

Ordinary WD can therefore be associated with low-rank structure even though its isolated step preserves spectral ratios. Its interaction with optimization has several mechanisms (Zhang et al., 2019). Factorization adds another: the model may depend on a product of weights rather than either factor alone. For query and key weights $W_Q,W_K$, write $P=W_Q^\top W_K$. When the factor dimensions are large enough to represent $P$, the classical identity is (Srebro et al., 2004)

Here $\lVert P\rVert_*=\sum_i\sigma_i(P)$ is the nuclear norm, and the minimum is over compatible factors $A,B$. This connects quadratic penalties on factors to a low-rank preference for their product. The minimization matters: arbitrary factors encountered during training need not attain it. Analyses of low-rank attention likewise require specific balance and dynamics conditions (Kobayashi et al., 2024).

Weight norms also affect effective step size. In a scale-invariant parameterization, rescaling the weights changes how much a given update moves the represented function (van Laarhoven, 2017). Hyperball addresses this directly by controlling the Frobenius norms of both weights and optimizer updates (Wen et al., 2026). For a small task step, the angular displacement in parameter space is approximately

The component $D_{t,\perp}$ is tangent to the weight sphere, and $\langle A,B\rangle_F=\operatorname{tr}(A^\top B)$. When the task direction has a controlled norm, $\eta_t\alpha/\lVert W_t\rVert_F$ serves as a proxy for the effective learning rate (ELR). Thus, changing weight scale can change learning dynamics even without directly changing spectral shape.

2.3 Reading the spectrum: notation and rank metrics

For nonzero $W\in\mathbb R^{m\times n}$, order the singular values as $\sigma_1\ge\cdots\ge\sigma_R\ge0$, where $R=\min(m,n)$ includes zero modes. Define $r_i=\sigma_i/\sigma_1$ and, for a probability vector $q$, $H(q)=-\sum_iq_i\log q_i$, with $0\log0=0$. The spectral norm is $\lVert W\rVert_2=\sigma_1$. In the table, $k$ is the number of leading modes retained, and normalized entropy assumes $R>1$.

Stable rank and eRank both ignore overall scale, but they summarize the spectrum differently. Stable rank compares total squared energy with the largest mode; eRank measures how evenly singular-value mass is distributed. Output-energy effective rank instead uses squared singular values. None of these is attention entropy, which is computed from post-softmax attention probabilities (Roy & Vetterli, 2007; Vaswani et al., 2017; Zhai et al., 2023). Below, “rank” refers to the named statistic, not necessarily algebraic rank.

3. Spectral-power weight decay: direction, theory, and spectrum shaping

3.1 Definition and associated penalty

To move beyond equal relative shrinkage, replace each singular value in the decay direction by a power:

Three useful cases are

At $p=1$, this is ordinary WD. At $p=0$, the active polar factor assigns the same absolute decay to every nonzero mode. At $p=2$, large modes receive more decay in both absolute and relative terms. Zero modes contribute zero to $S_0$, consistent with the active polar factor and the nuclear-norm subdifferential (Cai et al., 2010; Higham, 1986; Watson, 1992).

For $p>0$, this direction is the gradient of a unitarily invariant penalty:

The exponent in the direction is $p$, whereas the Schatten penalty has order $p+1$ (Lewis, 1995). At $p=0$, the penalty is the nonsmooth nuclear norm, and $S_0$ is one of its subgradients (Cai et al., 2010; Watson, 1992). This case is also studied as a way to encourage low-rank weights (Andriianov et al., 2026).

We apply the direction through the decoupled update

We keep the same convention for fixed base coefficients and the shared schedule. A practical point is worth emphasizing: changing $p$ changes both the norm of $S_p(W)$ and how that norm is distributed across modes. Equal numerical values of $\lambda_p$ do not match decay strength across powers.

3.2 Computing $\mathcal{S}_p$ WD

For integer $p\ge0$, the direction can be computed without an SVD. Let $m$ and $n$ be the output and input dimensions, choose the smaller Gram matrix, and write the active polar factor as $Q_W=UV^\top$. For any nonnegative integer $k$,

These identities are exact. In practice, we approximate $Q_W$ with Polar Express (Amsel et al., 2026).

Algorithm 1. Computing spectral-power weight decay with a smaller Gram matrix and a polar-factor approximation. Odd powers need only matrix products; even powers also use Polar Express. The final line allows ordinary WD and the spectral term to be used together.

There is no need to form a Gram matrix at $p=0$ or $p=1$. For larger powers, repeated squaring reuses intermediate results—for example, $G^4=(G^2)^2$. The cost depends on matrix shape, exponent, and the number of polar iterations; avoiding an SVD does not make the extra decay term free. The check below uses eight Polar Express iterations, denoted PE8.

3.3 Pure decay: separating scale from shape

Start with decay alone. Let continuous time $t$ represent cumulative schedule time, $\sum_j\eta_j$, and set $\lambda=\lambda_p$. Since decay preserves singular directions, each positive singular value follows

For an initially positive mode, the solutions for the powers used below are

The relative shrinkage rate makes the difference between powers especially clear:

At $p=1$, every mode loses the same fraction of its size. For $p>1$, large modes shrink faster in relative terms, so the normalized spectrum flattens. For $0\le p<1$, small active modes shrink faster in relative terms and can reach zero in finite continuous time.

The continuous flow should not be confused with a finite explicit step. At $p=0$, the positive-part solution has the same form as nuclear-norm proximal soft-thresholding (Cai et al., 2010; Parikh & Boyd, 2014). But an explicit polar step that crosses zero produces $|\sigma_i-\eta\lambda|$, rather than clipping the mode to zero. For the explicit method, “tail suppression” is therefore more precise than claiming automatic rank truncation.

While $W\ne0$,

Only at $p=1$ is the relative Frobenius contraction rate independent of the current spectrum.

Figure 1. Pure continuous decay from a shared initial spectrum, at different cumulative decay exposures $\tau$. Each curve is normalized by its current largest singular value. At $p=0$, small modes reach zero first; at $p=1$, all normalized curves coincide; at $p=2$, the relative spectrum becomes flatter.

The upward movement of the normalized $p=2$ curves does not mean that any singular value grows. Every active mode is shrinking; the largest one simply shrinks proportionally faster. This distinction will matter when we compare trained models.

Pure decay can also approximate a local segment of training when $\alpha\lVert D_t\rVert_F\ll\lambda_p\lVert S_p(W_t)\rVert_F$. A falling learning rate alone does not imply this condition: the task and decay terms share the same multiplier $\eta_t$.

The coefficient can change shape, too. For one explicit step, let $r_i=\sigma_i/\sigma_1$ and $\kappa=\eta\lambda\sigma_1^{p-1}$, the relative decay applied to the leading mode. If the step preserves signs and ordering,

For $0<r_i<1$, the derivative is zero at $p=1$, positive for $p>1$, and negative for $0\le p<1$. The exponent determines the direction of the shape change; the coefficient determines how far the step moves in that direction. Ordinary WD is the special case in which changing isolated decay strength leaves normalized shape untouched.

3.4 A linear toy model: from weight spectra to output spectra

We can now add a fitting objective and solve for the resulting spectrum. Let the dense teacher be $A=U_A\operatorname{diag}(a_i)V_A^\top$, with singular values $a_i$. Consider

Here $\lambda$ is a coefficient in the objective; for plain gradient descent, it corresponds to $\lambda_p/\alpha$ in our update notation. The trace inequality allows the optimum to be chosen with the teacher’s singular vectors (Lewis, 1995). The squared error is strongly convex and the spectral penalty is convex, so the optimal matrix is unique. Its singular values $s_i$ solve the scalar problems (Parikh & Boyd, 2014)

Ordinary WD gives $s_i=a_i/(1+\lambda)$ at $p=1$, scaling all teacher modes equally. For $p=2$ and $\lambda>0$,

so large teacher modes are compressed more strongly in relative terms. At $p=4$, the scalar equation again has a unique nonnegative root. At $p=0$, the solution is singular-value soft-thresholding (Cai et al., 2010). These are different spectral responses to the same fitting task.

We verify the solutions with plain gradient descent, using a proximal step at $p=0$. Whitened inputs then let us check exactly how the learned weight spectrum appears in the outputs.

Finite-sample check. We construct $N=512$ paired positive/negative samples in dimension $d=6$. Reduced QR yields a centered design $X\in\mathbb R^{N\times d}$ satisfying

to a maximum absolute error of $6.66\times10^{-16}$; the largest absolute coordinate mean is $1.87\times10^{-17}$. The non-diagonal teacher has singular values $(4,2,1,0.4,0.15,0.05)$, and we use $\lambda=0.2$. This construction gives an exactly whitened finite-sample input up to numerical precision (Kessy et al., 2018).

The optimized singular values $\hat\sigma_i$ closely match the analytic values $s_i^\star$:

The last column checks a separate identity. Since the output matrix $Y=XW^\top$ is centered,

Thus, the nonzero covariance eigenvalues are $\sigma_i(W)^2$, while the singular values of $Y/\sqrt N$ are $\sigma_i(W)$. This identity holds for any $W$ under these inputs; it is distinct from the preceding column’s check that optimization reached the analytic solution.

From the $p=0$ solution to the $p=4$ solution, output-energy effective rank rises from $1.869$ to $2.923$, and activation eRank rises from $2.812$ to $4.100$. Both reflect a less concentrated spectrum, but the former weights modes by squared singular values and the latter by singular values.

Figure 2. The linear teacher experiment. Top left: analytic and optimized weight spectra. Top right: each run’s regularized objective minus its own optimum. Bottom left: output-covariance eigenvalues. Bottom right: singular values of the centered outputs divided by $\sqrt N$.

The agreement between the top-left and bottom-right panels is a consequence of whitening, not an additional empirical correlation. The optimization-gap panel rules out incomplete convergence as the explanation for the different endpoint spectra; it does not compare a common unregularized loss across powers.

For general centered inputs with covariance $C_x$, the output covariance is $WC_xW^\top$, rather than $WW^\top$. The input spectrum and its alignment with the weight’s singular vectors then matter as well. This is why the clean weight-to-activation correspondence here should not be assumed in a Transformer.

3.5 An eight-layer Transformer: does the spectral effect survive training?

We next compare $p\in\{0,1,2,4\}$ in an 8-layer, width-512 SwiGLU Transformer. We first hold the decay coefficient fixed, then sweep it separately for each power.

Varying the power. How does the spectrum change beyond the leading mode?

Figure 3. Normalized weight spectra for the eight-layer Transformer. Lines show the median across layers; shading shows the 25th–75th percentiles across layers. The horizontal axis is fractional singular-value index, and the vertical axis is $\sigma_i/\sigma_1$.

The clearest separation is at $p=4$, especially in the MLP expansion and projection matrices. More of their spectrum remains large relative to the leading mode. Q and K show much smaller differences, with the curves largely overlapping. The pure-decay tendency is still visible after training, but its magnitude depends strongly on the matrix family.

Figure 4. Spectral distributions of the eight MLP expansion matrices. Left: $\log_{10}\sigma_i$. Right: $\log_{10}(\sigma_i/\sigma_1)$, using each matrix’s own largest singular value. Each matrix contributes equal probability mass. The vertical axis is mass per bin; smoothing is only for display.

The two panels separate absolute size from relative concentration. At $p=4$, the upper end of the absolute spectrum moves left, while the normalized bulk moves toward zero—that is, toward $\sigma_i/\sigma_1=1$. A flatter normalized spectrum can therefore come from suppressing the head, without increasing the absolute size of the other modes. The taller peak indicates that mass is concentrated in a narrower range of log singular values, not that the matrix has acquired more modes.

At the fixed coefficient $\lambda=0.01$, the endpoint statistics are:

MLP stable rank changes the most, while QK-product stable rank also rises. Attention entropy is not monotone across all four powers: it decreases slightly from $p=0$ to $p=1$ before increasing. A less concentrated weight spectrum does not determine the post-softmax attention distribution.

Varying the coefficient. To separate the direct decay effect from training, we first apply analytic pure decay to the spectrum of a shared checkpoint:

Figure 5. Analytic decay from the same checkpoint, with no task updates, over the fixed duration $T=6$ indicated in the plot. Columns correspond to $p=0,1,2,4$; rows show absolute and top-mode-normalized log spectra. Here $T$ is decay time, not the polar-iteration count in Algorithm 1.

At $p=1$, increasing the coefficient translates the absolute log-spectrum while leaving the normalized distribution exactly unchanged. At $p>1$, it also compresses the spread between the head and the bulk, most visibly at $p=4$. The sharper normalized peak is another view of a flatter singular-value spectrum. At $p=0$, the relative tail is instead suppressed.

After full training, increasing the coefficient from $0.003$ to $0.03$ gives the following changes in the MLP expansion matrices:

Figure 6. MLP expansion spectra at the end of full training, using the same coefficient values as Figure 5. Unlike the pure-decay comparison, these checkpoints have followed different task-and-decay trajectories. The normalized $p=1$ curves nearly overlap; the other powers show a larger shape response.

The trained MLP spectra retain the main pattern of the pure-decay comparison, but not its exact invariances. Ordinary WD produces only a small normalized-shape change here. That observation does not extend to every matrix family: in the same ordinary-WD sweep, mean Q and K stable ranks fall by about 7.7% and 14.4%, respectively. Once task updates are included, even ordinary WD can alter the learned spectral shape.

Figure 7. Final-checkpoint metrics from the coefficient sweep. Top row: MLP spectral norm, Frobenius norm, and stable rank. Bottom row: MLP entropy effective rank, QK-product stable rank across 32 heads, and validation bits per byte (BPB; lower is better).

At $p=4$, increasing the coefficient from $0.003$ to $0.03$ raises mean MLP stable rank from about $200.7$ to $250.9$, while lowering the spectral norm from about $2.54$ to $1.75$. The Frobenius norm also falls. There is no contradiction: stable rank is $\lVert W\rVert_F^2/\lVert W\rVert_2^2$, so it can rise when the leading singular value falls proportionally more than the Frobenius norm. The much smaller change in entropy effective rank suggests that the effect is chiefly a redistribution of relative magnitude, rather than the appearance of many new active dimensions.

The validation panel is equally important. Stronger $p=4$ decay produces the highest ranks in this sweep, but also worse BPB. The experiment establishes control over spectral geometry; it does not support maximizing rank as a training objective. Equal raw coefficients also do not give an equal-strength comparison across powers.

The remaining question is how to use this control within a training recipe. We turn to a case where the task-update geometry and the decay geometry can be adjusted separately.

4. Case study: Head-wise Muon and V spectral regularization

4.1 From head-wise to full-matrix V updates

Public reports for GLM-5, Kimi K3, and Qwen3.8-Next use head-wise Muon on query, key, and value weights (GLM-5 Team, 2026; Kimi Team, 2026; Qiu et al., 2026); DeepSeek-V4.1-Flash instead uses head-wise updates only for query and key (DeepSeek-AI, 2026).

Use H for head-wise updates and F for full-matrix updates, in Q/K/V order. HHH splits all three projections by head. HHF changes only V, applying Muon to the full matrix spanning all heads. Since Muon orthogonalizes task momentum, changing this grouping changes the task-update geometry. It does not directly make the weights or activations orthogonal.

Q/K determine where information is read from; V supplies the content being read and aggregated (Vaswani et al., 2017). Whether V heads learn complementary content features therefore deserves separate consideration. Head-wise polar shapes each momentum block separately, while full-matrix polar processes the entire V momentum jointly, including directional overlap across heads. This suggests a natural interpretation of HHF: joint updates may help the heads make fuller use of the available representation space. The learned representations change, while the architecture and parameter count remain the same.

We first compare HHH with HHF. We then keep the HHF update rule and add spectral decay to V, asking whether a separate adjustment to its weight spectrum helps.

4.2 Keeping the update rule, adjusting V decay

We retain ordinary WD and add a $p=2$ term to the full V weight:

The implementation uses LR-coupled coefficients: both $0.1$ and $\lambda_V$ multiply the current learning rate $\eta_t\alpha$. The definitions of $\lambda_1$ and $\lambda_2$ above translate this convention into our notation. The ideal decay direction is the gradient of $\lambda_1\lVert W_V\rVert_F^2/2+\lambda_2\sum_i\sigma_i(W_V)^3/3$. In an isolated step that does not cross zero, mode $i$ has relative shrinkage

The ordinary term shrinks every mode by the same fraction; the $p=2$ term adds pressure in proportion to its singular value. This changes the regularization while leaving the Muon update rule in place.

We compute the added term from the pre-update weight as $(WW^\top)\operatorname{polar}(W)$, approximating the polar factor with Polar Express (Amsel et al., 2026). The comparison HHH → HHF → HHF + $\mathcal{S}_p$ is between separate training runs, not successive stages of one run. Keeping the update rule fixed also does not mean that the realized task updates stay identical once the trajectories diverge.

4.3 Validation gains and their relation to rank

Final validation NLL is $3.01968$ for HHH, $3.01596$ for HHF, and $3.01324$ for the selected HHF + $p=2$ run. Moving from HHH to HHF changes the V task-update scope. Adding $p=2$ decay then improves on the HHF control by about $0.00272$, at $\lambda_V=7\times10^{-5}$.

The two HHF trajectories have very similar overall V weight RMS, and the added spectral step is much smaller than the task step. This is a weak regularizer acting throughout training, not a decay-dominated regime. Pure-decay analysis describes the bias of the extra term; its effect on loss comes from its interaction with task updates.

Figure 8. HHH, HHF, and the selected HHF + full-V $p=2$ run from a common initialization. Left: full validation trajectories. Middle: the late-training window, at steps $\ge14\mathrm{k}$. Right: validation NLL relative to the HHF control, with that window shaded.

The full-scale curves nearly overlap, while the difference plot reveals persistent differences throughout the late-training window: HHF has lower validation loss than HHH, and adding $p=2$ decay improves it further.

Figure 9. Endpoint V geometry versus final validation NLL. Top row: weight eRank, weight stable rank, and actual activation eRank. Bottom row: fixed-input activation eRank, actual activation head CKA, and fixed-input head CKA. Ranks are not divided by dimension. Crosses and stars denote HHH and HHF; circles, squares, and diamonds denote the HHF $p=0,1,2$ sweeps; triangles denote HHH + $p=2$. Heavy outlines mark the best point in each family. The dashed line is the HHF control loss.

From HHH to HHF, the distinction is clearer in V’s content representations. Actual activation eRank rises from about $410$ to $434$, while cross-head CKA falls from about $0.525$ to $0.474$. The fixed-input comparison likewise shows higher eRank and lower CKA. On the sampled inputs, HHF’s content responses span more effective directions and show less statistical overlap across heads. This is consistent with one possible explanation: full-V updates help the heads learn more complementary content features, reducing representation redundancy that may arise with head-wise updates. The fixed-input result also locates a difference in the V map itself, rather than attributing it entirely to changes upstream.

The plot offers another interesting comparison: some HHH + $p=2$ runs have higher weight eRank or stable rank than HHF, yet worse loss. HHF’s advantage is therefore more naturally interpreted through the content representations it learns. The effective directions in the joint activations, complementarity across heads, and the task’s use of those directions together offer a fuller explanation than weight rank alone. This also connects to recent Muon discussions of the relationship between spectral shape and task performance (Goyal & Ramkumar, 2026).

Within HHF, spectral decay offers finer control over the representations. Actual activation-eRank points form an almost vertical cloud: similar effective dimensions accompany different losses. Compared with the HHF control, the best $p=2$ point has slightly lower weight eRank, nearly unchanged actual activation eRank, modestly higher fixed-input eRank, and higher values of both CKAs. Similar effective dimensions can therefore accommodate different ways of organizing content and different task outcomes.

The best $p=1$ control also lies very close to the best $p=2$ point. This makes V regularization an especially useful component to tune separately: adjusting ordinary decay already helps, while spectral powers additionally let us choose how shrinkage is distributed across modes. The more interesting question is how to align that extra freedom with the needs of the content representations.

We conjecture that full-V updates and spectral decay can improve effective representational capacity in complementary ways. The former processes update directions jointly across heads, helping them learn complementary content; the latter adjusts relative mode magnitudes, allowing task updates to make fuller use of the representation space. For example, $p>1$ suppresses overly dominant modes while preserving the relative contribution of the bulk and tail, potentially letting the same parameters support richer effective representations. This capacity involves both the number of effective directions and how information is distributed across heads. The next experiment asks how much these internal structures can differ when final validation loss is almost the same.

5. Case study: nearly the same pretraining loss, different rank geometry

5.1 Choosing a spectral preference for each module

We compare ordinary WD with an asymmetric recipe that assigns different powers to Attention and MLP weights, then look for runs with nearly matching final validation loss.

Attention mixes information across tokens; the MLP transforms features within each token. This suggests testing different spectral preferences: $p_A=0$ for Attention, to put relatively more pressure on weak modes, and $p_M=2$ for the MLP, to reduce the dominance of large modes. The experiment is motivated by low-rank attention analysis and module-wise decay allocation (He et al., 2025; Kobayashi et al., 2024), rather than an assumption that either module has a universally optimal rank.

Both families use Muon for matrix updates and the independent-coefficient convention: $\eta_t$ multiplies both branches, but $\lambda_A$ and $\lambda_M$ receive no additional factor of $\alpha$. There is no extra ordinary-WD term. These coefficients therefore cannot be compared numerically with the LR-coupled coefficients in Section 4.

The baseline uses $p_A=p_M=1$; the asymmetric recipe uses $p_A=0,p_M=2$. Here $A$ and $M$ denote Attention and MLP. Both families are allowed to tune their module-wise coefficients over the same grid of base learning rates and decay coefficients:

This gives $3\times3\times3=27$ configurations per family, or 54 in total. Below, we compare the two runs with the closest final validation losses. They share the model, initialization, seed, ordered data stream, and validation data. Both train for 7,268 steps with $\alpha=8\times10^{-4}$:

The final losses differ by approximately $1.47\times10^{-5}$ nats.

Figure 10. Validation trajectories of the selected ordinary-WD and asymmetric runs. Left: training after warmup. Right: a close-up around the final learning-rate cooldown, marked by the dashed line. Both runs finish at step 7,268.

The trajectories are close well before the final point. Their loss gap is about $0.00224$ nats before cooldown and narrows further by the end. Similar fitting outcomes make the internal-geometry comparison below especially interesting.

5.2 Similar loss leaves room for different internal geometry

In this pair, asymmetric decay produces more concentrated Attention weight spectra and higher-stable-rank MLP spectra, broadly matching the intended preferences. “Attention” in these weight statistics refers to the projection weights, not the post-softmax attention maps. The QK-product stable rank nevertheless rises. A product’s spectrum depends on the alignment of its factors’ singular vectors as well as their singular values, so lower factor ranks need not translate into a lower product stable rank.

Figure 11. Nearly matched final validation loss, different endpoint geometry. The right panel reports changes relative to the ordinary-WD baseline: Attention weight stable rank, MLP weight stable rank, QK-product stable rank, dimension-normalized MLP eRank, and normalized attention entropy.

MLP stable rank increases by 40.3%, while its dimension-normalized eRank increases by only 1.8%. These measures emphasize different features: the former is particularly sensitive to the leading singular value; the latter summarizes the distribution across all modes. The result is better described as a substantial change in relative spectral concentration than as a 40% increase in representational dimension.

The accompanying forward-pass measurements also differ. From ordinary WD to asymmetric decay, normalized attention entropy falls from $0.5814$ to $0.5672$, while mean maximum attention probability rises from $0.2408$ to $0.2671$. Q activation RMS rises from $1.2088$ to $1.3064$, and MLP output RMS falls from $2.0829$ to $1.7333$. Thus, nearly equal average validation loss can coexist with different attention distributions and internal scales.

Figure 12. Normalized weight spectra at the selected endpoints. Attention projections become more concentrated, while MLP spectra become flatter. The norm measurements in the table give the corresponding absolute-scale comparison.

The logarithmic horizontal axis emphasizes the leading modes and compresses much of the remaining spectrum. A modest visual separation across many middle and tail modes can still matter for stable rank, which sums $(\sigma_i/\sigma_1)^2$ over indices without logarithmic weighting. The normalized curves also hide an important scale difference: Attention spectral norm is larger in the asymmetric run, whereas MLP spectral norm is smaller.

These runs retain different internal spectra and activation statistics at nearly the same pretraining loss. Spectral decay thus offers an interesting form of control: exploring different information allocations and representation structures at similar fitting performance, then comparing how they behave in transfer, fine-tuning, and robustness.

6. A related extension: spectral projection in the decay branch

A power law changes shrinkage across the whole spectrum. We can also ask which modes should receive it. Recent work uses a shifted Gram matrix and a symmetric matrix sign to select momentum subspaces above or below a threshold (Goyal & Ramkumar, 2026). Applying the same construction to weights gives a band-selective decay rule without changing the task optimizer.

Constructing the projector. Take $W\in\mathbb R^{m\times n}$ with $m\ge n$, and choose a singular-value threshold $\tau>0$ that is not equal to any singular value. Set $G=W^\top W\in\mathbb R^{n\times n}$ and

Here $I_n$ is the identity. The symmetric matrix sign keeps eigenvectors and replaces each eigenvalue by its sign: if $A=Z\operatorname{diag}(\xi_i)Z^\top$, then $\operatorname{msign}(A)=Z\operatorname{diag}(\operatorname{sign}\xi_i)Z^\top$. Since $\xi_i=\sigma_i(W)^2-\tau^2$, its positive and negative eigenspaces correspond exactly to modes above and below the threshold. $P_{\mathrm{top}}$ and $P_{\mathrm{bot}}$ project onto these subspaces.

The source paper applies this operation to momentum $M$, constructing $P_{\mathrm{top}}(M)$ and $P_{\mathrm{bot}}(M)$ from $M^\top M-\tau^2I$. With $Q_M=\operatorname{polar}(M)$, the two task directions are

The selected subspace receives the polar direction; the other retains raw momentum. For weight decay, the analogous mixed rule applies $S_p$ in one subspace and ordinary WD in the other:

Alternatively, leave the unselected subspace unchanged by decay. To distinguish this from the mixed rule, write

Computing the projector without an SVD. Normalize the symmetric shifted matrix as $Z_0=A/(\lVert A\rVert_F+\varepsilon)$, then apply the convergent fifth-degree Newton–Schulz iteration

This odd polynomial preserves eigenvalue signs and drives nonzero modes toward $\pm1$. Forming $C_j$ allows $C_j^2$ to be reused. Modes near the threshold correspond to eigenvalues near zero and take longer to resolve; the source paper uses 20 FP32 iterations for both sign calls in NanoGPT (Goyal & Ramkumar, 2026). At a finite iteration count, the resulting matrices are approximate projectors, not exact binary selectors. Accuracy depends on the spectral gap around the threshold.

The resulting mode-wise update. For an isolated top-selective step, expressed in the current singular basis and small enough to keep the mode coefficients nonnegative,

In continuous time, the corresponding pure-decay law is $\dot\sigma_i=-\lambda_{\tau,p}\sigma_i^p\mathbf1\{\sigma_i>\tau\}$. The threshold selects the modes, the power sets their shrinkage law, and the coefficient sets its strength. Recomputing the projectors from the current $W$ also lets us assign separate powers and coefficients to the two subspaces. For $m\ge n$,

Setting one coefficient to zero restricts decay to one band. Setting both powers to one and both coefficients equal recovers ordinary WD. For $m<n$, use $WW^\top$ and left-multiply by the corresponding projectors, as in Section 3.2.

This extension separates two decisions: which modes to regularize, and how strongly shrinkage should depend on their size. Its usefulness in training still needs to be tested; the construction itself only specifies the decay rule.

7. Closing thoughts

Ordinary WD scales all singular values equally in an isolated step. $\mathcal{S}_p$ WD changes that rule with one exponent: $p<1$ puts relatively more pressure on weak modes, while $p>1$ puts more pressure on strong ones. The linear model makes the effect exact; the eight-layer Transformer shows that it can remain visible after task-driven training.

The case studies show concrete uses for this control. In the V experiment, full-matrix updates learn more spread-out content representations with less cross-head overlap, and further regularization tuning improves the default HHF recipe. In the asymmetric experiment, nearly matching validation loss still accompanies markedly different weight spectra and activation statistics. These results invite a further question: how should mode magnitudes and roles be allocated to make fuller use of a model’s effective representational capacity?

The useful addition is a way to specify and test spectral preferences rather than leaving them entirely to the optimizer and scalar decay. Powers, coefficients, module allocation, and band selection offer progressively finer choices. Which ones help depends on the task and training regime. Stage-dependent spectral regularization in ORCA (Y. Liu et al., 2026) and quantization-oriented spectral decay in S²D (Chavan et al., 2026) suggest concrete settings in which to study that question.

Spectrum shaping may also serve as a tool for training stability. Spectral-norm reparameterization has been used to mitigate attention entropy collapse and improve Transformer training stability (Zhai et al., 2023). We conjecture that $p>1$ or spectral-band selection could target excessively amplified modes, helping control spectral-norm growth, reduce loss spikes, and widen the range of stable learning rates.

Weight decay need not shrink every mode in the same proportion. Choosing a different shrinkage law gives us control over the spectrum; whether that control is useful has to be answered by the task.