Computer Science
mHC: Manifold-Constrained Hyper-Connections
This paper presents a novel framework, mHC, which enhances residual connections by addressing training and scalability challenges through manifold constraints, promising advancements in foundational model designs.
Imagine building with LEGOs. A simple, deep tower (a basic neural network) can get wobbly and fall. Someone invented a special LEGO piece (a 'residual connection') that acts like a super-strong internal support beam, letting you build much taller, stable towers. Then, another builder tried adding lots of extra crisscrossing beams ('Hyper-Connections') for even more strength, but this made the whole structure complicated and surprisingly unstable again. This paper introduces a new, smarter way to add those extra beams ('mHC'). It's like using precisely engineered brackets that add strength without messing up the main support structure, resulting in the tallest, strongest, and most stable tower yet.
- 01
Proposes a framework to project residual connections onto a manifold to restore identity mapping.
- 02
Addresses training instability and scalability issues of Hyper-Connections.
- 03
Demonstrates superior scalability and performance improvements of mHC.
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.