mHC: Manifold-Constrained Hyper-Connections
TL;DR
Imagine building with LEGOs. A simple, deep tower (a basic neural network) can get wobbly and fall. Someone invented a special LEGO piece (a 'residual connection') that acts like a super-strong internal support beam, letting you build much taller, stable towers. Then, another builder tried adding lots of extra crisscrossing beams ('Hyper-Connections') for even more strength, but this made the whole structure complicated and surprisingly unstable again. This paper introduces a new, smarter way to add those extra beams ('mHC'). It's like using precisely engineered brackets that add strength without messing up the main support structure, resulting in the tallest, strongest, and most stable tower yet.
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
- 1Proposes a framework to project residual connections onto a manifold to restore identity mapping.
- 2Addresses training instability and scalability issues of Hyper-Connections.
- 3Demonstrates superior scalability and performance improvements of mHC.
Adversarial AI reveals mechanisms and treatments for disorders of consciousness
Imagine your brain is like a city with millions of roads and traffic systems. When you're awake and conscious, traffic flows in complex, coordinated patterns. In a coma, something has gone wrong — but we've never had a great way to figure out exactly which roads are broken or how to fix them. This study built a very smart AI that learned to tell the difference between 'awake brain' and 'coma brain' by studying hundreds of thousands of brainwave recordings. Then, like a detective, the AI was pitted against a simulated model of the brain to figure out: what changes in the brain's wiring would explain the difference? The AI figured out — on its own, without being told — that two key things go wrong in a coma: a specific circuit deep in the brain (called the basal ganglia indirect pathway) gets disrupted, and the brain's 'braking system' (inhibitory neurons) starts working too hard in the wrong places. The researchers then checked these predictions against real patient data, and both checked out. The AI also suggested that zapping a specific deep brain region with high-frequency electrical pulses might help wake people up — and early evidence from human patients supports this idea.
Gene conversion empowers natural selection in a clonal fish species
Unfortunately, the content of this research abstract could not be accessed due to paywall restrictions. Without being able to read the actual findings about gene conversion in clonal fish species, I cannot provide an accurate explanation of what the researchers discovered or why it matters.
Direct detection of an asteroid’s heliocentric deflection: The Didymos system after DART
NASA crashed a spacecraft into an asteroid moon called Dimorphos in 2022, and scientists have now measured that this impact actually nudged the entire asteroid system slightly off its path around the Sun. This is the first time humans have measurably changed how a celestial body orbits the Sun, proving that we can potentially deflect dangerous asteroids heading toward Earth.
The dynamics of AMPA receptors underlies the efficacy of ketamine in treatment resistant patients with depression
Think of your brain as having billions of tiny locks and keys. One particular lock — called the AMPA receptor — sits on brain cells and helps them talk to each other using the chemical glutamate. In people with hard-to-treat depression, this study found that those locks are less plentiful than normal, especially in emotional brain regions. When doctors gave these patients ketamine, it actually changed how many of those locks were available on the cell surface — and the bigger that change was, the better the patient felt. So ketamine isn't just temporarily numbing pain; it appears to be physically restoring a broken communication system in the brain. The scientists confirmed this by using a special brain scan (PET scan) with a radioactive tracer that literally glows where those AMPA receptor locks are located, letting them count them in real time in living people.