mstar.model.glm5_next.mhc#
GLM-5.3-Flash mHC: manifold-constrained hyper-connections.
Functions
|
Model entry: replicate embeddings |
|
Project a positive |
|
Write a sublayer output back into the streams: place by |
Classes
|
One mHC site: learned collapse/placement/mixing weights from the live streams. |
|
Model exit: collapse streams |
|
RMSNorm with no learned weight, over the last (flattened 4*hidden) axis. |
- class mstar.model.glm5_next.mhc.Glm5NextHyperConnection(hidden_size, hc_mult=GLM5_NEXT_HC_MULT, hc_eps=GLM5_NEXT_HC_EPS, hc_sinkhorn_iters=GLM5_NEXT_HC_SINKHORN_ITERS, rms_norm_eps=GLM5_NEXT_RMS_NORM_EPS)[source]#
Bases:
ModuleOne mHC site: learned collapse/placement/mixing weights from the live streams.
- Parameters:
- forward(hidden_streams)[source]#
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class mstar.model.glm5_next.mhc.Glm5NextHyperHead(*args, **kwargs)[source]#
Bases:
ModuleModel exit: collapse streams
[B, S, H, D] -> [B, S, D].- forward(hidden_streams)[source]#
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class mstar.model.glm5_next.mhc.UnweightedRMSNorm(eps=GLM5_NEXT_RMS_NORM_EPS)[source]#
Bases:
ModuleRMSNorm with no learned weight, over the last (flattened 4*hidden) axis.
- Parameters:
eps (float)
- forward(x)[source]#
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- mstar.model.glm5_next.mhc.expand_streams(hidden_states, hc_mult=GLM5_NEXT_HC_MULT)[source]#
Model entry: replicate embeddings
[B, S, D]into[B, S, hc_mult, D].