Rethinking the shape convention of an MLP
Multi-layer perceptrons (MLPs) conventionally follow a narrow-wide-narrow
design where skip connections operate at the input/output dimensions while
processing occurs in expanded hidden spaces. We challenge this convention by
proposing wide-narrow-wide (Hourglass) MLP blocks where skip connections
operate at expanded dimensions while residual computation flows through narrow
bottlenecks. This inversion leverages higher-dimensional spaces for incremental
refinement while maintaining computational efficiency through parameter-matched
designs. Implementing Hourglass MLPs requires an initial projection to lift
input signals to expanded dimensions. We propose that this projection can
remain fixed at random initialization throughout training, enabling efficient
training and inference implementations. We evaluate both architectures on
generative tasks over popular image datasets, characterizing
performance-parameter Pareto frontiers through systematic architectural search.
Results show that Hourglass architectures consistently achieve superior Pareto
frontiers compared to conventional designs. As parameter budgets increase,
optimal Hourglass configurations favor deeper networks with wider skip
connections and narrower bottlenecks-a scaling pattern distinct from
conventional MLPs. Our findings suggest reconsidering skip connection placement
in modern architectures, with potential applications extending to Transformers
and other residual networks.
68df2db5df49fb0df1e03c13
Meng-Hsi Chen
68df2db5df49fb0df1e03c14
Yu-Ang Lee
68df2db5df49fb0df1e03c15
Feng-Ting Liao
claimed_verified
2025-10-03T12:30:06.450Z
643fb7332397d8eef5b844cd
/avatars/e403a19fc13e478d5929c67028230b0e.svg
Feng-Ting Liao
user
FengTing
68df2db5df49fb0df1e03c16
Da-shan Shiu
Rethinking the shape convention of an MLP
Multi-layer perceptrons (MLPs) conventionally follow a narrow-wide-narrow
design where skip connections operate at the input/output dimensions while
processing occurs in expanded hidden spaces. We challenge this convention by
proposing wide-narrow-wide (Hourglass) MLP blocks where skip connections
operate at expanded dimensions while residual computation flows through narrow
bottlenecks. This inversion leverages higher-dimensional spaces for incremental
refinement while maintaining computational efficiency through parameter-matched
designs. Implementing Hourglass MLPs requires an initial projection to lift
input signals to expanded dimensions. We propose that this projection can
remain fixed at random initialization throughout training, enabling efficient
training and inference implementations. We evaluate both architectures on
generative tasks over popular image datasets, characterizing
performance-parameter Pareto frontiers through systematic architectural search.
Results show that Hourglass architectures consistently achieve superior Pareto
frontiers compared to conventional designs. As parameter budgets increase,
optimal Hourglass configurations favor deeper networks with wider skip
connections and narrower bottlenecks-a scaling pattern distinct from
conventional MLPs. Our findings suggest reconsidering skip connection placement
in modern architectures, with potential applications extending to Transformers
and other residual networks.
Hourglass MLP blocks, with skip connections in expanded dimensions and narrow bottlenecks, outperform conventional narrow-wide-narrow MLPs in generative tasks across image datasets.
multi-layer perceptrons MLPs wide-narrow-wide Hourglass skip connections residual computation parameter-matched designs random initialization generative tasks Pareto frontiers architectural search Transformers residual networks