Search papers, labs, and topics across Lattice.
This paper introduces DynamicGate-MLP, a novel MLP architecture that combines regularization via learned structural dropout with input-dependent conditional computation. The model learns continuous gate probabilities to generate discrete execution masks, enabling sample-specific computation paths and suppressing unnecessary operations. DynamicGate-MLP is trained with a compute budget penalty and a Straight-Through Estimator (STE) and demonstrates improved compute efficiency compared to MLP baselines and MoE-style variants across several datasets.
DynamicGate-MLP learns to selectively activate MLP units based on the input, achieving better compute efficiency without sacrificing performance.
Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.
