Search papers, labs, and topics across Lattice.
The paper introduces ERASE, a novel scheduling approach that optimizes the training of modern recommendation systems by overlapping forward and backward passes through early backpropagation. By detaching subgraphs to eliminate downstream gradient dependencies, the method allows for concurrent execution on separate CUDA streams, effectively utilizing accelerator capacity. The results show a significant improvement in training throughput of up to 9.51% on a large-scale click-through-rate model while maintaining performance stability.
Early backpropagation can boost training throughput by over 9% without sacrificing model performance.
Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph's backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to $9.51\%$ while keeping normalized entropy close to the baseline.
