Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the "entrapment" phenomenon in decentralized random walk learning, where the model gets stuck in a small network region, hindering convergence. They show that using Metropolis-Hastings for weighted sampling can cause this entrapment. To mitigate this, they propose Metropolis-Hastings with Levy Jumps (MHLJ), which introduces occasional long-range transitions to improve exploration. They provide a convergence rate analysis and empirical results demonstrating that MHLJ effectively eliminates entrapment and accelerates decentralized learning.
Random walk learning can get stuck in "entrapment," but adding Levy jumps unsticks it and speeds up decentralized learning.
We study decentralized learning over networks where data are distributed across nodes without a central coordinator. Random walk learning is a token-based approach in which a single model is propagated across the network and updated at each visited node using local data, thereby incurring low communication and computational overheads. In weighted random-walk learning, the transition matrix is designed to achieve a desired sampling distribution, thereby speeding up convergence under data heterogeneity. We show that implementing weighted sampling via the Metropolis-Hastings algorithm can lead to a previously unexplored phenomenon we term entrapment. The random walk may become trapped in a small region of the network, resulting in highly correlated updates and severely degraded convergence. To address this issue, we propose Metropolis-Hastings with Levy jumps, which introduces occasional long-range transitions to restore exploration while respecting local information constraints. We establish a convergence rate that explicitly characterizes the roles of data heterogeneity, network spectral gap, and jump probability, and demonstrate through experiments that MHLJ effectively eliminates entrapment and significantly speeds up decentralized learning.