Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
13 papers published across 1 lab.
Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations, demonstrating superior computational efficiency and improved model quality.
Xronos is proposed, a collaborative fine-tuning framework for heterogeneous CPU edge devices that uses TP as its execution backbone and combines lightweight profiling with heterogeneity-aware tensor partitioning to reduce the straggler bottleneck.
SIF is introduced, a metric built on the notion of partial-nodes that is independent of historical workload knowledge that matches both theory and production and COMPASS-ABS is proposed, which employs the COMPact-ASSured (COMPASS) algorithm to confine the cluster state within a tight Anchor-Based Space (ABS).
GeoMesh is presented, a synchronous geo-distributed training framework for heterogeneous GPUs that balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting.
AccelPact, a parallel runtime enabling zero-I/O in-memory fault recovery for sharded distributed training, and operates directly on native communicator instances, providing an efficient foundation for resilient deep learning.
Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations, demonstrating superior computational efficiency and improved model quality.
Xronos is proposed, a collaborative fine-tuning framework for heterogeneous CPU edge devices that uses TP as its execution backbone and combines lightweight profiling with heterogeneity-aware tensor partitioning to reduce the straggler bottleneck.
SIF is introduced, a metric built on the notion of partial-nodes that is independent of historical workload knowledge that matches both theory and production and COMPASS-ABS is proposed, which employs the COMPact-ASSured (COMPASS) algorithm to confine the cluster state within a tight Anchor-Based Space (ABS).
GeoMesh is presented, a synchronous geo-distributed training framework for heterogeneous GPUs that balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting.
AccelPact, a parallel runtime enabling zero-I/O in-memory fault recovery for sharded distributed training, and operates directly on native communicator instances, providing an efficient foundation for resilient deep learning.
DeepShare is a scheduler that uses a continuous tenant-assurance signal to coordinate these decisions at runtime to achieve a more advantageous utilization-QoS trade-off than optimizing quotas, scheduling, and resource sharing independently.
An optimized open-source pipeline-parallel runtime is built that co-designs scheduling and parallelism through a JCT-aware scheduling layer and pipeline-integrated multi-token prediction.
The contribution is an experimentally validated integration repair applying an existing checkpoint-alignment principle, confined to one model revision and controlled configuration; it does not establish general determinism, task-quality equivalence, concurrent-serving gains, or capacity beyond GPU memory.
Four parallelism plans are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count.