Search papers, labs, and topics across Lattice.
This paper investigates the integration of attention layers in Memory Augmented Autoencoders (MemAE) for unsupervised outlier detection within Federated Learning (FL) frameworks. The authors identify a gap in existing aggregation techniques for MemAE models and propose novel guided aggregation methods that enhance model robustness in non-IID datasets, particularly in scenarios with unbalanced data across edge nodes. The results indicate that these new aggregation strategies significantly improve performance, even for shallow autoencoders, making them viable for deployment in resource-constrained environments.
Novel aggregation techniques for Memory Augmented Autoencoders in Federated Learning can enhance anomaly detection performance, even in shallow models.
Attention layers are the backbone of today's most powerful and impactful models. Models with multi-million and billion parameters rely on contextual knowledge provided by attention layers. However, their use goes well beyond just being the core component of large language models. One particularly interesting application is in Memory Augmented Autoencoders (MemAE), specifically for unsupervised representation learning in outlier detection tasks. It was shown that attention helps these models be more effective in centralized learning scenarios. Our work aims to address the lack of specialized aggregation techniques in Federated Learning (FL) when it comes to MemAE models. In this paper we analyze the intricacies of the architecture behind Memory Augmented Autoencoders, and propose novel, guided approaches to effectively aggregate these models in federated scenarios. We demonstrate our approach on non-IID datasets and show that these novel aggregation schemes are more robust when dealing with numerous edge nodes in environments with unbalanced datasets, specifically for unsupervised anomaly detection scenarios. This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.
