Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a serverless MapReduce framework built on Kubernetes with Knative and Apache Kafka for real-time data processing in logistics. The system comprises five loosely coupled services for data ingestion, processing, and aggregation, leveraging Redis for metadata and AWS S3 for persistent storage. Experimental results demonstrate effective scaling with increasing data volume and support for scale-to-zero, on-demand processing.
Serverless architectures can now handle MapReduce workloads with the introduction of a framework that scales to zero and processes data in real-time.
Modern logistics systems tend to generate continuous streams of data from sources such as GPS, IoT sensors, and logistics management systems. The aggregation, processing, and analysis of data have become vital for monitoring operations, optimizing efficiency, and responding quickly to decision making tasks. In this paper, an event-driven MapReduce framework for real-time data processing in logistics environments is presented. This system runs on Kubernetes with Knative and utilizes Apache Kafka as the backbone for communication between the components. This platform is composed of five loosely coupled services that receive, process, and aggregate the incoming data in real-time. Redis is used to preserve workflow metadata, while an AWS S3 service provides persistent storage for the framework. The design is inspired by the MapReduce programming model. It integrates Function-as-a-Service (FaaS) principles with distributed processing techniques that allow configurable scaling based on the workload demands and the underlying hardware. Experimental evaluation shows that the system can scale effectively as the input data volume increases while supporting scale-to-zero, on-demand processing.