Search papers, labs, and topics across Lattice.
This study investigates the performance and optimization of a BentoML-based AI inference system for scalable model serving, addressing a critical gap in AI deployment research. By evaluating a pre-trained RoBERTa sentiment analysis model under realistic workload scenarios, the authors establish baseline performance metrics and identify bottlenecks in the inference pipeline. The introduction of optimization strategies at multiple levels of the serving stack leads to significant improvements in efficiency and scalability, demonstrating practical solutions for real-world AI inference challenges.
Optimizing AI inference can boost throughput and reduce latency, revealing strategies that enhance performance under real-world traffic conditions.
AI research often emphasizes model design and algorithmic performance, while deployment and inference remain comparatively underexplored despite being critical for real-world use. This study addresses that gap by investigating the performance and optimization of a BentoML-based AI inference system for scalable model serving developed in collaboration with graphworks.ai. The evaluation first establishes baseline performance under three realistic workload scenarios. To ensure a fair and reproducible assessment, a pre-trained RoBERTa sentiment analysis model is used throughout the experiments. The system is subjected to traffic patterns following gamma and exponential distributions in order to emulate real-world usage conditions, including steady, bursty, and high-intensity workloads. Key performance metrics, such as latency percentiles and throughput, are collected and analyzed to identify bottlenecks in the inference pipeline. Based on the baseline results, optimization strategies are introduced at multiple levels of the serving stack to improve efficiency and scalability. The optimized system is then reevaluated under the same workload conditions, and the results are compared with the baseline using statistical analysis to quantify the impact of the applied improvements. The findings demonstrate practical strategies for achieving efficient and scalable AI inference with BentoML. The study examines how latency and throughput scale under varying workloads, how optimizations at the runtime, service, and deployment levels affect response time, and how deployment in a single-node K3s cluster influences resilience during disruptions.
