Search papers, labs, and topics across Lattice.
This field study evaluates the deployment of two large inference workloads on a Huawei Ascend 910 system, revealing significant engineering challenges when migrating from GPU-based to non-GPU AI accelerators. The research identifies eight critical limitations, including incomplete operator support and numerical faults, which necessitated extensive modifications and operational safeguards to ensure reliable performance. Ultimately, the study quantifies the integration effort and concurrency behavior, providing a valuable reference for teams considering non-GPU solutions for large-model inference.
Migrating large-model inference to non-GPU accelerators like Huawei Ascend reveals eight critical limitations that could derail performance and reliability.
Non-GPU AI accelerators are increasingly adopted as alternatives to general-purpose GPUs for large-model inference, but the real engineering cost of migrating demanding workloads beyond CUDA remains poorly documented. We present a field study of deploying two large inference workloads on a 16-device Huawei Ascend 910 system using CANN and vLLM-Ascend: an LLM-as-a-judge safety and alignment evaluation pipeline based on a W8A8 MoE judge model, DeepSeek-V4-Flash, and a multimodal medical vision--language benchmark based on DeepSeek-V4-Flash-Vision for MMMU and MMMU-Pro. Making these workloads reliable required twelve source-level patches to the vendor inference plugin, disabling several high-throughput features to preserve numerical correctness, and adding operational safeguards for recurring device-level failures. We summarize the main platform limitations in eight categories: incomplete operator and feature support, fragile parallelism, numerical faults in low-level kernels, immature graph compilation, unstable advanced features, limited scalability, weak observability, and ecosystem fragmentation. For each category, we report the symptoms, evidence, and likely causes. We also quantify the integration effort, concurrency behavior, and benchmark quality to show that both workloads were served correctly. Our study provides a reproducible reference for teams evaluating or operating non-GPU accelerators for large-model inference.
