Search papers, labs, and topics across Lattice.
GraphLeap decouples graph construction from feature updates in Vision GNNs by using the previous layer's features for graph construction while simultaneously constructing the graph for the next layer, enabling concurrent execution. This reformulation eliminates the sequential dependency between graph construction and message passing, which is the primary bottleneck in ViGs. An FPGA accelerator implementing GraphLeap achieves up to 95.7x speedup over CPU and 8.5x speedup over GPU baselines on isotropic and pyramidal ViG models, demonstrating real-time inference feasibility.
Vision GNNs can achieve near 100x speedups on FPGAs by decoupling graph construction from feature updates, enabling concurrent execution without significant accuracy loss after fine-tuning.
Vision Graph Neural Networks (ViGs) represent an image as a graph of patch tokens, enabling adaptive, feature-driven neighborhoods. Unlike CNNs with fixed grid biases or Vision Transformers with global token interactions, ViGs rely on dynamic graph convolution: at each layer, a feature-dependent graph is built via k-nearest-neighbor (kNN) search on current patch features, followed by message passing. This per-layer graph construction is the main bottleneck, consuming 50--95\% of graph convolution time on CPUs and GPUs, scaling as $O(N^2)$ with the number of patches $N$, and creating a sequential dependency between graph construction and feature updates. We introduce GraphLeap, a simple reformulation that removes this dependency by decoupling graph construction from feature update across layers. GraphLeap performs the feature update at layer $\ell$ using a graph built from the previous layer's features, while simultaneously using the current layer's features to construct the graph for layer $\ell+1$. This one-layer-lookahead graph construction enables concurrent graph construction and message passing. Although using prior-layer features can introduce minor accuracy degradation, lightweight fine-tuning for a few epochs is sufficient to recover the original accuracy. Building on GraphLeap, we present the first end-to-end FPGA accelerator for Vision GNNs. Our streaming, layer-pipelined design overlaps a kNN graph construction engine with a feature update engine, exploits node- and channel-level parallelism, and enables efficient on-chip dataflow without explicit edge-feature materialization. Evaluated on isotropic and pyramidal ViG models on an Alveo U280 FPGA, GraphLeap achieves up to $95.7\times$ speedup over CPU and $8.5\times$ speedup over GPU baselines, demonstrating the feasibility of real-time Vision GNN inference.
