Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to semantic and panoptic segmentation by framing it as a language modeling task, where segmentation masks are represented as sequences of discrete tokens derived from Run Length Encoding (RLE). The authors modify the Pix2Seq architecture to autoregressively generate these RLE tokens, incorporating strategies to compress token sequence length for efficient video processing and panoptic segmentation. Experiments on image and video segmentation datasets demonstrate competitive performance despite computational limitations.
Autoregressive language models can now perform semantic and panoptic segmentation by directly predicting run-length encoded mask tokens, offering a unified approach for images and videos.
This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks and then train a modified version of Pix2Seq \cite{p2s} to output these RLE tokens through autoregression. We propose novel tokenization strategies to compress the length of the token sequence to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our proposed models on two datasets to show that they are competitive with the state of the art in spite of being bottlenecked by our limited computational resources.
