Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ESsEN, a compact vision-language model designed for low-resource settings, demonstrating that two-tower encoder architectures outperform one-tower encoders in discriminative English tasks. By integrating traditional convolutional networks into the two-tower transformer framework, the authors achieve parameter efficiency while maintaining performance across various tasks. The findings highlight the potential for training effective vision-language models with significantly fewer resources, making this technology more accessible to researchers and practitioners alike.
Two-tower encoders can outperform their one-tower counterparts in low-resource vision-language tasks, enabling effective modeling with far fewer parameters.
Vision-language modeling is rapidly increasing in popularity with an ever expanding list of available models. In most cases, these vision-language models have parameters in the tens of billions, which is necessary for some needs, but in many cases smaller models are necessary (e.g., on edge devices or independent robotic platforms). Unfortunately, there is little research in producing light-weight models or in training them with small datasets. Inspired by the language learning progression and data sparsity in child development, in this paper, we address both of these goals in a systematic fashion. We show that two-tower encoder models are superior to one-tower encoders in low-resource settings for discriminative English tasks. We show also that incorporating traditional convolutional networks into the two-tower transformer architecture can help produce parameter efficient vision-language models. Finally, we show that the cross-modal fusion module of two-tower encoders can vary significantly in shape and size while producing the same results. In addition, we present ESsEN, a compact vision-language model that can be trained end-to-end with relatively few resources that performs as well on several tasks with only a fraction of the parameters compared to other models. The experimental results and the tools we present here make vision-language modeling more accessible to a wider variety of researchers.