Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Art-based Reinforcement Training (ART), a novel method for fine-tuning multimodal large language models (MLLMs) by optimizing raw visual inputs instead of modifying the model's internal weights. This approach allows for effective parameter-efficient fine-tuning without altering the computational graphs of precompiled models, making it compatible with high-throughput engines like vLLM. The results demonstrate that ART achieves accuracy comparable to Low-Rank Adaptation (LoRA) across various mathematical and structured tool-use benchmarks, showcasing its potential in enhancing model performance efficiently.
ART fine-tunes multimodal LLMs by optimizing visual inputs, achieving competitive accuracy without altering model architecture.
There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.