Search papers, labs, and topics across Lattice.
The paper addresses the data scarcity problem in specialized domains by proposing ChimeraLoRA, a method for generating synthetic data using diffusion models fine-tuned with LoRA. ChimeraLoRA separates the LoRA adapter into a class-shared component for class priors and per-image components for fine-grained details, trained with a semantic boosting technique to preserve class bounding boxes. By composing these LoRA components during generation, the method achieves a balance between diversity and detail, leading to improved downstream classification accuracy on few-shot datasets.
Achieve detail-rich and diverse synthetic data from few-shot examples by mixing class-level and image-specific LoRAs, boosting downstream classification accuracy.
Beyond general recognition tasks, specialized domains including privacy-constrained medical applications and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to supplement underrepresented regions of real data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While an image-wise LoRA trained on a single image captures fine-grained details yet offers limited diversity, a class-wise LoRA trained over all shots produces diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into a class-shared LoRA~$A$ for class priors and per-image LoRAs~$\mathcal{B}$ for image-specific characteristics. To expose coherent class semantics in the shared LoRA~$A$, we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose $A$ with a mixture of $\mathcal{B}$ using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images are both diverse and detail-rich while closely aligning with the few-shot real distribution, yielding robust gains in downstream classification accuracy.
