Search papers, labs, and topics across Lattice.
The paper introduces Du-FreqNet, a diffusion model enhanced with dual ControlNet branches and frequency-domain loss, to generate realistic microscopy images of optical microrobots with depth-dependent optical characteristics. This addresses the scarcity of high-quality microscopy datasets needed for training 3D perception systems for autonomous microrobot operation. By incorporating depth-specific mesh layers and adaptive frequency-domain loss, Du-FreqNet achieves a 20.7% improvement in SSIM over baseline methods and demonstrates effective generalization to unseen poses, ultimately enhancing downstream tasks like 3D pose and depth estimation.
Synthesizing realistic microscopy images of microrobots, complete with depth-dependent optical effects, is now possible even with limited training data, thanks to a frequency-aware diffusion model.
Optical microrobots actuated by optical tweezers (OT) are important for cell manipulation and microscale assembly, but their autonomous operation depends on accurate 3D perception. Developing such perception systems is challenging because large-scale, high-quality microscopy datasets are scarce, owing to complex fabrication processes and labor-intensive annotation. Although generative AI offers a promising route for data augmentation, existing generative adversarial network (GAN)-based methods struggle to reproduce key optical characteristics, particularly depth-dependent diffraction and defocus effects. To address this limitation, we propose Du-FreqNet, a dual-control, frequency-aware diffusion model for physically consistent microscopy image synthesis. The framework features two independent ControlNet branches to encode microrobot 3D point clouds and depth-specific mesh layers, respectively. We introduce an adaptive frequency-domain loss that dynamically reweights high- and low-frequency components based on the distance to the focal plane. By leveraging differentiable FFT-based supervision, Du-FreqNet captures physically meaningful frequency distributions often missed by pixel-space methods. Trained on a limited dataset (e.g., 80 images per pose), our model achieves controllable, depth-dependent image synthesis, improving SSIM by 20.7% over baselines. Extensive experiments demonstrate that Du-FreqNet generalizes effectively to unseen poses and significantly enhances downstream tasks, including 3D pose and depth estimation, thereby facilitating robust closed-loop control in microrobotic systems.
