Search papers, labs, and topics across Lattice.
This paper introduces a unified framework for handwriting generation that integrates online and offline methods through a compact physical brush model and a differentiable rendering module. By addressing the challenges of linking stroke dynamics to pixel-level appearance and enabling end-to-end learning, the authors successfully generate realistic handwritten images that maintain both structural integrity and fine-grained textures. Extensive experiments and real-world applications, including robotic calligraphy, demonstrate the framework's effectiveness in achieving high fidelity in both structure and appearance.
Bridging the gap between online trajectory estimation and offline image synthesis, this framework achieves unprecedented realism in handwritten text generation.
Realistic handwritten text generation plays an important role in numerous applications, such as font design, biometric authentication, and robotic calligraphy. Existing methods are typically divided into two independent paradigms: online approaches that estimate handwriting trajectories and offline approaches that synthesize realistic handwriting images. While online models capture structural and temporal dynamics, they often lack fine-grained textures, whereas offline models reproduce realistic appearance but discard stroke order. However, unifying online and offline models remains challenging due to (1) the lack of an explicit physical model linking stroke kinematics to pixel-level appearance and (2) the absence of paired trajectory-image datasets. Moreover, enabling end-to-end learning requires a differentiable rendering process across motion and appearance domains. To address these challenges, we propose a compact physical brush model that bridges stroke dynamics and visual appearance, together with a differentiable rendering module that converts stroke trajectories into stylized images. By integrating these components, we propose a unified online-offline handwriting generation framework via differentiable brush rendering. The proposed framework consists of four core modules: 1) a text-to-stroke generator that predicts the target stroke conditioned on the given text and style image, 2) a brush parameter observer that extracts brush model parameters from style references, 3) a differentiable brush renderer that maps a stroke sequence and physical brush parameters into a handwritten image, and 4) a zero-shot image refiner that refines rendered images via diffusion models. Extensive experiments and real-world robotic calligraphy demonstrations validate our approach, achieving both structural and visual fidelity.
