Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of synthetic data generated in NVIDIA Isaac Sim for training fruit detection models, specifically YOLO variants, under limited real-world data scenarios. They compared real-only, synthetic-only, and hybrid training approaches, testing on both in-domain and domain-shifted real-world datasets. Results indicate that hybrid training, combining synthetic and real data, achieves performance close to real-only training and improves robustness under domain shift, while also enabling successful real-time deployment on a Jetson Orin NX.
Synthetic data, often touted as a panacea, only shines for fruit detection when paired with real-world data, offering a practical path to reducing annotation effort without sacrificing too much accuracy.
This study investigates the effectiveness of synthetic data for sim-to-real transfer in object detection under constrained data conditions and embedded deployment requirements. Synthetic datasets were generated in NVIDIA Isaac Sim and combined with limited real-world fruit images to train YOLO-based detection models under real-only, synthetic-only, and hybrid regimes. Performance was evaluated on two test datasets: an in-domain dataset with conditions matching the training data and a domain shift dataset containing real fruit and different background conditions. Results show that models trained exclusively on real data achieve the highest accuracy, while synthetic-only models exhibit reduced performance due to a domain gap. Hybrid training strategies significantly improve performance compared to synthetic-only approaches and achieve results close to real-only training while reducing the need for manual annotation. Under domain shift conditions, all models show performance degradation, with hybrid models providing improved robustness. The trained models were successfully deployed on a Jetson Orin NX using TensorRT optimization, achieving real-time inference performance. The findings highlight that synthetic data is most effective when used in combination with real data and that deployment constraints must be considered alongside detection accuracy.
