Search papers, labs, and topics across Lattice.
This paper addresses the challenge of recognizing the temporal order of overlapping sounds in human-robot interaction by developing a deep learning framework that utilizes a multi-branch convolutional neural network (CNN). The system processes various audio features, achieving high accuracy rates of 99% in balanced overlaps and 91% under amplitude variation, while also demonstrating real-time feasibility. These findings underscore the potential of deep learning to enhance sound order recognition in complex auditory environments, which is crucial for applications like first responder detection systems.
Achieving 99% accuracy in recognizing overlapping sounds could revolutionize how robots interact in real-world scenarios, especially in emergency situations.
Recognizing the temporal order of overlapping sounds is an underexplored challenge in human-robot interaction (HRI), with direct relevance to applications such as first responder detection systems. This paper presents a deep learning framework for real-time sound order recognition using recordable buzzers that emit distinct non-verbal sounds (cat meows, dog barks, helicopter noises). A multi-branch convolutional neural network (CNN) processes Mel spectrograms, Mel-frequency cepstral coefficients (MFCCs), and short-time Fourier transform (STFT) features, with an attention-based fusion mechanism to emphasize critical temporal cues. Experiments were conducted under same-amplitude, varied-amplitude, and unseen sound conditions. The proposed system achieved 99% accuracy in balanced overlaps, 91% under amplitude variation, and 74% on unseen test data with normalization. These results demonstrate that deep learning can reliably recognize sound order in overlapping conditions, supporting practical HRI scenarios. While experiments were conducted on carefully controlled synthetic overlaps, we additionally report latency benchmarks demonstrating real-time feasibility and provide an extended discussion on generalization, ecological validity, and deployment challenges in real-room environments.
