Search papers, labs, and topics across Lattice.
This paper addresses the challenge of unsupervised temporal action segmentation in long, untrimmed videos by introducing a probabilistic embedding approach that models frame representations as Gaussian distributions. By sampling from these distributions before estimating pseudo labels, the method mitigates the limitations of deterministic embeddings, which often lead to local optima during training. The proposed approach achieves significant improvements over state-of-the-art methods, with up to 20.7% enhancement in Mean of F1 (MoF) and 19.0% in F1-score across various datasets.
Probabilistic embeddings can boost action segmentation performance by over 20%, overcoming the local optimum pitfalls of deterministic methods.
This paper concerns the problem of unsupervised temporal action segmentation for long, untrimmed videos. Recent successful approaches follow a joint representation learning and clustering paradigm, where optimal transport (OT) is adopted to produce pseudo labels for learning frame representations. These approaches alternate between estimating pseudo labels using OT and optimizing the parameters with gradient descent during training, where OT is used for obtaining the final temporal action segmentation. A major limitation of these works is that they learn a deterministic embedding for frame representations. The iterative procedure between learning deterministic embeddings based on pseudo labels and estimating pseudo labels from the learned embedding can thus get quickly stuck in a local optimum. As an alternative, we thus propose to learn a probabilistic embedding for frame representations. The embeddings are modeled by Gaussian distributions and we sample from the distributions before estimating the pseudo labels. We evaluate our approach on several challenging temporal action segmentation datasets and achieve results comparable to, and in some cases, better than the state of the art. Compared to baselines with deterministic embeddings, our approach improves MoF up to 20.7\% and F1-score up to 19.0\%. Our code is available at https://github.com/derkbreeze/PEOT.
