Search papers, labs, and topics across Lattice.
This paper introduces D-CLOT, a novel approach to unsupervised action segmentation that addresses the issue of representation-prototype inconsistency in existing optimal transport (OT) methods. By incorporating a graph-constrained module and a dual refinement mechanism, D-CLOT re-estimates action prototypes from refined frame embeddings, significantly enhancing segmentation accuracy. The results show that both D-CLOT and its variant D-CLOT$_{B}$ achieve substantial improvements across five benchmarks, including a new state-of-the-art on the challenging Assembly101 dataset, with gains of up to +12.7 F1 and +10.2 mIoU.
Action segmentation accuracy skyrockets with D-CLOT, achieving up to +12.7 F1 by resolving representation-prototype inconsistencies in unsupervised settings.
Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. We identify this \emph{representation--prototype inconsistency} as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions. To address this issue, we build on the recently introduced CLOT, which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry. We study two instantiations that share the same backbone, graph module, and objective: D-CLOT updates the prototypes using $k$-means, whereas D-CLOT$_{B}$ updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to $+12.7$ F1 and $+10.2$ mIoU (YTI) and activity-level gains of up to $+8.9$ F1 (FS-Eval). We further establish the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work. Extensive ablations and sensitivity analyses demonstrate that the two refinement mechanisms are complementary and robust.
