23 min listen
Tracking Anything with Decoupled Video Segmentation
Tracking Anything with Decoupled Video Segmentation
ratings:
Length:
34 minutes
Released:
Sep 21, 2023
Format:
Podcast episode
Description
Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. Code is available at: https://hkchengrex.github.io/Tracking-Anything-with-DEVA
2023: Ho Kei Cheng, Seoung Wug Oh, Brian L. Price, Alexander Schwing, Joon-Young Lee
https://arxiv.org/pdf/2309.03903v1.pdf
2023: Ho Kei Cheng, Seoung Wug Oh, Brian L. Price, Alexander Schwing, Joon-Young Lee
https://arxiv.org/pdf/2309.03903v1.pdf
Released:
Sep 21, 2023
Format:
Podcast episode
Titles in the series (100)
LIMA: Less Is More for Alignment: Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preference... by Papers Read on AI