Make-A-Video: Text-to-Video Generation without Text-Video Data

me@ryanorban.com (Ryan Orban) — Thu, 29 Sep 2022 00:00:00 +0000

Summary

Uriel Singer, Adam Polyak, and the Meta AI team present Make-A-Video — a system for generating short video clips from text prompts, trained without paired text-video data. The key insight is a clever decomposition of the problem: the model learns what the world looks like from large-scale text-image pairs (where paired data is abundant), then separately learns how things move from unlabeled video footage. This avoids the fundamental data bottleneck that had stalled text-to-video generation research: video paired with captions is expensive to collect at scale, but raw video and image-caption datasets exist in enormous quantities.

Diffusion on Ryan Orban

Make-A-Video: Text-to-Video Generation without Text-Video Data

Summary