Contrastive Learning

ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training

Norelli, Fumero, Maiorca, Rodolà et al. (Sapienza University, arXiv:2210.01738, 2022) show that any two unimodal models can be composed into a zero-shot multimodal system by finding approximate shared nearest neighbors across their embedding spaces, given only a small set of coupled pairs. The result challenges the assumption that multimodal capability requires joint training.