Beyond Language Modeling: An Exploration of Multimodal Pretraining

In this episode, we discuss Beyond Language Modeling: An Exploration of Multimodal Pretraining by Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie. The paper investigates native multimodal foundation models by training from scratch on diverse visual and language data using the Transfusion framework. Key findings include the effectiveness of Representation Autoencoder for unified visual representation, synergy between vision and language data, emergence of world modeling from unified pretraining, and the role of Mixture-of-Experts in efficient multimodal scaling. The study also reveals a scaling asymmetry with vision requiring more data than language, which MoE architectures can balance to enable truly unified multimodal models.