We explore the limits of what vision-language models get about language in our Oral Paper at ICLR 2023
At the same time, vision-language models are also not perfect. Here, we explore the limits of these models, highlighting where and why they might fail. This blog post is a short/high-level description of our recent paper that will be presented as an .
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2023). When and why vision-language models behave like bags-of-words, and what to do about it?. ICLR.
Introduction
What’s a vision-language model?
Vision-Language models have revolutionized the field by leveraging the synergy between visual and linguistic data to perform various tasks. While many vision-language models have been introduced in the literature, CLIP is the most well-known and widely adopted model.
Through the embedding of images and captions in the same vector space, CLIP allows cross-modal reasoning, enabling users to perform tasks such as zero-shot image classification, and text-to-image retrieval with good accuracy. CLIP uses a contrastive learning approach to learn embeddings for images and captions.
A short introduction to contrastive learning
Contrastive learning makes it so that CLIP can learn to associate images with their corresponding captions by minimizing the distance between them in a shared vector space. This approach has proven highly effective, as demonstrated by the impressive results achieved by CLIP and other contrastive-based models.
Contrastive loss is used to compare pairs of images and captions in a batch and optimizes the model to maximize the similarity between embeddings of matching image-text pairs and to decrease the similarity between other pairs in the batch.
An example of a possible batch and training step is seen in the image below:
- Purple squares contain embeddings for all the captions, and green squares contain embeddings for all the images.
- The squares of the matrix contain the dot product (read as “cosine similarity”, since the embeddings are normalized) of all the image embeddings in the batch and all the text embeddings.
- Blue squares contain the dot product between the pairs for which the model has to maximize the similarity, the other white squares are similarities we want to minimize (because each one of those squares contains the similarity of a non-matching image-text pair e.g., the image of a cat and the description “my vintage chair”).
Farewell
Thanks for reading! I hope this was interesting. Vision-Language models can already do many things and we can’t wait to see what future models, such as GPT4 can do!
Acknowledgements
Thanks to
I have also fine-tuned CLIP on fashion data. Here’s a blog post you might be interested in!
was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
SOCIAL SHARE CARD GENERATOR