Text-to-Image Diffusion Models are Zero Shot Classifiers
Dec 10, 2023
Speakers
Kevin Clark
Speaker · 0 followers
Priyank Jaini
Speaker · 0 followers
About
The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data.However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks.We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers.The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a proxy for that label's likelihood.We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge and comparing them with CLIP's zero-shot abilities. They perform competitively with CLIP on a wide range of zero-shot image classification datasets. Additionally, they achieve state-of-the-art results on shape/texture bias tests and can successfully perform attribute binding while CLIP cannot.Although generative pre-training is prevalent in NLP, visual foundation models often use other methods such as contrastive learning. Based on our findings, we argue that generative pre-training should be explored as a compelling alternative for vision and vision-language problems.The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data.However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks.We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers.The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a prox…
Organizer
NeurIPS 2023
Account · 617 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
SPACE: Single-round Participant Amalgamation for Contribution Evaluation in Federated Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Human-Guided Complexity-Controlled Abstractions
Watch later
Andi Peng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Opening Remarks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Prediction of Single Cell Perturbation with Deep Tensor Factorization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Mirror Diffusion Models for Constrained and Watermarked Generation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Google Research Opportunities for Academics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%