LANCE: Stress-testing Visual Models by Generating Language-guided
  Counterfactual Images
We propose an automated algorithm to stress-test a trained visual model by
generating language-guided counterfactual test images (LANCE). Our method
leverages recent progress in large language modeling and text-based image
editing to augment an IID test set with a suite of diverse, realistic, and
challenging test images without altering model weights. We benchmark the
performance of a diverse set of pretrained models on our generated data and
observe significant and consistent performance drops. We further analyze model
sensitivity across different types of edits, and demonstrate its applicability
at surfacing previously unknown class-level model biases in ImageNet.
6476c6606ab485374ba99c97
Viraj Prabhu
claimed_verified
2023-05-31T16:54:26.058Z
63fe4f92c0ec83fda436b6bf
/avatars/1021060e042ce3257444577724aa9d34.svg
Viraj Prabhu
user
virajp
6476c6606ab485374ba99c98
Sriram Yenamandra
claimed_verified
2023-06-01T05:54:27.298Z
640d3ece4b5450a208b275fd
/avatars/669e161448c0895b856065646d68c24c.svg
sriram
user
yvsriram
6476c6606ab485374ba99c99
Prithvijit Chattopadhyay
claimed_verified
2023-05-31T16:53:55.893Z
628d451386d23ad1560882c4
https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
Prithvijit Chattopadhyay
user
prithv1
6476c6606ab485374ba99c9a
Judy Hoffman
claimed_verified
2023-05-31T16:53:38.940Z
6477790c4ae93470ffc297bb
/avatars/f8d5e08719297a335731a117416d6336.svg
Judy Hoffman
user
judyhoffman
LANCE: Stress-testing Visual Models by Generating Language-guided
  Counterfactual Images
We propose an automated algorithm to stress-test a trained visual model by
generating language-guided counterfactual test images (LANCE). Our method
leverages recent progress in large language modeling and text-based image
editing to augment an IID test set with a suite of diverse, realistic, and
challenging test images without altering model weights. We benchmark the
performance of a diverse set of pretrained models on our generated data and
observe significant and consistent performance drops. We further analyze model
sensitivity across different types of edits, and demonstrate its applicability
at surfacing previously unknown class-level model biases in ImageNet.
A new algorithm generates language-guided counterfactual images to stress-test visual models, revealing performance drops and class-level biases.
language-guided counterfactual test images large language modeling text-based image editing pretrained models class-level model biases