Probing Image-Language Transformers for Verb Understanding

Grounding language to imaginative and prescient is a elementary downside for a lot of real-world AI methods corresponding to retrieving photographs or producing descriptions for the visually impaired. Success on these duties requires fashions to narrate totally different elements of language corresponding to objects and verbs to pictures. For instance, to differentiate between the 2 photographs within the center column beneath, fashions should differentiate between the verbs “catch” and “kick.” Verb understanding is especially troublesome because it requires not solely recognising objects, but additionally how totally different objects in a picture relate to one another. To beat this problem, we introduce the SVO-Probes dataset and use it to probe language and imaginative and prescient fashions for verb understanding.

Specifically, we think about multimodal transformer fashions (e.g., Lu et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Li et al., 2020), which have proven success on a wide range of language and imaginative and prescient duties. Nevertheless, regardless of sturdy efficiency on benchmarks, it’s not clear if these fashions have fine-grained multimodal understanding. Specifically, prior work reveals that language and imaginative and prescient fashions can succeed at benchmarks with out multimodal understanding: for instance, answering questions on photographs primarily based solely on language priors (Agrawal et al., 2018) or “hallucinating” objects that aren’t within the picture when captioning photographs (Rohrbach et al., 2018). To anticipate mannequin limitations, work like Shekhar et al. suggest specialised evaluations to probe fashions systematically for language understanding. Nevertheless, prior probe units are restricted within the variety of objects and verbs. We developed SVO-Probes to raised consider potential limitations in verb understanding in present fashions.

SVO-Probes consists of 48,000 image-sentence pairs and exams understanding for greater than 400 verbs. Every sentence might be damaged right into a triplet (or SVO triplet) and paired with optimistic and adverse instance photographs. The adverse examples differ in just one means: the Topic, Verb, or Object is modified. The determine above reveals adverse examples during which the topic (left), verb (center), or object (proper) doesn’t match the picture. This process formulation makes it attainable to isolate which elements of the sentence a mannequin has probably the most bother with. It additionally makes SVO-Probes more difficult than customary picture retrieval duties, the place adverse examples are sometimes fully unrelated to the question sentence.

To create SVO-Probes, we question a picture search with SVO triplets from a typical coaching dataset, Conceptual Captions (Sharma et al. 2018). As a result of picture search might be noisy, a preliminary annotation step filters the retrieved photographs to make sure we’ve a clear set of image-SVO pairs. Since transformers are skilled on image-sentence pairs, not image-SVO pairs, we want image-sentence pairs to probe our mannequin. To gather sentences which describe every picture, annotators write a brief sentence for every picture that features the SVO triplet. For instance, given the SVO triplet , an annotator may write the sentence “An animal lays within the grass.” We then use the SVO annotations to pair every sentence with a adverse picture, and ask annotators to confirm negatives in a last annotation step. See the determine beneath for particulars.

We look at whether or not multimodal transformers can precisely classify examples as optimistic or adverse. The bar chart beneath illustrates our outcomes. Our dataset is difficult: our customary multimodal transformer mannequin achieves 64.3% accuracy total (likelihood is 50%). Whereas accuracy is 67.0% and 73.4% on topics and objects respectively, efficiency falls to 60.8% on verbs. This outcome reveals that verb recognition is certainly difficult for imaginative and prescient and language fashions.

We additionally discover which mannequin architectures carry out finest on our dataset. Surprisingly, fashions with weaker picture modeling carry out higher than the usual transformer mannequin. One speculation is that our customary mannequin (with stronger picture modeling potential) overfits the prepare set. As each these fashions carry out worse on different language and imaginative and prescient duties, our focused probe process illuminates mannequin weaknesses that aren’t noticed on different benchmarks.

General, we discover that regardless of spectacular efficiency on benchmarks, multimodal transformers nonetheless wrestle with fine-grained understanding, particularly fine-grained verb understanding. We hope SVO-Probes may also help drive exploration of verb understanding in language and imaginative and prescient fashions and encourage extra focused probe datasets.

‍

Go to our SVO-Probes benchmark and fashions on GitHub: benchmark and fashions.