Image captioning as an assistive technology

IBM Research’s Science for Social Good initiative pushes the frontiers of artificial intelligence in service of positive societal impact. Partnering with non-profits and social enterprises, IBM Researchers and student fellows since 2016 have used science and technology to tackle issues including poverty, hunger, health, education, and inequalities of various sorts. For example, one project in partnership with the Literacy Coalition of Central Texas developed technologies to help low-literacy individuals better access the world by converting complex images and text into simpler and more understandable formats.

Working on a similar accessibility problem as part of the initiative, our team recently participated in the 2020 VizWiz Grand Challenge to design and improve systems that make the world more accessible for the blind. Posed with input from the blind, the challenge is focused on building AI systems for captioning images taken by visually impaired individuals.

IBM Research was honored to win the competition by overcoming several challenges that are critical in assistive technology but do not arise in generic image captioning problems. For full details, please check our winning presentation.

Image captioning has witnessed steady progress since 2015, thanks to the introduction of neural caption generators with convolutional and recurrent neural networks. ^{1 2} This progress, however, has been measured on a curated dataset namely MS-COCO. The scarcity of data and contexts in this dataset renders the utility of systems trained on MS-COCO limited as an assistive technology for the visually impaired. This motivated the introduction of Vizwiz Challenges for captioning images taken by people who are blind.

To sum up in its current art, image captioning technologies produce terse and generic descriptive captions. For this to mature and become an assistive technology, we need a paradigm shift towards goal oriented captions; where the caption not only describes faithfully a scene from everyday life, but it also answers specific needs that helps the blind to achieve a particular task. For example, finding the expiration date of a food can or knowing whether the weather is decent from taking a picture from the window. Vizwiz Challenges datasets offer a great opportunity to us and the machine learning community at large, to reflect on accessibility issues and challenges in designing and building an assistive AI for the visually impaired.

In our winning image captioning system, we had to rethink the design of the system to take into account both accessibility and utility perspectives.

Firstly on accessibility, images taken by visually impaired people are captured using phones and may be blurry and flipped in terms of their orientations. Therefore, our machine learning pipelines need to be robust to those conditions and correct the angle of the image, while also providing the blind user a sensible caption despite not having ideal image conditions. To address this, we use a Resnext network that is pretrained on billions of Instagram images that are taken using phones,³ and we use a pretrained network to correct the angles of the images.⁴

Secondly on utility, we augment our system with reading and semantic scene understanding capabilities. Many of the Vizwiz images have text that is crucial to the goal and the task at hand of the blind person. We equip our pipeline with optical character detection and recognition OCR. ^{5, 6} Then, we perform OCR on four orientations of the image and select the orientation that has a majority of sensible words in a dictionary. In order to improve the semantic understanding of the visual scene, we augment our pipeline with object detection and recognition pipelines.⁷

Finally, we fuse visual features, detected texts and objects that are embedded using fasttext with a multimodal transformer.⁸ To ensure that vocabulary words coming from OCR and object detection are used, we incorporate a copy mechanism in the transformer that allows it to choose between copying an out of vocabulary token or predicting an in vocabulary token.⁹ We train our system using cross-entropy pretraining and CIDER training using a technique called Self-Critical sequence training introduced by our team in IBM in 2017.¹⁰

Our work on goal oriented captions is a step towards blind assistive technologies, and it opens the door to many interesting research questions that meet the needs of the visually impaired. It will be interesting to train our system using goal oriented metrics and make the system more interactive in a form of visual dialog and mutual feedback between the AI system and the visually impaired.

IBM researchers involved in the Vizwiz competiton (listed alphabetically): Pierre Dognin , Igor Melnyk , Youssef Mroueh , Inkit Padhi , Mattia Rigotti , Jerret Ross and Yair Schiff .