Facebook’s New Tool Describes Photos To The Blind; Available Now On iOS App

As more and more photos get uploaded on to Facebook to make social networking a visually immersive experience, it poses a challenge for the visually impaired. If they are using a screen reader software on their smartphones all they get is the name of the friend who has posted the photo and a narration saying that there is a photo in the placeholder. All that is about to change with Facebook’s updated iOS app where with the use of deep convolutional neural network engine the app can generate a description of the photo and convey it to the listener. Facebook has launched the world first automatic alternative text that generates the description of the photo using object recognition technology.


Facebook’s initiative of building this tool began with a recent study that they conducted with the help of Cornell University researchers that showed that blind people were interested in interacting with visual content on social media but were feeling left out because there were no tools to describe photos. Facebook’s technical team set out to build this tool where descriptions would be generated as alt text; you know the HTML tag that describes the photo so that it would be picked up by any screen reader software. Facebook spent 10 months in building this tool so that it would be able to recognise not just what was happening in the foreground but also what was going on in the background.


Facebook relied on its computer vision platform that provides a visual recognition engine that recognises elements in the photo. It then fed learnable parameters to the deep convolutional neural network to learn to decipher new elements in the photos. It used a list of 100 concepts that were most prominent in all Facebook photos such as people’s appearance, nature, transportation, food, objects etc to generate the alt text. Since the image recognition is not without its flaws, the alt text generated says before every description “image may contain”.

Currently the feature is available on the iOS app in English for people in the U.S., U.K., Canada, Australia, and New Zealand. The feature will be rolled out to other platforms and other languages soon.

Source: #-Link-Snipped-#, #-Link-Snipped-# & #-Link-Snipped-#