Facebook’s iOS app uses AI and VoiceOver to describe the contents of photos to blind users

At a time when so many Facebook posts comprise a photo and a brief comment, there’s one group of people who get rather left out of the picture: those who are blind and partially sighted. That’s a problem Facebook is fixing, starting from today. The iOS app now uses artificial intelligence to figure out the content of photos, and Apple’s VoiceOver feature to read aloud a description of them.

Automatic alt text, which is coming to iOS today and later to Android and the web, recognizes objects in photos using machine learning […] While still in its early stages, the technology can reliably identify concepts in categories including transportation (“car,” “boat,” “airplane”), nature (“snow,” “ocean,” “sunset”), sports (“basketball court”), and food (“sushi”). The technology can also describe people (“baby,” “smiling,” beard”), and identify a selfie.

Facebook said that there were two approaches it could have taken to the problem, and it chose the one it believed would be most successful …

Matt King, a Facebook engineer who is blind, […] said “We could probably require people when they upload a photo: ‘please describe this for blind people.’ It would drive people nuts — that would never work at scale.” (This is the actual approach Twitter is taking to the problem, though adding descriptions is optional.)

Facebook says that it was conscious of the risks of misidentification, so the algorithm only tags photos with keywords if it is 80% certain – but applies a higher confidence requirement to sensitive topics, such as someone’s race.

King says the company is aware that this is just a start, and that there are still ‘millions’ of things the AI can’t identify, but that it will improve over time and make a huge difference to blind users.

The impact of doing something like this is really telling people who are blind, your ability to participate in the social conversation that’s going on around the world is really important to us. It’s saying as a person, you matter, and we care about you. We want to include everybody — and we’ll do what it takes to include everybody.

VoiceOver is one of a range of accessibility features built into iOS. Apple last year created a catalog of apps that take advantage of the feature, the same year that the American Foundation for the Blind honored Apple for the technology.