Apple's Screen Recognition, a computer vision system in iOS14 trained on images of apps in use, labels the UI elements on screen for users who are blind
Apple has always gone out of its way to build features for users with disabilities, and VoiceOver on iOS is an invaluable tool for anyone … Tweets: @vick08 , @edapthere , and @sommer Tweets: @vick08 : Great article from Tech Crunch on screen recognition feature of VoiceOver! iPhones can now automatically recognize and label buttons and UI features for blind users | TechCrunch https://techcrunch.com/... Edapt / @edapthere : iPhones can now automatically recognize and label buttons and UI features for blind users Follow us for regular updates Install our app: https://onelink.to/6vym2s Visit:https://edapt.me/ #Apple #iphonese2020 #blind #edapt #edaptschool #edaptlearningapp https://twitter.com/... Sommer Panage / @sommer : Awesome read about one of Apple's newest #accessibility features, Screen Recognition. It even features chats with a few of our own #a11y leaders here at Apple :-) https://techcrunch.com/...
Context & Ripple Effects
Screen Recognition is one stop in a long-running Apple accessibility arc: a year earlier, the company's head of global accessibility walked through Voice Control, Hover Text, and pointer support as the then-frontier of hands-free and low-vision use. What changes here is that the phone stops relying on developers having labeled their buttons — an on-device vision model trained on images of apps in use fills the gaps itself.
First-order effects
- Blind VoiceOver users can now navigate apps whose developers never added accessibility labels, because Screen Recognition recognizes and names buttons and UI elements directly from the screen.
Second-order effects
- App developers lose part of their excuse for shipping unlabeled interfaces — the OS now papers over bad accessibility hygiene, shifting the competitive baseline for what counts as an accessible app.
Third-order effects
- Interpreting the screen with on-device models becomes Apple's template: the same approach resurfaces in cognitive, vision, and speech accessibility features in 2023 and again in Apple Intelligence-powered accessibility features, moving from labeling elements to generating subtitles and voice on device.
The trend: Apple's accessibility tooling is steadily shifting from manually built aids toward on-device AI that interprets the interface itself, with each year's release building on the last.