Apple brings AI to accessibility — and eye-tracking wheelchair controls to Vision Pro

While most AI labs are busy chasing automated office tasks and chatbot companions, Apple just showed what frontier models can do when applied to sheer survival and independence.

Apple previewed a massive suite of accessibility updates powered by Apple Intelligence, scheduled to roll out across its ecosystem later this year. The standout features range from driving a motorized wheelchair purely with your eyes on Apple Vision Pro to using conversational language to click around your iPhone screen.

Speak what you see, and ditch the memorized commands

For anyone with motor limitations, Voice Control used to be a frustrating game of memorizing numbered grids or praying that app developers properly labeled their buttons.

With Apple Intelligence, Voice Control finally understands natural language. Instead of robotic commands, users can simply say what they see on screen. Telling your device to "tap the guide about best restaurants" or "open the purple folder" actually works, even inside visually complex apps like Apple Maps and Files where buttons lack accessibility labels.

Vision accessibility gets a parallel overhaul. In VoiceOver and Magnifier, users can now click the iPhone Action button to point the camera at their surroundings, ask what is in front of them, and ask natural follow-up questions. It can break down scanned bills, medical records, and physical environments step-by-step.

Meanwhile, Accessibility Reader now digests multi-column scientific layouts, tables, and charts, serving on-demand summaries and in-line translations without breaking customized fonts or colors.

Driving wheelchairs with Vision Pro and systemwide captions

For patients with advanced ALS or severe mobility impairments who cannot operate a physical joystick, Apple Vision Pro is turning into an assistive driving platform.

By leveraging the headset's built-in eye-tracking sensors, users can steer compatible power wheelchairs simply by looking where they want to go. Apple says the system handles diverse lighting environments without demanding constant recalibration, launching first in the U.S. with Tolt and LUCI drive systems over Bluetooth or wired connections.

Apple is also solving a persistent headache for the deaf and hard of hearing: uncaptioned media. A new on-device speech model will automatically generate real-time subtitles for uncaptioned videos across iPhone, iPad, Mac, Apple TV, and Vision Pro. Because it runs entirely on-device, personal videos shared by friends stay completely private.

Other notable additions include official support for Sony's modular Access controller across iOS, iPadOS, and macOS, a developer API to patch human sign language interpreters into ongoing FaceTime calls, and motion sickness cues for visionOS.

Why it matters

We think this release is a masterclass in deploying multimodal AI where it actually counts. While benchmarks obsess over code generation and trivia quizzes, applying computer vision to understand unlabeled app buttons solves a foundational usability barrier that software makers have neglected for decades.

It is also one of the sharpest real-world justifications for the Apple Vision Pro yet. Turning a 3,500-dollar spatial computer into a hands-free mobility rig instantly challenges the exorbitantly priced, proprietary eye-gaze hardware historically sold to patients with motor neuron diseases.

That said, the stakes could not be higher. When an AI agent hallucinates in customer service, you get a bad refund; when an eye-tracking system lags while steering a motorized wheelchair, someone gets hurt. Apple's fine print specifically warns against relying on these vision features for high-risk navigation, and making sure these systems never fail in chaotic real-world environments remains the ultimate test.

Still, keeping everything processed locally on-device while dramatically lowering the input friction is proof that AI is at its best when making everyday technology invisible.