Google adds Gemini screen Q&A and expressive captions to Android

Screen readers have historically had a frustrating blind spot: they can tell you there is an image of a guitar, but if you want to know what brand it is or what finish it has, you are out of luck.

In honor of Global Accessibility Awareness Day, Google rolled out a batch of AI-driven updates across Android, Chrome, and its research projects, putting its Gemini models to work in places where detail matters most.

TalkBack turns into a full conversation

Last year Google wired Gemini into TalkBack, Android's built-in screen reader, to generate AI descriptions for images that lacked alt text.

The tool is now conversational. When a friend texts a photo of their new gear, a low-vision user can ask follow-up questions to find out the make, color, or other objects in the frame.

Google also expanded this to cover the entire screen. If you are browsing a shopping app, you can ask Gemini about the fabric of a jacket or check if there is an active discount on the page.

Captions are getting more nuance too. Expressive Captions on Android can now reflect elongated speech patterns, spelling out prolonged words when a sports announcer shouts or a friend drags out a response, alongside tags for sounds like whistling or throat-clearing. For now, this is limited to English on Android 15 devices across the US, UK, Canada, and Australia.

Tackling scanned PDFs and non-standard speech

Google is also patching several persistent pain points across its broader software ecosystem.

On the desktop side, Chrome now features automatic optical character recognition for scanned PDFs. Screen readers previously ran into a brick wall with image-only documents; the browser will now recognize text so users can listen, highlight, copy, and search as usual.

For students, the College Board’s Bluebook testing app used for SAT and AP exams on Chromebooks now integrates seamlessly with Google's native accessibility stack, including the ChromeVox screen reader and Dictation tools.

Meanwhile, Project Euphonia, which helps people with non-standard speech, has opened its code repositories on GitHub. Google is also backing a new initiative with University College London to build speech recognition datasets for 10 African languages.

Why it matters

We think this is one of the smartest ways to deploy multimodal models: embedding them invisibly into system-level accessibility rather than forcing people into a standalone chatbot app.

For low-vision users, being able to interrogate a live screen changes accessibility from passive readout into active exploration. Whether you are sorting through a confusing checkout page or examining an untagged photo, follow-up Q&A removes an enormous amount of friction.

The clear risk, however, comes down to hallucination. Multimodal models still misread product details and text layouts, and for someone relying entirely on the AI to navigate a purchase or parse critical paperwork, a confident error can have real-world consequences.

Across the industry, this raises the bar for Apple and Microsoft. Accessibility is rapidly evolving past basic text-to-speech engines into full-screen semantic reasoning, turning assistive tech into the ultimate real-world stress test for multimodal AI.