Pixel 11 features AI that translates sign language into text

Google’s Pixel 11 introduces a new feature designed to help deaf and hard-of-hearing users communicate, utilising artificial intelligence (AI) to translate sign language into text. The feature was developed by Google’s AI laboratory, DeepMind, and debuted through the latest Gboard and Live Transcribe services available on the Pixel 11. Previously, deaf or hard-of-hearing users might have relied on typing messages to communicate with those who do not understand sign language. However, with the development of the sign-language-to-text (SL2T) AI model, users can now communicate more quickly as their signing is directly translated into text using the Gemini chatbot.

DeepMind explained that the technology processes sign language in a novel way. Rather than interpreting raw video, the SL2T model works with a separate on-device model that converts the video recording into a geometric coordinate framework, which is then sent to Google’s servers. ‘Existing sign language grammars fail to capture the rich, non-linear aspects of sign language, such as non-manual markers and spatial constructions,’ DeepMind stated. ‘Translating directly from geometric coordinate frameworks removes these artificial vocabulary constraints and allows translation quality to improve directly with data.’

The model was trained on over 100,000 hours of multilingual sign language data, with approximately a quarter of that data representing American Sign Language (ASL). Globally, there are more than 70 million deaf and hard-of-hearing people who communicate using around 200 different sign languages. Although the company faced initial challenges, training the SL2T model on multiple languages simultaneously allowed it to learn a shared underlying structure common across different sign languages, laying a crucial foundation for broader application.