Google has introduced SL2T, a sign-language-to-text model trained on over 100,000 hours of data, enabling American Sign Language users to translate gestures into English text on Pixel 11 devices using Gboard and Live Transcribe
Google has begun deploying its sign-language-to-text (SL2T) model on consumer devices, starting with American Sign Language (ASL) translation for Pixel 11 smartphones. The system enables users to convert signed gestures into English text in real time, supporting communication through Gboard and Live Transcribe without the need for manual typing. This marks a transition from research prototypes to a public-facing application, with the initial rollout limited to ASL-to-English translation on a single device line.
Unlike speech recognition, sign-language translation requires interpreting complex, simultaneous movements of the hands, face, and body, each contributing to meaning through a distinct grammar and vocabulary. SL2T addresses this by combining computer vision and language modeling: the system uses Google's MediaPipe Holistic model to extract geometric coordinates-known as pose landmarks-from the user's camera feed. These landmarks represent the positions and movements of the hands, face, and body, and are transmitted to the translation model while the original video is immediately discarded. This approach is intended to reduce privacy risks by minimizing the amount of sensitive visual data processed beyond the device.
Model Training and Evaluation
SL2T was trained on more than 100,000 hours of sign-language video data spanning over 50 sign languages, with approximately 25% of the dataset consisting of ASL. The model does not rely on intermediate glosses-simplified sign labels-but instead translates directly from pose landmarks to text, aiming to capture spatial relationships and non-manual signals such as facial expressions. According to Google, this cross-lingual training strategy was designed to help the model generalize across different sign languages, dialects, and signing styles.
On the FLEURS-ASL benchmark, which measures translation quality for ASL, Google reports a zero-shot BLEURT score of 70. This figure, while developer-reported and not yet independently verified, is described as substantially higher than previous results for similar tasks. The company also states that the model was tested against practical challenges, including latency, left-handed and one-handed signing, and the risk of generating output when no signing is present. One-handed signing is particularly relevant for smartphone users who may need to hold the device while signing.
Deployment, Privacy, and User Involvement
The SL2T system is designed for streaming translation, processing live video input rather than completed recordings. By extracting and transmitting only pose landmarks, Google aims to limit the exposure of raw visual data, addressing privacy concerns associated with camera-based translation. The company reports that Deaf users and organizations were involved throughout development, including in data collection, testing, and evaluation. An advisory committee representing Deaf organizations provided input on system design and deployment priorities.
At launch, SL2T supports ASL-to-English translation on Gboard and Live Transcribe for Pixel 11 devices. Google has indicated plans to expand support to additional devices and sign languages, but no timeline or technical details have been provided. The company also suggests that future applications could include sign-language generation and other assistive technologies, though these remain at the research or planning stage.
Technical and Social Limitations
While SL2T represents a technical advance in sign-language translation, several limitations remain. The system's accuracy and reliability outside controlled benchmarks have not been independently evaluated, and its performance for sign languages other than ASL is untested in consumer settings. The translation process depends on the quality of pose estimation, which may be affected by lighting, camera angle, background, and user movement. The model's ability to handle regional dialects, rapid signing, or non-standard signing styles is not fully characterized. Additionally, the privacy approach-discarding video after extracting pose landmarks-reduces but does not eliminate the risk of sensitive data exposure, especially if pose data could be reconstructed or misused.
Human oversight remains essential, particularly in contexts where translation errors could have significant consequences. The system does not provide explanations or confidence scores for its output, and users must remain alert to possible misinterpretations. As with other machine-learning-based accessibility tools, the effectiveness of SL2T will depend on ongoing evaluation, user feedback, and transparent reporting of failures and limitations.
Sign-language translation by machine learning requires the system to interpret simultaneous, multi-channel signals-hand shapes, movements, facial expressions, and body posture-each contributing to meaning in ways that differ from spoken language. Computer vision models extract pose landmarks from video, which are then mapped to text by a language model trained on large datasets. This process is sensitive to the quality and diversity of training data, the accuracy of pose estimation, and the ability of the model to generalize beyond its training environment. Unlike spoken-language translation, where audio can often be segmented into words, sign-language translation must resolve overlapping signals and spatial relationships, making the task particularly challenging for automated systems.
