Look to Speak App for Android – controlled by eye movements

Look to Speak is a new Android App, produced by Google Creative Lab as part of their Start with One, Invent for Many projects, whereby an experiment that started by focusing on a single person, could be expanded to be relevant to more.

Free to download and customisable, this app aims to give a limited phrased based communication system to those who need assistance in supplementing their speech. Designed with Speech and Language Therapist, Richard Cave, the idea behind this project was to use the Google platform and machine learning to give an option to people to use their eyes to select a pre-stored phrase or sentence.

Look to speak app screen shot, containing a list of phrases in two columns

Although described as an eyegaze access method, it is not what we associate with true eyegaze – the user does not look at the phrase directly for selection. Instead, an elimination system is used, where the inbuilt smartphone or tablet camera is used to interpret the user looking to the left and right in an exaggerated motion to gradually narrow down the range of options available until only one remains. This is then spoken aloud using the Text to Speech settings of the smart phone or tablet.

In looking left and right to make selections, the user is not focusing on the two columns of options displayed on the screen. Instead, they need to focus off screen, a larger eye movement. Sensitivity of the selection method can be adjusted in two parameters. The distance to look off screen can be set to one of three predetermined lengths (small, medium or large) in order to trigger the selection of the contents of that column. The duration of the movement before selection can also be adjusted (quick, medium or long). A third upwards eye movement function changes between cancelling a previous selection or engaging snooze mode. Sound feedback on when a successful selection has been made can be turned on and off.

To disengage from snooze mode, a sequence of one, two or three movements must be carried out, which again can be set in the settings menu.

The settings menu cannot be accessed using eye movements, so assistance will be required in setting up the app. However, a handy Setup Helper is included, which helps position the device for best use (slightly below eye level is recommended) along with a practice screen to try out settings.

When using two eyes, shorter distance and duration settings and positioned just below eye level, the app is very responsive. With one eye closed, for me, the app was still usable, but it took longer for selections to be recognised. It appears to work for individuals wearing glasses. It does take effort to move eyes sufficiently for the app to recognise the movement, so fatigue or eye strain may be an issue for some users.

Another thing to note is that the eyes must return to look at the screen in between selections to the same side. This means, for someone who might have memorised the sequence of eye movements required for a certain phrase, it is not possible to continue to look to one side, rather two distinct eye movements, returning to centre in between, are required.

The selection method can take some getting used to, particularly for users who might be familiar with eyegaze on Windows based devices, as it seems counterintuitive to look off screen.

It is worth noting that the touch screen remains in operation with this app, and a direct selection is also possible.

The vocabulary on screen can be personalised. Although 16 phrases are pre programmed, these can be edited, and a total of 20 phrases included. Phrases can be up to 40 characters in length.  Editing is simple and straight forward, and done by accessing the phrasebook under the setting menu (again assistance will be required to set this up).

Phrases can be selected in four or five movements. Once the message has been selected, it is flashed onscreen in full screen mode.

The app does not require internet access to work, all content is stored locally on the device.

Overall, this is a very promising app. While not a fully-fledged AAC system, it does provide a neat backup option for those who require support for their speech. It may also be beneficial to those who need a limited number of phrases in certain situations, i.e. when communicating with unfamiliar others or in a noisy environment. It may likewise be useful to store command phrases for voice assistants, if speech is not clear enough, i.e. “Ok Google, WhatsApp video call John”.

The addition of a keyboard to create unique phrases would be a major benefit. If there was the possibility to integrate the messages created with other apps, such a text messaging or email, the use of the eye movement as an access method not just for AAC could be maximised.

Look to Speak: Helping accessibility users communicate – YouTube
Getting started with Look to Speak – YouTube

Leave a Reply

Your email address will not be published. Required fields are marked *