GazeSpeak & Microsoft’s ongoing efforts to support people with Motor Neuron Disease (ALS)

Last Friday (February 17th) New Scientist published an article about a new app in development at Microsoft called GazeSpeak. Due to be released over the coming months on iOS, GazeSpeak aims at facilitating communication between a person with MND (known as ALS in the US, I will use both terms interchangeably) and another individual, perhaps their partner, carer or friend. Developed by Microsoft intern, Xiaoyi Zhang, GazeSpeak differs from traditional approaches in a number of ways. Before getting into the details however it’s worth looking at the background, GazeSpeaker didn’t come from nowhere, it’s actually one of the products of some heavyweight research into Augmentative and Alternative Communication (AAC) that has been taking place at Microsoft over the last few years. Since 2013, inspired by football legend and ALS sufferer Steve Gleason (read more here) Microsoft researchers and developers have put the weight of their considerable collective intellect to bear on the subject of increasing the ease and efficiency of communication for people with MND.

Last year Microsoft Research published a paper called ”
AACrobat: Using Mobile Devices to Lower Communication Barriers and Provide Autonomy with Gaze-Based AAC” (abstract and pdf download at previous link) which proposed a companion app to allow an AAC user’s communication partner assist (in an non-intrusive way) in the communication process. Take a look at the video below for a more detailed explanation.

This is an entirely new approach to increasing the efficiency of AAC and one that I suggest, could only have come from a large mainstream tech organisation who have over thirty years experience facilitating communication and collaboration.

Another Microsoft research paper published last year (with some of the same authors at the previous paper) called “Exploring the Design Space of AAC Awareness Displays” looks at importance of a communication partners “awareness of the subtle, social, and contextual cues that are necessary for people to naturally communicate in person”. There research focused on creating a display that would allow the person with ALS express things like humor, frustration, affection etc, emotions difficult to express with text alone. Yes they proposed the use of Emoji, which are a proven and effective way a similar difficulty is overcome in remote or non face to face interactions however they went much further and also looked at solutions like Avatars, Skins and even coloured LED arrays. This, like the other one above, is an academic paper and as such not an easy read but the ideas and solutions being proposed by these researchers are practical and will hopefully be filtering through to end users of future AAC solutions.

That brings us back to GazeSpeak, the first fruits of the Microsoft/Steve Gleason partnership to reach the general public. Like the AACrobat solution outlined above GazeSpeak gives the communication partner a tool rather than focusing on tech for the person with MND. As the image below illustrates the communication partner would have GazeSpeak installed on their phone and with the app running they would hold their device up to the person with MND as if they were photographing them. They suggest a sticker with four grids of letters is placed on the back of the smart phone facing the speaker. The app then tracks the persons eyes: up, down, left or right, each direction means the letter they are selecting is contained in the grid in that direction (see photo below).

man looking right, other person holding smartphone up with gazespeak installed

Similar to how the old T9 predictive text worked, GazeSpeak selects the appropriate letter from each group and predicts the word based on the most common English words. So the app is using AI in the form of machine vision to track the eyes and also to make the word prediction. In the New Scientist  article they mention that the user would be able to add their own commonly used words and people/place names which one assumes would prioritize them within the prediction list. In the future perhaps some capacity for learning could be added to further increase efficiency. After using this system for a while the speaker may not even need to see the sticker with letters, they could write words from muscle memory. At this stage a simple QR code leading to the app download would allow them to communicate with complete strangers using just their eyes and no personal technology.

Highlights from BETT

The BETT show

Bett (the British Educational Training and Technology show) was held over 4 days last week in London. This is one of the world leading exhibition in relation to technology for primary through to third level education.
A good amount of stands were focused on IT support for schools, cloud storage, and the IT curriculum for schools, and there were also a number of talks and presentations taking place, but due to time constraints, I was unable to make any of those.

Below are some of the highlights that I can across during the day:
Literacy –

  • Clicker have just brought out SuperKeys as an iPad onscreen keyboard. This is a two hit keyboard with word prediction and looks really nice. More info here.
  • Claro now have a PDF annotator app with text to speech. Also Claro MagX – an app for magnification and zooming.
  • IntoWords – text to speech and word prediction for android, iOS, Chrome and Mac.
  • Engaging Eyes – Not eyegaze, but works on improving visual tracking for those experiencing difficulty with reading.
  • Text Reader – a reading pen that can capture 80 letters per second to read back. Can add headphones.
  • iOS keyboards from Assistiveware – Keedogo and Keedogo plus are high viz onscreen keyboards, with word prediction in the plus version. Keeble has customisable keyboards, that can be switch accessible, but it’s the first one (that I’ve seen) that allows for select on release or select on time out, so handy for those individuals who use the touchscreen to stabilise their fingers.

AAC

  • Clicker Com, an AAC app, is due out shortly (March?). Many have used Clicker as an AAC system in the past to good effect, so having a dedicated app is a plus.
  • SymbolKommunikation – scheduling apps that can be modified remotely.
  • Widgit now have an app for iPad – can be used as a voice output device, or to create symbol based documents includes all the Widgit symbols, plus ability to add your own. £54stg.
  • Proloquo4text – from Assistive Ware – similar to Predictable, a text based iOS app. Has Quick talk, saved phrases, and the ability to auto detect language changes.
  • Widgit are bringing out a language comprehension screening tool later this year for ages 2-5 yrs.

Voice Recognition

  • Dragon now doing a school site licence for £900. – unlimited profiles

Mindmapping

  • Matchware – MindView 6 will be fully compatible with Dragon 13 voice commands. Due out in March, but they are going to forward me a trial.
  • EssayWriter – for Mac and Windows – gives a split screen so that the mindmap and document are visible at the same time. 30 day trial at www.fasteressays.com

Coding

  • Smart Robot Coding School from SK Telecom – Albert, a smart little robot will work from commands you give him. Can be used to practice coding, numeracy skills, play games etc.
  • Wishtrac – provides a vast amount of PC peripherals, but has a really good computing curriculum and kits (now required in UK, but could be useful to summer camps here). Based on the Raspberry PI

Music

  • Magix – provides a simplified movie editing, web design and graphic editor. Also has a music maker, that appears to work by dragging and dropping samples, but you can also add your own, free 30 day trial – education.magix.com

Maths

  • Daydreameducation.co.uk – educational posters and some maths drill apps

Vision

  • Kurtweil have a nice iOS app for those with visual difficulties/dyslexia – KNFB reader.