GazeSpeak & Microsoft’s ongoing efforts to support people with Motor Neuron Disease (ALS)

Last Friday (February 17th) New Scientist published an article about a new app in development at Microsoft called GazeSpeak. Due to be released over the coming months on iOS, GazeSpeak aims at facilitating communication between a person with MND (known as ALS in the US, I will use both terms interchangeably) and another individual, perhaps their partner, carer or friend. Developed by Microsoft intern, Xiaoyi Zhang, GazeSpeak differs from traditional approaches in a number of ways. Before getting into the details however it’s worth looking at the background, GazeSpeaker didn’t come from nowhere, it’s actually one of the products of some heavyweight research into Augmentative and Alternative Communication (AAC) that has been taking place at Microsoft over the last few years. Since 2013, inspired by football legend and ALS sufferer Steve Gleason (read more here) Microsoft researchers and developers have put the weight of their considerable collective intellect to bear on the subject of increasing the ease and efficiency of communication for people with MND.

Last year Microsoft Research published a paper called ”
AACrobat: Using Mobile Devices to Lower Communication Barriers and Provide Autonomy with Gaze-Based AAC” (abstract and pdf download at previous link) which proposed a companion app to allow an AAC user’s communication partner assist (in an non-intrusive way) in the communication process. Take a look at the video below for a more detailed explanation.

This is an entirely new approach to increasing the efficiency of AAC and one that I suggest, could only have come from a large mainstream tech organisation who have over thirty years experience facilitating communication and collaboration.

Another Microsoft research paper published last year (with some of the same authors at the previous paper) called “Exploring the Design Space of AAC Awareness Displays” looks at importance of a communication partners “awareness of the subtle, social, and contextual cues that are necessary for people to naturally communicate in person”. There research focused on creating a display that would allow the person with ALS express things like humor, frustration, affection etc, emotions difficult to express with text alone. Yes they proposed the use of Emoji, which are a proven and effective way a similar difficulty is overcome in remote or non face to face interactions however they went much further and also looked at solutions like Avatars, Skins and even coloured LED arrays. This, like the other one above, is an academic paper and as such not an easy read but the ideas and solutions being proposed by these researchers are practical and will hopefully be filtering through to end users of future AAC solutions.

That brings us back to GazeSpeak, the first fruits of the Microsoft/Steve Gleason partnership to reach the general public. Like the AACrobat solution outlined above GazeSpeak gives the communication partner a tool rather than focusing on tech for the person with MND. As the image below illustrates the communication partner would have GazeSpeak installed on their phone and with the app running they would hold their device up to the person with MND as if they were photographing them. They suggest a sticker with four grids of letters is placed on the back of the smart phone facing the speaker. The app then tracks the persons eyes: up, down, left or right, each direction means the letter they are selecting is contained in the grid in that direction (see photo below).

man looking right, other person holding smartphone up with gazespeak installed

Similar to how the old T9 predictive text worked, GazeSpeak selects the appropriate letter from each group and predicts the word based on the most common English words. So the app is using AI in the form of machine vision to track the eyes and also to make the word prediction. In the New Scientist  article they mention that the user would be able to add their own commonly used words and people/place names which one assumes would prioritize them within the prediction list. In the future perhaps some capacity for learning could be added to further increase efficiency. After using this system for a while the speaker may not even need to see the sticker with letters, they could write words from muscle memory. At this stage a simple QR code leading to the app download would allow them to communicate with complete strangers using just their eyes and no personal technology.

Microsoft backs Enable Ireland’s digital transformation

Dan Klein, Microsoft Ireland, with Christina McCarthy and Siobhan Long, Enable Ireland

Microsoft has donated a software package to Enable Ireland allowing the charity to move its operations to the cloud through Office 365 for its 1,200 staff and volunteers.

The announcement was made at the graduation ceremony for 17 students of the Foundations in Assistive Technology course, delivered by Enable Ireland and Dublin Institute of Technology (DIT).

Microsoft has supported the Assistive Technology course for 15 years, hosting training sessions at the company’s campus in Sandyford, Co Dublin and hosting fundraisers including an annual funding drive and volunteering to decorate local facilities.

A total of 365 participants have graduated from this course since 2001, including people with disabilities and their families, employers, educators and carers. The course trains people in the use of Assistive Technologies including desktops, tablets, smart phones and smart home devices as well as leisure pursuits such as gaming, music and photography.  The course is aimed at providing a solid foundation in AT for diverse audiences: adults who are AT users, therapists, teachers, IT professionals, families and others.

Siobhan Long, national manager, assistive technology training service, Enable Ireland, said: “Well done to the 17 fantastically talented individuals who have worked so hard and today graduated from the Foundations in Assistive Technology course. We recently conducted an assistive technology survey which showed that users overwhelming appreciated the difference that AT makes to their lives.

This national online Assistive Technology survey was undertaken as part of the research process which informs the recommendations made. A total of 236 Assistive Technology users responded to the survey. The findings dispel the widely-held belief that AT is expensive, with 64% of respondents indicating that they used technology costing less than €1,000. 41% of AT users reported that they had self-funded their own AT. Respondents were extremely positive on the perceived usefulness of their AT equipment with 61% reporting that they couldn’t manage without it. However, nearly 30% of respondents experienced frustration and delays in the process of securing their AT. Waiting times were also highly variable, with 54% reporting that they received their AT in three months. However, 15% had to wait over 6 months and 16% waited in excess of a year.

“For people with disabilities and older people, technology can change the most ordinary of
daily activities from the impossible to the possible. Assistive Technology can support people
to live to their fullest, to participate in education and employment and to live independently
and as part of their communities.” Senator John Dolan, CEO
Disability Federation of Ireland.

Dan Klein, Microsoft Ireland, said: “At Microsoft we believe it is critically important that we think about making technology accessible to all. Our collaboration with Enable Ireland has given our team an important window of insight into people’s needs and prompted new innovation… Enable Ireland does critically important work in moving the focus from the disability to the person – not least through its Assistive Technology course.”

Read more: http://www.techcentral.ie/oh991#ixzz4R1j0eQB9

Foundations in AT

Enable Ireland’s National AT training Service deliver a “Foundations in AT” Course which is accredited by DIT. The objective of the course is to provide participants with the AT knowledge and skills that they require to support AT users. The course duration is 3 days face-to-face training in Microsoft, Dublin and an equivalent of 7 days on-line learning.
Course Dates: 9th February, 8th March, 12th April 2016.
We caught up with a number of graduates at the 2015 graduation in Microsoft and got their view of the course.

FOR FURTHER INFORMATION or to reserve your place, please contact Shirley (sdeakin@enableireland.ie). Ph: (01) 2184100.

oneweek-hackathon

Steve Gleason

Microsoft’s first-ever company wide Hacakthon demonstrates what can be achieved when people work together with a common goal. The project brought together a cross-discipline team of researchers, engineers, designers, program managers and media professionals from across Microsoft. The global Hackathon’s grand prize winner “Ability Eye Gaze” had produced some great results for a former footballer, Steve Gleason now living with ALS (Motor Neuron’s Disease). Part of the team’s accomplishments were to help Gleason “drive” his wheelchair. The team used eyegaze technology with the Surface Pro 3, created a user interface to navigate the wheelchair and devised a way to use a Kinect sensor to safely manoeuvre around objects when detected.

See further details on blog post below

http://blogs.microsoft.com/firehose/2014/07/31/oneweek-hackathon-outcome-as-individual-parts-we-were-good-together-we-were-so-much-better/