GazeSpeak & Microsoft’s ongoing efforts to support people with Motor Neuron Disease (ALS)

Last Friday (February 17th) New Scientist published an article about a new app in development at Microsoft called GazeSpeak. Due to be released over the coming months on iOS, GazeSpeak aims at facilitating communication between a person with MND (known as ALS in the US, I will use both terms interchangeably) and another individual, perhaps their partner, carer or friend. Developed by Microsoft intern, Xiaoyi Zhang, GazeSpeak differs from traditional approaches in a number of ways. Before getting into the details however it’s worth looking at the background, GazeSpeaker didn’t come from nowhere, it’s actually one of the products of some heavyweight research into Augmentative and Alternative Communication (AAC) that has been taking place at Microsoft over the last few years. Since 2013, inspired by football legend and ALS sufferer Steve Gleason (read more here) Microsoft researchers and developers have put the weight of their considerable collective intellect to bear on the subject of increasing the ease and efficiency of communication for people with MND.

Last year Microsoft Research published a paper called ”
AACrobat: Using Mobile Devices to Lower Communication Barriers and Provide Autonomy with Gaze-Based AAC” (abstract and pdf download at previous link) which proposed a companion app to allow an AAC user’s communication partner assist (in an non-intrusive way) in the communication process. Take a look at the video below for a more detailed explanation.

This is an entirely new approach to increasing the efficiency of AAC and one that I suggest, could only have come from a large mainstream tech organisation who have over thirty years experience facilitating communication and collaboration.

Another Microsoft research paper published last year (with some of the same authors at the previous paper) called “Exploring the Design Space of AAC Awareness Displays” looks at importance of a communication partners “awareness of the subtle, social, and contextual cues that are necessary for people to naturally communicate in person”. There research focused on creating a display that would allow the person with ALS express things like humor, frustration, affection etc, emotions difficult to express with text alone. Yes they proposed the use of Emoji, which are a proven and effective way a similar difficulty is overcome in remote or non face to face interactions however they went much further and also looked at solutions like Avatars, Skins and even coloured LED arrays. This, like the other one above, is an academic paper and as such not an easy read but the ideas and solutions being proposed by these researchers are practical and will hopefully be filtering through to end users of future AAC solutions.

That brings us back to GazeSpeak, the first fruits of the Microsoft/Steve Gleason partnership to reach the general public. Like the AACrobat solution outlined above GazeSpeak gives the communication partner a tool rather than focusing on tech for the person with MND. As the image below illustrates the communication partner would have GazeSpeak installed on their phone and with the app running they would hold their device up to the person with MND as if they were photographing them. They suggest a sticker with four grids of letters is placed on the back of the smart phone facing the speaker. The app then tracks the persons eyes: up, down, left or right, each direction means the letter they are selecting is contained in the grid in that direction (see photo below).

man looking right, other person holding smartphone up with gazespeak installed

Similar to how the old T9 predictive text worked, GazeSpeak selects the appropriate letter from each group and predicts the word based on the most common English words. So the app is using AI in the form of machine vision to track the eyes and also to make the word prediction. In the New Scientist  article they mention that the user would be able to add their own commonly used words and people/place names which one assumes would prioritize them within the prediction list. In the future perhaps some capacity for learning could be added to further increase efficiency. After using this system for a while the speaker may not even need to see the sticker with letters, they could write words from muscle memory. At this stage a simple QR code leading to the app download would allow them to communicate with complete strangers using just their eyes and no personal technology.

Create inclusive content with Office Mix and Sway

Here in Enable Ireland AT service we have been investigating using the Office Mix plugin for PowerPoint to create more engaging and accessible eLearning content. While we are still at the early stages and haven’t done any thorough user testing yet, so far it shows some real promise.

From the end user perspective it offers a number of advantages over the standard YouTube style hosted video. Each slide is marked out allowing the user to easily skip forward or back to different sections. So you can skip forward if you are comfortable with a particular area of the presentation or more importantly revisit parts that may have not been clear. The table of contents button makes this even easier by expanding thumbnail views of all the slides which directly link to the relevant sections of the video. There is also the ability to speed up or slow down the narration. Apart from the obvious comic value of this it is actually a very useful accessibility feature for people who may be looking at a presentation made in a language not native to them or by someone with a strong regional accent. On the flip side it’s also a good way to save time, the equivalent of speed reading.

From the content creator’s perspective it is extremely user friendly. Most of us are already familiar with PowerPoint, these additional tools sit comfortably within that application. You can easily record your microphone or camera and add to a presentation you may have already created. Another feature is “Inking”, the ability to write on slides and highlight areas with different colour inks. You can also add live web pages, YouTube videos (although this feature did not work in my test), questions and polls. Finally the analytics will give you a very good insight as to what areas of your presentation might need more clarification as you can see if some chooses to look at a slide a number or times. You can also see if slides were skipped or questions answered incorrectly.

Below is a nice post outlining some ways to create inclusive content using Office Mix and Sway, Microsoft’s other new(ish) web based presentation platform. Below that is a much more detailed introduction to Office Mix using… yes you guessed it Office Mix.

How Office Mix and Sway can help with student inclusion – Gerald Haigh

Low-cost phone and tablet holders

It’s hard to beat the quality of mounting equipment offered by mounting suppliers such as Dassey or Rehadapt or even mainstream suppliers like Ram Mounts.  These mounting systems are designed to keep your hardware safe, made to last and they look good.

However, these mounting solutions also tend to be expensive and may be far from the budget of a user who may just require a second mount to take here and there with them.

There are many options for low-cost mounts that still provide the function of holding your phone or tablet so you can use it effectively.

Many low-cost mounts can be found on Amazon, eBay or even bought from supermarket chains such as Aldi or Lidl.  So it’s worth keeping an eye out as some of these products sell for as little as a few euro.

An example of mount recently bought from Lidl for €4.  It comprises of a spring loaded cradle, goose neck and spring loaded clamp. Although it will not take excessive pressure, but it works quite well for holding a phone at eye level for light touch screen use.

Low cost mount from lidl

Other online examples can be seen here

Adjustable Tablet Stand Holder Foldable

Breett Universal Phone Holder


New website to support people to use keyboard shortcuts

Girl typing on computer keyboard

Sharon’s Shortcuts is a new educational resource for people who primarily use keyboard shortcuts to access a computer.  The site contains different sections covering common tasks carried out using a PC.  All the keyboard shortcuts mentioned in this site are standard, Windows shortcuts that anyone can use.

While it’s easy to find plenty of tutorials and step by step instructions for using a PC that are mouse-based, this unique website gives step by step instructions on using a PC without the mouse making it a useful resource for screen reader users.

Sharon has over 10 years experience supporting people with a vision impairment and also provides One to One Tutoring Sessions for specific IT skills, getting to grips with work based systems, or a program of study towards a qualification like ECDL.

In this blogpost Sharon discusses her website and her tutoring services.

Listen to Sharon’s blogpost

The Big Life Fix

Just when we thought 2016 couldn’t get any better (in an AT sense) BBC make a prime time TV show with a huge focus on the design and construction of bespoke AT solutions. Although aired in December on BBC due to regional restrictions it’s not available to many on this side of the Irish Sea on iPlayer so you may not have had the chance to see full episodes yet. The good news is full episodes are beginning to make their way onto YouTube and are well worth a look. The general theme of the Big Life Fix would be how technology has the power to improve lives. Although not just about what we call assistive technology, it is more broad in scope covering many different types of technology challenge with the goal of democratising and demystifying solutions. AT does play a big part in many of the challenges however.

The first episode (a clip of which I’ve embedded below) introduces us to James, a young photographer who is having difficulty operating his SLR camera. The solution created for James features all the exciting technology and techniques being utilised every day by Makers around the world: Arduino microprocessor, 3D printing, AppInventor as well as some good old fashioned hardware hacking. The iterative nature of the design process is well illustrated with James critically evaluating the initial prototype and providing insights which significantly change the direction of the design.The other AT related challenge in this first episode features a graphic designer called Emma who due to tremors which are a symptom of her Parkinson’s, is unable to draw or sign her name. After a number of prototypes and lots of research a very clever solution is arrived at which seems to be extremely effective, leading to a rather emotional scene (have the hankies ready).

The Big Life Fix beautifully portrays both the potential of AT to improve the quality of life as well as the personal satisfaction a maker might get from participating in a successful solution. I can see this show sowing the seeds for a strong and equitable future for assistive technology.

Finally, the icing on the cake is that all the solutions featured on the show are Open Source with all the source code, design files and build notes that were used to print, shape and operate the solutions publicly available on GitHub. Nice work BBC. Take a look at the clip below (UPDATE: Full Episodes now on YouTube. Not sure how long they will stay there though).

AbilityNet’s Robin Christopherson honoured for services to digital inclusion with MBE

How refreshing to see that Digital Inclusion gets the nod in this year’s UK MBE awards, with AbilityNet’s Robin Christopherson receiving well-deserved recognition for his longstanding contribution to the world of accessibility, and digital accessibility in particular.

Here he is, speaking at the Tech4Good Awards in July 2016: living and breathing digital inclusion in his daily life:



Inbuilt Accessibility – AT in mainstream technology

There is of course some cross over between the different AT highlights of 2016 I have included here. An overall theme running through all the highlights this year is the mainstreaming of AT. Apple, Google and Microsoft have all made significant progress in the areas previously mentioned: natural language understanding and smart homes. This has led to easier access to computing devices and through them the ability to automate and remotely control devices and services that assist us with daily living tasks around the house. However these developments are aimed at the mainstream market with advantages to AT users being a welcome additional benefit. What I want to look at here are the features they are including in their mainstream products specifically aimed at people with disabilities with the goal of making their products more inclusive. Apple have always been strong in this area and have lead the way now for the last five years. 2016 saw them continue this fine work with new features such as Dwell within MacOS and Touch Accommodations in iOS 10 as well as many other refinements of already existing features.  Apple also along with Siri have brought Switch Control to Apple TV either using a dedicated Bluetooth switch or through a connected iOS device in a method they are calling Platform Switching. Platform Switching which also came out this year with iOS 10 “allows you to use a single device to operate any other devices you have synced with your iCloud account. So you can control your Mac directly from your iPhone or iPad, without having to set up your switches on each new device” (need to be on the same WiFi network). The video below from Apple really encapsulates how far they have come in this area and how important this approach is.

Not to be outdone Microsoft bookended 2016 with some great features in the area of literacy support, an area they had perhaps neglected for a while. They more than made up for this last January with the announcement of Learning Tools for OneNote. I’m not going to go into details of what Learning Tools offers as I have covered it in a previous post. All I’ll say is that it is free, it works with OneNote (also free and a great note taking and organisation support in its own right) and is potentially all many students would need by way of literacy support (obviously some students may need additional supports). Then in the fourth quarter of the year they updated their OCR app Office Lens for iOS to provide the immersive reader (text to speech) directly within the app.

Finally Google who would probably have the weakest record of the big 3 in terms of providing inbuilt accessibility features (to be fair they always followed a different approach which proved to be equally effective) really hit a home run with their Voice Access solution which was made available for beta testing this year. Again I have discussed this in a previous post here where you can read about it in more detail. Having tested it I can confirm that it gives complete voice access to all Android devices features as well as any third party apps I tested. Using a combination of direct voice commands (Open Gmail, Swipe left, Go Home etc.) and a system of numbering buttons and links, even obscure apps can be operated. The idea of using numbers for navigation while not new is extremely appropriate in this case, numbers are easily recognised regardless of voice quality or regional accent. Providing alternative access and supports to mainstream Operating Systems is the corner stone of recent advances in AT. As the previous video from Apple showed, access to smartphones or computers gives access to a vast range of services and activities. For example inbuilt accessibility features like Apple’s Switch Control   or Google’s Voice Access open up a range of mainstream Smart Home and security devices and services to people with alternative access needs where before they would have to spend a lot more for a specialist solution that would have probably been inferior.

Dawn of the Personal Digital Assistants

Speech Recognition has been around a long time by technology standards however up until about 2010 most of it was spent languishing in Gartner’s wonderfully named “Trough of Disillusionment”. This was partly because the technology hadn’t matured enough and people were frustrated and disappointed when it didn’t live up to expectations, a common phenomenon identified by the previously alluded to Hype Cycle. There are a couple of reasons why Speech Recognition took so long to mature. It’s a notoriously difficult technical feat that requires sophisticated AI and significant processing power to achieve consistently accurate results. The advances in processing power were easy enough to predict thanks to Moore’s Law. Progress in the area of AI was a different story entirely. Speech Recognition relies first on pattern recognition, but that only takes it so far. To improve the accuracy of speech recognition improvements in the broader area of natural language processing were needed. Thanks to the availability of massive amounts of data via the World Wide Web, much of it coming from services like YouTube we have seen significant advances in recent years. However there is also human aspect to the slow uptake of speech driven user interfaces, people just weren’t ready to talk to computers. 2016 is the year that started to change.

Siri (Apple) who was first on the scene and is now 5 years old and getting smarter all the time came to MacOS and AppleTV this year. Cortana (Microsoft) who started on Windows Phone, then to the desktop with Windows 10, made her way onto Xbox One, Android and iOS and is soon to be embodied in all manner of devices according to reports. Unlike Siri, Cortana is a much more sociable personal digital assistant, willing to work and play with anyone. By this I mean Microsoft have made it much easier for Cortana to interact with other apps and services and will be launching the Cortana Skills Kit early next year. As we’ve seen in the past it’s this kind of openness and interoperability that takes technologies in directions not envisaged and often leads to adaption and adoption as personal AT. If there was a personal digital assistant of the year award however, Amazon Echo and Alexa would get it for 2016. Like Microsoft, Amazon have made their Alexa service easy for developers to interact with and many manufacturers of Smart Home products have jumped at the opportunity. It is the glowing reviews from all quarters however that makes the Amazon Echo stand out (a self-proclaimed New Yorker Luddite to the geeks at CNET). Last but not least we have Google. What Google’s personal digital assistant lacks in personality (no name?) it makes up for with stunning natural language capabilities and an eerie knack of knowing what you want before you do. Called Google Now on smartphones (or just Google App? I’m confused!), similar functionality without some of the context relevance is available through Voice Search in Chrome. They also offer voice to text in Google Docs which this year has been much improved with the addition of a range of editing commands. There is also the new Voice Access feature for Android currently in beta testing but more on that later. In the hotly contested area of the Smart Home Google also have a direct competitor to Amazons Echo in their Google Home smart speaker. Google are a strong player in this area, my only difficulty (and it is an actual difficulty) is saying “ok Google”, rather than rolling off the tip of my tongue it kind of catches at the back requiring me to use muscles normally reserved for sucking polo mints. Even though more often than not I mangle this trigger phrase it always works and that’s impressive. So who is missing? There is one organisation conspicuous by their absence with the resources in terms of money, user data and technology who are already positioned in that “personal” space. Facebook would rival Google in the amount of data they have at their disposal from a decade of video, audio and text, the raw materials for natural language processing. If we add to this what Facebook knows about each of its users; what they like, their family, friends and relationships (all the things they like), calendar, history, interests… you get more than a Personal Digital Assistant, maybe Omnipersonal Digital Assistant would be more accurate. The video below which was only released today (21/12/16) is of course meant as a joke (there are any number of things I could add here but I’ll leave it to the Guardian). All things considered however it’s only a matter of time before we see something coming out of Facebook in this area and it will probably take things to the next level (just don’t expect it to be funny).

What does this all mean for AT? At the most basic level Speech Recognition provides an alternative to the keyboard/mouse/touchscreen method of accessing a computer or mobile device and the more robust and reliable it is the more efficiently it can be used. It is now a viable alternative and this will make a massive difference to the section of our community who have the ability to use the voice but perhaps for any number of reasons cannot use other access methods. Language translation can be accurately automated, even in real time like the translation feature Skype launched this year. At the very least this kind of technology could provide real-time subtitling but the potential is even greater. It’s not just voice access that is benefiting from these advances however, Personal Digital Assistants can be interacted with using text also. Speech Recognition is only a part of the broader area on Natural Language Processing. Advances in this area lead directly to fewer clicks and less menu navigation. Microsoft have used this to great effect in their new “Tell me what you want to do” feature in their Office range. Rather than looking through help files or searching through menus you just type what tool you are looking for, in your own words, and it serves it right up!

Natural Language Processing will also provide faster and more accurate results to web searches because there is a better understanding of actual content rather than a reliance on keywords. In a similar way we are seeing this technology working to provide increased literacy supports as the computer will be able to better understand what you mean from what you type. Large blocks of text can be summerised, alternative phrasing can be suggested to increase text clarity. Again the new Editor feature in Microsoft Word is made possible by this level of natural language understanding.

2016 – Technology Trends and Assistive Technology (AT) Highlights

As we approach the end of 2016 it’s an appropriate time to look back and take stock of the year from an AT perspective. A lot happened in 2016, not all good. Socially, humanity seems to have regressed over the past year. Maybe this short term, inward looking protectionist sentiment has been brewing longer but 2016 brought the opportunity to express politically, you know the rest. While society steps and looks back technology continues to leap and bound forward and 2016 has seen massive progress in many areas but particularly areas associated with Artificial Intelligence (AI) and Smart Homes. This is the first in a series of posts examining some technology trends of 2016 and a look at how they affect the field of Assistive Technology. The links will become active as the posts are added. If I’m missing something please add it to the comments section.

Dawn of the Personal Digital Assistants

Game Accessibility

Inbuilt Accessibility – AT in mainstream technology 

Software of the Year – The Grid 3

Open Source AT Hardware and Software

The Big Life Fix

So although 2016 is unlikely to be looked on kindly by future historians… you know why; it has been a great year for Assistive Technology, perhaps one of promise rather than realisation however. One major technology trend of 2016 missing from this series posts is Virtual (or Augmented) Reality. While VR was everywhere this year with products coming from Sony, Samsung, Oculus and Microsoft its usefulness beyond gaming is only beginning to be explored (particularly within Education).

So what are the goals for next year? Well harnessing some of these innovations in a way where they can be made accessible and usable by people with disabilities at an affordable price. If in 2017 we can start putting some of this tech into the hands of those who stand to benefit most from its use, then next year will be even better.

Now Amazon Alexa can be used with Harmony hub remote

Amazon Echo is a smart speaker developed by The device consists of a 9.25-inch (23.5 cm) tall cylinder speakerNow Amazon Alexa can be used with any Harmony hub-based remote, such as Harmony Elite or Harmony Companion. You can say “Alexa, turn on the TV”

Amazon have three products, the Echo, Dot, and Tap that all have Alexa Voice Service. Users access music, information, news, sports scores, weather, and more. There is a growing number of smart home devices and applications that work with these devices that will enable greater access, especially for people with physical disabilities. One new product Echo works with is Logitech’s Harmony Hub. The Harmony Hub and app allows users to control multiple appliances on one digital device. People can use their smartphone, tablet, or Echo to control lights, stereos, the television, and more.
To use the Echo and Harmony Hub as a hands-free remote users must link the two devices together. One way to link the two devices is through the use of IFTTT, a free web-based service. Once all of these components are linked together, users can control devices, such as a television, with their voice.
Here is a quick video demonstrating the two devices and IFTTT working together.