Eyegaze for Musical Expression

Background – What is eyegaze?

Eyegaze is an alternative way of accessing a computer using eye movements to control the mouse. It is achieved through a combination of hardware and software. The hardware is a USB perhipal called an eye tracker. The eye tracker is positioned underneath the computer monitor. It contains a camera and Infrared lights. The user is positioned between 500 and 1000 mm from the monitor (600mm is usually about right) where the camera has a clear view of their eyes. The Infrared lights highlight the user’s pupils (think of red eye in photographs where a flash has been used) and create reflections on the user’s eyeballs. After a calibration process where the user looks at a dot moving around the screen, the software can accurately tell where the user is looking based on the reflections and movements of the pupil. For computer access the user will also need tome method of clicking. There are 3 methods usually used. Dwell is the most common method. This is where the click is automated. If the user holds their gaze (dwells) on a button or icon for more than a specified time duration, usually somewhere from .5 to 1.5 sec, a click is sent. A slight variation of this is used in some software designed for eyegaze where the button is activated after the user dwells on it. The main difference here is that the second method offers us the ability to select different dwell times for different buttons. The other input methods are less common. The first would be to use an external switch as a mouse click, the second would be to use a deliberate blink (longer than a normal blink to prevent accidental clicks) as a mouse click.  

Eye Tracker Devices

  • Tobii Tracker 4C https://gaming.tobii.com/tobii-eye-tracker-4c/ – This is a great option for those wanting to use eyegaze for activities like music and gaming but have other AT as their main access method. It is every bit as good as the two much more expensive “AT” eye trackers below and costs in the region of €170.
  • Tobii PC Eye Plus and Mini https://www.tobiidynavox.com/products/devices/ – The PC Eye Mini and PC Eye Plus are probably the most popular AT eye trackers. The mini will work well on a monitor up to 19”, the Plus also contains a high quality microphone array to support speech recognition, it also has a switch input port. The Plus will work on screens up to 28”.
  • EyeTech TM5 https://eyetechds.com/eye-tracking-products/tm5-mini-eye-tracker/. The EyeTech TM5 is quite similar to the Tobii PC Eye Mini. One key difference that might influence the choice of this eye trackers is that it supports a slightly closer user position.

Challenges associated with playing music using eye movement

These are a number of difficulties we might encounter when playing music using eye movements but all can be overcome with practice and by using some common music production tools and techniques. Eye gaze as an input method is quite restrictive. You only have one point of direct access, so you can think of it like playing a piano with one finger. To compound this difficulty and expand the piano analogy, because your eyes are also your input you cannot queue up your next note like a one fingered piano player might. Eyegaze in itself is just eye pointing, using it as an access method will require some input (click) ether a switch or a dwell (automatic click after a specific time duration, usually somewhere from .5 to 1.5 sec). If you are using dwell for input then this will add a layer of difficulty when it comes to timing. You could set the dwell to be really fast (like .1 second) but you may run into accidental activations in this case, for example playing a note as you are passing over it on the way to your intended note. Some of the specialist eyegaze software instruments like EyeHarp, EyePlayMusic and ii-music overcome this by using a circular clock style interface. This allows them set the onscreen buttons to instant activation and because of the radial layout each note can be directly accessed from the centre without passing over another note. Using the radial design if our eyes are in a central position all notes are equal distance from us and can be accessed in the most efficient way but we are still left with the “one finger piano” restriction. This means no chords and only the option of playing at a slower tempo. Using mainstream music productions like sequencers, arpeggiators or chord mode can overcome this limitation and allow us create much more complex music using eyegaze. A sequencer would allow you pre program accompanying notes with which to play along. An arpeggio is sometimes referred to as a broken chord. It is the notes of a chord played consecutively rather than simultaneously. Arpeggios are used a lot in electronic music. By playing arpeggios the slower input is offset by the additional life and movement provided by the arpeggio. Chord mode is something that can be set up in many digital audio workstations. You can map one note to automatically play the accompanying notes required to make it a chord. Live looping could also be used. In looping we would record a section being played live, then loop it back and play other notes over it. Other effects like delay, reverb and many more besides, will also allow is make interesting music.

Expression is another difficulty when playing music using eye tracking. By expression we mean how an accomplished musician can play the same note in different ways to make it more expressive. Velocity is a common means of expression, you can think of this a how fast/hard a note is struck. Velocity can affect volume and other qualities of the instrument’s sound. Another common means of expression is provided pedals like those on an organ or piano. Using eyegaze we really only have the ability to turn the note on or off. Some of the software however breaks note areas up into sections, each one giving an increased velocity (see photo below).           

Software for playing music with Eyegaze

  • Eye Harp http://theeyeharp.org/ One of the first software instruments made specifically for eyegaze, the EyeHarp remains one of the best options. This software was originally developed as a college project (I guessing he got a first!) and rather than let it die developer Zacharias Vamvakousis made it available free and open source. After a few years with now updates the news is that there are some big updates on the way. We are looking forward to seeing what they have in store for us.
animated gif showing the eyeharp performance screen. a clock type circular interface divided into sections. eyes at the centre of the circle

Another option for eyegaze music production is using software like the Grid 3 or Iris to create an eyegaze accessible interface for a mainstream digital audio workstation. The demo below is done using Ableton Live however any software that offers keyboard mapping or keyboard shortcuts (so any quality software) could be used in the same way.

Logitech Adaptive Gaming Kit – The Final Key to Disabled Gaming

Last year I wrote a review of the Xbox adaptive controller. I detailed how it had opened up the world of gaming to many people with a disability after years of looking longingly at gamers who delved into another round of FIFA or Grand Theft Auto. By the time I was done I realised that now only one barrier remained the barrier of cost. Thankfully that is where Logitech has stepped in with their new gaming accessory kit to alleviate some of that financial pressure.

Taking a quick look back at the review of the Xbox adaptive controller you’ll see that the controller connects with the Xbox and where it becomes adaptive is that it can be used with any form of adaptive devices that you may use depending on your disability, most often those devices are series of different pressure pads or buddy buttons. In my case I use the adaptive controller along with a series of about 4 to 6 buddy buttons to act as the trigger buttons on the top of the normal Xbox controller, buttons I normally otherwise would never be able to access restricting me in 90% of games available on the Xbox.

To Quote Brad Pitt in Seven “What’s in the Box?”

Before I even get as far as describing what is in the box funnily enough I’m going to describe the box itself. Logitech seem to have taken to take all aspects of the adaptive nature of the product into account by making the packaging more accessible. The tape sealing the box shut has Loops at the end for somebody with limited use of their hands and weak grip to easily pull the box open. Inside there is a huge array of devices each of which is packaged in a plastic bag (not for the environmentalists) that are loose and slippy so the device can be easily slid out.

So that’s the box itself dealt with it. now what is inside the box? The box contains an array of 12 different pressure activation buttons (see photo below). These activation buttons vary in size and in response time and are designed to suit a variety of different disabilities. Logitech have also included two sheets of stickers that you can apply to each button you’re using , these stickers identify which button on the Xbox controller your activation pressure buttons represent.

the logitech kit has 4 switch types. All black from left is the light touch button (4 in kit), large button (3 in kit), Variable triggers (2 in kit) and small button (3 in kit)

It has also taken into account the frustration that is involved when one button slips at the most crucial of points by including a collection of velcro stickers  and two pads that can interconnect with one another that sit across your lap and hold your buttons in place making them more accessible to you when you need them most. Now you’re far less likely to have them slip from underneath your hand as you are about to shoot that last enemy in Fortnite or score the winning goal in FIFA.

It’s All About the Money, Cost?

It’s very simple if you are living on disability allowance alone gaming is still very expensive. The consoles themselves are expensive not to mention the price of the games.

Unfortunately like most things once you add in the word disability there is a further cost. The Xbox adaptive controller on its own is not very useful for most people with a disability and that unit itself cost in the region of €80.

The adaptive controller must be combined with the activation pressure buttons that are most often used in conjunction with the adaptive controller. This is where the price starts to go up very very quickly.

Each buddy button can cost in the region of 60 to €80. When you consider that I need to use a minimum of 4 to 6 body buttons to use the adaptive controller to it’s full potential you can see how the cost can rocket very quickly. That’s a potential cost of €480 to fully equip you with the buttons you need.

So taking that into account Logitech gaming accessory pack price of €99 is a complete bargain with a variety of 12 different pressure buttons included within the pack. They are more lightweight and possibly will take less of a beating than some of the official ones which appear to have a more sturdy build but it is a fantastic opportunity.

Have a look at the video below to learn more about the process that made this kit possible.

 Even if you are not a gamer but use a number of pressure activation buttons or buddy buttons around the house in your day-to-day life then the Logitech gaming accessory it could be a solution for you.

Get your Adaptive Gaming Kit from Logitech here

Learning Tools – Using technology to support learning and facilitate collaboration in education

Yesterday Microsoft Ireland hosted a half-day workshop for second level students using technology for additional support within education. This workshop came about thanks to Tara O’Shea, Community Affairs Manager at Microsoft and Stephen Howell, Academic Program Manager. Tara has been a huge supporter of Enable Ireland Assistive Technology Service over the last decade and been the driving force behind many of the successful projects we have collaborated on. Stephen would be a very familiar face to anyone involved in that space where technology and education meet, not just in Ireland but internationally.

The goal of the workshop was to introduce some of the collaboration tools available to students using Office365, additional supports available to students with maths or language difficulties and to provide alternative ways to produce and present content. Obviously as Microsoft was hosting there was an emphases on their tools nevertheless Stephen was quite open about how similar features are available on other platforms. We (Enable Ireland AT) pride ourselves on providing independent recommendations; the best solution for the user is the solution they use best. The practice of schools forcing students down any particular route: Microsoft, Google or Apple, is restrictive and cause difficulties if there are specific access or support needs. Microsoft and Google though offer more browser-based tools that mean users are free to use any device. I should also acknowledged that Microsoft have really upped their game in the areas of Education and Accessibility over the last few years.   

Collaboration

Fostering collaboration is a cornerstone of modern education and promotes a vital real world skill (teamwork) that will serve students throughout their lives. The screenshot below from Facebook (Stephanie McKellop) and illustrates a way that tools we may have considered more for remote collaboration, can be used within a classroom or lecture hall.

Facebook screenshot from user
Stephanie McKellop.
I learned today that a group of students used a Google doc to take lecture notes -- they all took notes simultaneously in a collective file.
They would mark places they were confused or couldn't follow the lecturer. other students would see and explain.
at the end of the semester they have a massive document of note, questions and explanations from peers

Although this example uses Google Docs, Microsoft OneNote could also be used in this way. In fact there would be a number of advantages to using OneNote such as the ability to incorporate Ink annotations and drawings, audio & video and adding whiteboard or print text using Office Lens.

When it comes to collaboration, Microsoft Teams is at the centre. Teams is a unified communications platform, basically it’s like WhatsApp or Facebook Messenger but with tonnes of additional features. Through Teams you can not only instant message, video/audio call or share desktops but you can also work on shared documents, whiteboards or mind maps. There are also plugins for many third party apps and services, so if you are already collaboration app or service there is probably an integration available. Stephen demonstrated how a tool like Teams could be used in a classroom session by setting up a class team and getting everyone to work on a short Sway presentation (we mentioned Sway in a previous post a couple of years ago, don’t understand why everyone isn’t using it by now). Once everyone had completed their presentation they posted a link to the class message stream and Stephen showed it on the large screen. Okay, this exercise could have been done without Teams but using the service made it so much easier and more importantly everything was recorded for students to revisit in their own time.

Support

We have looked at Microsoft Learning Tools numerous times on this blog over the last few years (read this post is you want to know more about Learning Tools). Thankfully, since its introduction as a plugin for OneNote in 2016 it has gone from strength to strength. Features like Immersive Reader are now standalone apps and have also found their way into many other Office365 apps like Word and Outlook. Some other apps Stephen introduced are listed below with a brief description. They are all free so we encourage you to download and try them yourselves.

Microsoft Math: If you are familiar with the language-learning app Duolingo, this app takes a similar approach to teaching Mathematics. Short challenges with rewards and feedback. Gamifying Maths

Snip & Sketch: Lets you quickly and easily capture content from the web (pictures, text etc), draw and annotate it and share with other apps.

Microsoft WhiteBoard: Provides a blank canvas where you can collaborate with others and share with the class

Microsoft Translator: Useful for translations or transcriptions. Stephen also showed how it can be a great way to practice pronunciation when learning to speak a foreign language.    

XAC (XBOX Adaptive Controller) User Review

I have always been a bit of a gamer. From Tetris on the original Gameboy to Sonic and the SEGA Mega Drive, I was always keen to pass the time away rapidly instructing a cartoon character to bounce from one side of the screen to another. Since I acquired my disability in 1999 though I felt
that large parts of this world were now no longer accessible to me. I felt with limited use of my arms and no use of my fingers consoles were out of the question. That changed recently when the Xbox brought out their new accessible controller.

I had tried to use several different games on the PlayStation and the Xbox, my nephew had a PlayStation and I had been able to use the left stick and some of the buttons on the ordinary controller but despite me telling him not to use the trigger buttons which were inaccessible to me I still got hammered several times by him on FIFA.

This new accessible controller seemed as though it would provide me with the opportunity to have the full experience of console gaming again, but who is going to buy an Xbox One and accessible controller just to see if they can use it or not? Thankfully Enable Ireland came to my rescue and
they allowed me to borrow their console and controller for the period of a month.


XBox Adaptive Controller (XAC)

The controller is simple to use and simple to set up. I needed some help to physically plug some aids in and out of the controller but apart from that it was a breeze.

The controller is setup for people of all abilities. The variety of configurations is as wide as the number of disabilities of the people who it is geared to provide for.

The xbox adaptive controller with some compatible accessories, switches, one handed joystick


I used the controller mainly for games like FIFA, Ryse, Forza 5, and some slightly more intricately controlled games like Grand Theft Auto and Battlefield.

Some games I used just the accessible controller with the coloured plug in switches that Enable Ireland provided alongside the console.

For other more complicated games, I used the Co-Pilot feature. The Co-Pilot feature allows you to use the ordinary controller as best you can while using the accessible controller switches for any bits or buttons on the ordinary controller that you can’t access.

Forza 5

Forza 5 cover
Forza switch setup. 4 switches. break, go , left , right

My setup for Forza, the car racing game, was the simplest of all. I took 4 of the aid switches and plugged them into the accessible controller, one was plugged into RT for the accelerator, one was plugged into LT for the brake, and the remaining two were plugged into the left and right ports on
the d-pad. I placed the RT switch under my elbow to continuously accelerate, which then meant my hands only had to focus on the three remaining buttons for steering and braking. That was a huge success, and meant I did not need any assistance throughout any of the gameplay on that particular game. Though that does not mean I was a great driver!

using elbow switch for accellator left only 3 switches to operate and drive successfully

FIFA 19

FIFA 19 Cover
switch setup for FIFA 19. One switch on arm rest, two on right leg, one on left leg and the xbox controller

For FIFA I used the Co-Pilot feature. I used the ordinary controller as I had done previously with my nephew, steering my player with the left stick while passing, tackling, shooting, etc with the usual A, B, X, and Y buttons.

I used the Xbox Accessible Controller then for the sprint and switch player options. I simply plugged in the switches into the RT and LT ports on the accessible controller and played normally on the ordinary controller while occasionally tapping the switches to change player or holding them down
with my elbow to sprint.

A very successful and intelligent solution which resulted in a 5-1 victory for me over my nephew! His face was a picture 🙂

Ryse, GTA & Battlefield

Ryse cover
Grand Theft Auto cover
Battlefield cover

Each of these I played with a similar set up to FIFA (pictured above). I used the Co-Pilot feature, the ordinary controller in conjunction with the accessible controller with four switches plugged into the RT, LT, RB, and LB ports.

Mainstream controller supplemented by a switch on the armrest, two on right knee and one on left

These games were a bit more intricate in their controls in comparison to the others and a little more difficult to use as a result. The accessible controller meant though that it was possible for me to at least give it a go. This controls setup was good and meant that I actually completed the story mode of Ryse, on easy.

I could play the vast majority of GTA and Battlefield without any difficulty, but there were certain issues. To use the character’s “special abilities” in GTA you had to press down on both the left and right sticks. I think you could set that up but that would require two more switches which I didn’t have.

Also, on occasion, while I had all the right buttons the scenario in the game was so complex that it involved pressing a number of buttons and steering at least one, if not both, sticks at the same time. It was almost equivalent to playing some musical instrument. On one mission I did have to fall back on some assistance from my nephew.

Conclusion

While it is still not quite the same as gaming prior to my disability the Xbox Accessible Controller has reopened the prospect of gaming properly on a regular basis and owning a console of my own again. This was a world that I thought had long left me behind but thanks to Microsoft and Xbox I’m
right back in the game!

Eye Control – Inbuilt EyeGaze Access for Windows 10

Just yesterday Microsoft announced what is possibly their biggest step forward in functionality within their Ease of Access accessibility settings since Windows 7. Eye Control is an inbuilt feature to facilitate access to the Windows 10 OS using the low cost eyegaze peripheral the Tracker 4 C from Tobii. More about what you can actually do with Eye Control below but first a little background to how this has come about.

Steve Gleeson and his son

Former American Football professional and MND (ALS) sufferer Steve Gleason (above) challenged Microsoft in 2014 to help people affected by this degenerative condition through the advancement eye tracking technology. This initial contact lead to the development of a prototype eye gaze controlled wheelchair, receiving lots of publicity and generating increased awareness in the process. However it was never likely to be progressed to a product that would be available to other people in a similar situation. What this project did achieve was to pique the interest of some of the considerable talent within Microsoft into the input technology itself and its application, particularly for people with MND.

A combination of factors felt on both sides of the Atlantic have proved problematic when it comes to providing timely AT support to people diagnosed with MND. Eyegaze input is the only solution that will allow successful computer access as the condition progresses, eye movement being the only ability left in the final stages of the illness. However, historically the cost of the technology meant that either insurance, government funding or private fundraising was the only means by which people could pay for eyegaze equipment. Usually this resulted in a significant delay which, due to the often aggressive nature of MND meant valuable time was lost and often the solution arrived too late. This situation was recognized by Julius Sweetland who led the development of Optikey, an Open Source computer access/AAC solution designed to work with low cost eye trackers back in 2015. Interestingly some of the innovative features of Optikey seem to have made it to Eye Control on Windows 10 (Multi-Key selection called Shape Writing on Eye Control – see gif below).

demo of shap writing on Eye Control - works like swiping on a touch keyboard. dwell on the first letter of a word, glance at subsequent letters and dwell on last letter. word is entered

Since the initial Steve Gleason Wheelchair hack there has been a steady stream of high quality research papers coming from people at Microsoft on the subject of eyegaze input and MND solutions. This should have been a hint that something like Eye Control was on the horizon. EyeGaze input has promised to break into the mainstream several times over the last decade however with Eye Control and support for devices being included in the core Windows OS it has never been this close.

For more background on the path to Eye Control see this Microsoft blog post from Microsoft:  From Hack to Product, Microsoft Empowers People with Eye Control for Windows 10

Want to find out how to get early access to Eye Control or get some more information on the functionality read this post from Tobii (be warned there are still bugs):  How to get started with Eye Control on Windows.

Accessibility Checker for Word Tutorial

The Accessibility Checker feature has been part of Microsoft Office for the last few iterations of the software package. It provides a fast and easy way to check whether the content you are producing is accessible to users of assistive technology. By making accessibility accessible Microsoft have left no room for excuses like “I didn’t know how…” or “I didn’t have time..”. You wouldn’t send a document to all your colleagues full of misspellings because you were in a hurry would you? The one criticism that could have been leveled at Microsoft was perhaps they didn’t provide enough support to new users of the tool. As I said above it’s easy to use but sometimes users need a little extra support, especially when you are introducing them to something that may be perceived as additional work. Thankfully Microsoft have filled that gap with a 6 part tutorial video which clearly explains why and how to get started using Accessibility Checker. Part 1 is a short introduction (embedded below) followed by a video on each important accessibility practice; Alternative Text, Heading Styles, Hyperlinks, File naming and Tables. Each video is accompanied by a short exercise to allow you put your new skill into practice immediately. The whole tutorial can be completed in under 20 minutes. This tutorial should be a requirement for anybody producing documents for circulation to the public. Have a look at the introduction video below.