Eyegaze for Musical Expression

Background – What is eyegaze?

Eyegaze is an alternative way of accessing a computer using eye movements to control the mouse. It is achieved through a combination of hardware and software. The hardware is a USB perhipal called an eye tracker. The eye tracker is positioned underneath the computer monitor. It contains a camera and Infrared lights. The user is positioned between 500 and 1000 mm from the monitor (600mm is usually about right) where the camera has a clear view of their eyes. The Infrared lights highlight the user’s pupils (think of red eye in photographs where a flash has been used) and create reflections on the user’s eyeballs. After a calibration process where the user looks at a dot moving around the screen, the software can accurately tell where the user is looking based on the reflections and movements of the pupil. For computer access the user will also need tome method of clicking. There are 3 methods usually used. Dwell is the most common method. This is where the click is automated. If the user holds their gaze (dwells) on a button or icon for more than a specified time duration, usually somewhere from .5 to 1.5 sec, a click is sent. A slight variation of this is used in some software designed for eyegaze where the button is activated after the user dwells on it. The main difference here is that the second method offers us the ability to select different dwell times for different buttons. The other input methods are less common. The first would be to use an external switch as a mouse click, the second would be to use a deliberate blink (longer than a normal blink to prevent accidental clicks) as a mouse click.  

Eye Tracker Devices

  • Tobii Tracker 4C https://gaming.tobii.com/tobii-eye-tracker-4c/ – This is a great option for those wanting to use eyegaze for activities like music and gaming but have other AT as their main access method. It is every bit as good as the two much more expensive “AT” eye trackers below and costs in the region of €170.
  • Tobii PC Eye Plus and Mini https://www.tobiidynavox.com/products/devices/ – The PC Eye Mini and PC Eye Plus are probably the most popular AT eye trackers. The mini will work well on a monitor up to 19”, the Plus also contains a high quality microphone array to support speech recognition, it also has a switch input port. The Plus will work on screens up to 28”.
  • EyeTech TM5 https://eyetechds.com/eye-tracking-products/tm5-mini-eye-tracker/. The EyeTech TM5 is quite similar to the Tobii PC Eye Mini. One key difference that might influence the choice of this eye trackers is that it supports a slightly closer user position.

Challenges associated with playing music using eye movement

These are a number of difficulties we might encounter when playing music using eye movements but all can be overcome with practice and by using some common music production tools and techniques. Eye gaze as an input method is quite restrictive. You only have one point of direct access, so you can think of it like playing a piano with one finger. To compound this difficulty and expand the piano analogy, because your eyes are also your input you cannot queue up your next note like a one fingered piano player might. Eyegaze in itself is just eye pointing, using it as an access method will require some input (click) ether a switch or a dwell (automatic click after a specific time duration, usually somewhere from .5 to 1.5 sec). If you are using dwell for input then this will add a layer of difficulty when it comes to timing. You could set the dwell to be really fast (like .1 second) but you may run into accidental activations in this case, for example playing a note as you are passing over it on the way to your intended note. Some of the specialist eyegaze software instruments like EyeHarp, EyePlayMusic and ii-music overcome this by using a circular clock style interface. This allows them set the onscreen buttons to instant activation and because of the radial layout each note can be directly accessed from the centre without passing over another note. Using the radial design if our eyes are in a central position all notes are equal distance from us and can be accessed in the most efficient way but we are still left with the “one finger piano” restriction. This means no chords and only the option of playing at a slower tempo. Using mainstream music productions like sequencers, arpeggiators or chord mode can overcome this limitation and allow us create much more complex music using eyegaze. A sequencer would allow you pre program accompanying notes with which to play along. An arpeggio is sometimes referred to as a broken chord. It is the notes of a chord played consecutively rather than simultaneously. Arpeggios are used a lot in electronic music. By playing arpeggios the slower input is offset by the additional life and movement provided by the arpeggio. Chord mode is something that can be set up in many digital audio workstations. You can map one note to automatically play the accompanying notes required to make it a chord. Live looping could also be used. In looping we would record a section being played live, then loop it back and play other notes over it. Other effects like delay, reverb and many more besides, will also allow is make interesting music.

Expression is another difficulty when playing music using eye tracking. By expression we mean how an accomplished musician can play the same note in different ways to make it more expressive. Velocity is a common means of expression, you can think of this a how fast/hard a note is struck. Velocity can affect volume and other qualities of the instrument’s sound. Another common means of expression is provided pedals like those on an organ or piano. Using eyegaze we really only have the ability to turn the note on or off. Some of the software however breaks note areas up into sections, each one giving an increased velocity (see photo below).           

Software for playing music with Eyegaze

  • Eye Harp http://theeyeharp.org/ One of the first software instruments made specifically for eyegaze, the EyeHarp remains one of the best options. This software was originally developed as a college project (I guessing he got a first!) and rather than let it die developer Zacharias Vamvakousis made it available free and open source. After a few years with now updates the news is that there are some big updates on the way. We are looking forward to seeing what they have in store for us.
animated gif showing the eyeharp performance screen. a clock type circular interface divided into sections. eyes at the centre of the circle

Another option for eyegaze music production is using software like the Grid 3 or Iris to create an eyegaze accessible interface for a mainstream digital audio workstation. The demo below is done using Ableton Live however any software that offers keyboard mapping or keyboard shortcuts (so any quality software) could be used in the same way.

Leave a Reply

Your email address will not be published. Required fields are marked *