Technology to support Remote/Virtual AT Assessment

In Enable Ireland’s National AT Service we have conducted many Remote AT Assessments over the last decade. David Banes (@DaveBanesAccess) first introduced the idea to us based on his work in AbilityNet. We used Skype to connect directly with a client, or more often a local therapist with the client present. On most occasions it was just used to replace the initial “getting to know each other” meeting. Technology would not necessarily be involved but plans for the follow up face-to-face assessment would be made. Occasionally, supported by the limited view available from a USB webcam and screen sharing software, we would assist the therapist and their client setting up and trialling a number of solutions we had sent to them. For hardware solutions at least, the remote assessment was never seen as an appropriate substitution for a real world multidisciplinary AT Assessment.    

Recently things have changed and the world is a different place. Only a few months ago I would have said that the progress of technology was substantially outpacing the rate at which society was changing. Due to Covid-19 society changed dramatically in a matter of weeks and for the first time in a generation, if ever, technology is playing catch-up. Fortunately many of the remote working, collaboration and communication tools required for a post pandemic society were already developed. It did not take long although for the surge of new users to expose flaws in these platforms, either in functionality, privacy or security but Microsoft, Zoom, Google and others have been busy honing their respective tools and in just a few months there have been major improvements. Most of these tools offer video conferencing, screen sharing and sometimes remote control. This means they alone could be enough to provide some remote support, training or assessment. In themselves however they still do not offer anything to enable us to provide support for dedicated hardware AT for communication or computer access or areas like mounting and mobility (wheelchairs and power-chairs). With no end to Covid-19 in sight, at least in the immediate future, social distancing is the new normal and we are hyper-aware of the transmission of viruses and infection control, particularly to those considered medically vulnerable.

Where does this leave the AT assessment process?

The multidisciplinary AT Assessment has long been considered best practice. This is where the AT user, Occupational Therapist, Clinical Engineer or Assistive Technologist, maybe a Speech and Language Therapist and a PA or family member, all get together in a room to talk about goals and try solutions. In current circumstances, this obviously cannot happen. One alternative offered by the three services below might be Augmented Reality (AR) remote support.

Augmented Reality

Augmented Reality is a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view (Oxford Dictionary). One of the best known uses of this technology to date was the game Pokémon Go which was a global success from 2016. The technology has been steadily improving and all the tech giants (Microsoft, Apple, Google) are putting a great deal of research and development into it. Google were early adopters of AR with Google Glass back in 2013. Being first to the market however backfired for them somewhat as consumers weren’t quite ready for Glass as a product, many citing privacy concerns because of the camera. Although while it failed as a consumer product, it found it niche in enterprise and how it is being used for technical support in manufacturing and by doctors within medicine is very close to how it could be used in AT.

You don’t need smart glasses or a head mounted system to make use of AR support (although as we will see, in practice maybe you do). The three solutions below can all be used on a Tablet or SmartPhone. There are however, a number of obvious advantages to using a head mounted camera and display. Both hands are free to work on the task without a second person being needed to hold the mobile device. The visual support is permanently in the eye line with the head mounted display so you don’t have to look away from the task or person you are working with to check instructions. Also the person/people providing support have the best possible view through the head mounted camera, this first person view will include the hands person being supported so instruction or correction can be provided at the right time.

Tablet/SmartPhone Based AR Remote Support

The three apps below offer a similar service, the idea is that the person or people providing support can look through the phone or tablet camera and use drawing tools to mark/highlight objects in the scene. These annotations should then hold their position in real space even if the camera is moved. This last part is crucial but also where (up until recently at least) the technology did not quite perform well enough.

TeamViewer Pilot

TeamViewer is a screen sharing and remote support tool used by IT departments across the world. Teamviewer Pilot is their AR support app that allows you to provide or receive remote support with real world hardware or tasks. There is obvious potential in this technology but from my own, admittedly brief tests, I didn’t find it accurate enough to be of practical use. Since I tested it, however they have harnessed the LIDAR capabilities of the new iPad Pro.  LIRAR which stands for “Light Detection and Ranging” is the same technology that is making autonomous vehicles possible. The LIDAR scanner on the iPad Pro can measure the distance of objects up to five metres away both indoors and outside. The availability of LIDAR on mobile devices and smart glasses will transform AR. It will increase accuracy significantly and certainly warrant another look at TeamViewer Pilot.

Microsoft Dynamics 365 Remote Assist

Microsoft Dynamics 365 Remote Assist seems to offer similar functionality as TeamViewer Pilot. I have not tested the Microsoft Dynamics Remote Assist app so cannot comment on how it compares to Teamviewer Pilot. The ability to call in anyone in your organisation using Microsoft Teams would seem to give it a huge advantage to those already using that communication and collaboration platform. Combined with the HoloLens, Microsoft’s mixed reality head mounted display, this starts to look like a very exciting option (see video below).  

Vuforia Chalk

Android and Chrome screenshots of Chalk
Chalk on Android (left 2 screen shots) and desktop Chrome browser.

Vuforia Chalk is the final AR remote support solution. Although probably the least well know out of the three in mainstream technology Vuforia are well established in the area of AR. My Chalk tests didn’t go well. Both the Android App and the Chrome tab inexplicably closed during the short test. Both computer (Intel Core i5 10th Gen, 8GB RAM) and smartphone (Samsung Galaxy A51) would be mid-range spec and relatively new so if they are having difficulty running the software I can’t imagine it going well on older, lower spec devices. Even when it was running some basic features were unavailable, no screen annotation and it didn’t even recognise when I turned my phone to landscape orientation. As with TeamViewer Pilot this was not an in depth testing of the platform but unlike TeamViewer Pilot, Chalk has given me no reason to consider giving it a second look.

Edit: Looking at my screen shot above, or more specifically the number of Chrome tabs I have open, I might be a little harsh blaming Chalk for crashing..

AR Support Apps Summary

In summary if you are in a position where you have to remotely guide someone in the setup or use of some AT hardware I wouldn’t recommend either of the solutions I tested. TeamViewer Pilot certainly has potential and may in the near future become as indispensable a tool for hardware as their screen-sharing app is for software. For now, sharing video using something like WhatsApp supported by photos returned with annotations is a much better option.

Head Mounted Camera and display

So as we’ve seen above sharing video with live AR annotations using a mobile device isn’t currently stable or mature enough to be of practical use for AT support. I would also suggest that it also doesn’t really make sense. The advantages of the AR aspect are completely offset by having to hold the mobile device in position. This is where smart glasses and head mounted cameras and displays come into their own. None of the options below cost below about €2000 but looking at the demo videos you really get a sense of what this technology could offer a remote assessment. If something like the HoloLens was available to local centres where it could be borrowed by therapists, family or other carers, it would greatly increase the quality and efficiency of AT support. They could call on specialists to provide advice using Teams and provide something close to the multidisciplinary AT assessment or support while limiting direct contact with the AT user.

Realware HMT-1 head mounted camera and display. Thick black band that goes around the back of the head. Thin band for the top of the head. Camera on right pointing forward. Display on arm from right, display positioned below right eye.
Realware HMT-1

Realware HMT-1 works with both TeamViewer Pilot and Microsoft Dynamics Remote Assist. High quality camera, 7” equivalent head mounted display, voice commands

Microsoft HoloLens if using the Microsoft platform the HoloLens would be a much better option. In the US there are packages where you get a HoloLens and Dynamics Remote Assist for a monthly fee. Not sure if they are available in Ireland.

Glass – Google – Certainly the most discrete option here Glass, Google’s enterprise version of their smart glasses from a few years back is still going strong. Exactly what their recent acquisition of smart glasses manufacturer North means is hard to say, other than they are still very interested in AR glasses. North’s AR Glasses stood out because they didn’t look like smart glasses and their use of AR was minimalist, just notifications and some information. They also didn’t have a camera. Google might be getting ready for another push at consumer smart glasses rather than further developing their enterprise version.

Conclusion

It is still early days for this technology. Considering the nature of AT support however and specialists being widely dispersed geographically. A solution like Microsoft Dynamics 365 Remote Assistance combined with a HoloLens starts to make sense when you take into account the increased efficiencies and reduction of travel costs. If we are forced to continue the current practice of social distancing it may become an indispensable tool for an AT service.

Leave a Reply

Your email address will not be published. Required fields are marked *