Many programs come with dozens of buttons, menus and options, and most of us use only a few of them. For someone who controls a computer with their eyes, a switch or a touchscreen, every extra button makes the ones they need harder to reach. And for someone who finds too much choice confusing or tiring, a crowded screen with lots of options can be a huge barrier. Imagine being offered only the controls you need.
That is what you can see in this video: a browser game played with a few large, clearly labelled buttons on the screen. An AI assistant designed them in a few minutes, from a short description and a screenshot of the game’s controls.
Why this matters
OptiKey is a free on-screen keyboard that people control with their eyes, head, mouse, a joystick or a touchscreen. It can do much more than type. The charity SpecialEffect built EyeMine from it, so that people can play Minecraft with their eyes.
Software such as Grid 3 and Communicator lets you build grids of big buttons that control other programs. OptiKey has a similar idea, called dynamic keyboards. It has been there for years, but I think most people have never used it. In my experience, that is because the keyboards had to be written by hand, in a plain text file, which is a little outside most people’s comfort zone.
If writing code is not how you want to spend an afternoon, that is fine. It is exactly the part an AI assistant can take over.
What you do, and what the AI does
The AI assistant creates the keyboards, you open them in Optikey Pro and test with the app you want to control.
You tell it three things:
- Which program you want to control.
- Which buttons you want (up, down, jump) or even a general aim (play the game, control music, make a call).
- How the person will use it: eye gaze, head mouse, touchscreen, switch or mouse.
If it helps, you can add a screenshot of the program’s controls or ask the assistant to look up the program’s keyboard shortcuts. You can even ask the assistant to help you figure out the controls needed to successfully complete the chosen task or activity.
The AI assistant then:
- Decides how to lay out the buttons.
- Writes the files that OptiKey needs.
- Checks them for mistakes.
- Gives you a file to download.
Then you:
- Download the file and extract it into a folder.
- Tell OptiKey where the folder is.
- Try the buttons.
What you need
- A Windows computer with OptiKey Pro. OptiKey is free. Dynamic keyboards are documented for OptiKey Pro, so check which version you have.
- An AI assistant that can take a file and give files back. We tested the build document with Claude AI (pro account) and ChatGPT (free account). Other assistants should work, but we have not tested them.
- Our build document. It is a short instruction file that you give to the assistant. It tells the assistant how OptiKey keyboards work, shows it working examples, and asks it to hand you the finished files as downloads.
- A supporter, the first time. OptiKey needs setting up once for the way the person selects keys, such as eye gaze or touch. The OptiKey wiki explains this, and it is easier with someone alongside.
The prompt to use
Upload the build document to your assistant, then paste in the message below, or something similar in your own words. Fill in the square brackets and delete anything that does not apply.
I want to create an alternative accessible interface for [application] using OptiKey dynamic keyboards.
The app functions I want to make available are: I want to be able to [general aim or list the functions].
These dynamic keyboards will be accessed using: [Mouse / Eye tracker / Touchscreen / Switch / Joystick - delete those that do not apply].
I have uploaded a file with more instructions and examples. Please give me a first-pass design of the interface.That is the whole job. Everything else is in the build document. You should get a picture of the layout in words, the files to download, short set-up steps and a list of things to try. Treat it as a first draft and test every button.
Three examples
A game controller
The video above shows a browser game, controlled by on-screen buttons that the assistant designed from a screenshot of the game’s controls. It worked, although browser games can be hit and miss. A game like this is a good place to start because it uses just a few keys. Also, a lot of classic console games have started to appear as emulations playable in a web browser. While most are illegal copies at this point it’s only a matter of time before the big companies start doing the same. The picture below shows the keys that could be used for a PlayStation 1 controller.
A music remote
This was our main project. We built a simple remote for the Spotify desktop app, with big, labelled buttons for previous, play and pause, next, quieter, mute and louder, and a screen of favourite artists, albums and playlists. We tested it on Windows 10 and Windows 11.
The playback buttons were the easy part. Starting a favourite playing took much more trial and error, because Spotify opens a page but does not start it. If you would like a more detailed explanation of how we overcame this, please let us know in the comments.
One that only partly worked
As a test I got ChatGPT to have a go at both a Microsoft Teams interface and a WhatsApp interface, which was ambitious on both counts. On Teams some of the buttons did not work on the first test and both would take a lot more testing before making them available for general use. It would certainly be worth the effort if someone is looking for a little project. Optikey can only offer buttons for functions that have keyboard shortcuts, which is the main limitation of this approach.
What to expect
This is easy but “easy” can mean different things to different people.
- Making the keyboard is easy. Describing what you want and downloading the result takes minutes, and you do not need to read or write code.
- It is a first draft. The assistant may get something wrong. Test every button, ideally with the person who will use it, and ask for changes in plain words.
- It works best with programs that have clear keyboard shortcuts. The buttons press keys on the program’s behalf. If the program has no shortcut for something, the assistant cannot make a button for it.
- Browser games can be hit and miss. Mainly because most are not designed with any thought towards accessibility or keyboard support. A docked keyboard did not work for us with a game in full-screen mode. A windowed game worked better.
- Be careful with programs that can message other people. A button pressed by accident could send something to someone. Try a messaging program with a trusted person first and keep the buttons few and clearly labelled.
- Programs change. If an update moves things around, a button may stop working. Keep the downloaded files, so you can ask the assistant to fix or rebuild them.
- OptiKey has to be set up first. That is a one-off job and the OptiKey wiki covers it.
Who can use it?
Anybody who can use OptiKey. That includes eye gaze, head mouse, joystick, touchscreen, switch and mouse. We tested a touchscreen. We suspect a switch user could also use it with OptiKey set for game controller access, but we have not tested that.
Setting up OptiKey’s input is outside the scope of this post. These OptiKey wiki pages cover it:
Downloads
- The build document. Give this to your AI assistant. If you click the following link you can view it in your browser, then copy into Notepad. You can also right click and save the file directly. Download OptiKey_Dynamic_Keyboard_Builder.md.
- The Spotify remote, tested. Four keyboards in one zip, tested on Windows 10 and Windows 11 with the Spotify desktop app. The Music_Launcher.xml keyboard should be set as the start up keyboard in Optikey (unless you already use Optikey for computer access or AAC in which case add to dynamic Keyboard menu). The favourites screen has three working examples and some empty spaces for you to fill in. Download Spotify Keyboards
Keep your own copy of anything you download and test each file before someone relies on it.
Try it
Pick one program that someone you support finds hard to use, and one thing they would like to do with it. Describe just that to your assistant. Start with two or three buttons, test them, and add more once they work.
If you build something, we would like to hear about it, please tell us in the comments below.