Aiding Digital Interface Manipulation with Eye-Tracking and AI.

Reading time: ~3 minutes.

Are digital devices elderly-friendly?

Since my mother started using touchscreen devices, I’ve had the opportunity to observe how the relationship between older adults and technology can be tricky in ways I couldn’t have imagined before.

I assumed the major problem would be understanding the semiotics of the interfaces — the meaning of all the different signs and symbols commonly used in major applications and operating systems.

I was concerned about the lack of conventions regarding the behaviour of interface elements and the gestures a user must learn to master the software.

I was wrong. Or, at least, those weren’t the major problems.

The real problem.

What creates the most difficulties in manipulating touchscreen devices, especially mobiles, are the invisible and sensitive parts that trigger additional options.

The screen, that takes up nearly the entirety of the device, contains the only way to physically interact with the object (excluding voice activated assistants and physical buttons.), and requires a specific type of manipulation.

My father recently began using a tablet to read books, and he ended up placing the device on the desk using a stand to avoid touching it because, being accustomed to handling books, he would place his fingers on the screen, inadvertently triggering a range of unintended functions implemented within the interface.

My mother, on the other hand, despite being able to do whatever she wants with her smartphone, keeps pressing call to actions too slowly, performing a “press and hold” instead of a “tap". The latency, which is well optimised for the target audience of these devices, is too fast and unforgiving for them.

In the "First Principles of Interaction Design," Bruce Tognazzini suggests using an eye-tracking system to improve the precision of link triggering.

He essentially suggests that if you are looking at a certain point and tap on that particular spot, it’s likely that you intend to perform that action.

From this point, I started wondering whether it could be possible to use a similar principle to provide a better, more tailored experience for users. Theoretically, the machine should adapt to the user, not the other way around.

Making AI useful.

By using artificial intelligence (AI), it would be possible to “teach” the device to behave differently with different users, so that my mother’s “press and hold” could be correctly interpreted by the device as a simple “tap.”

This can be useful for a vast number of people e.g. who suffer from motor impairments, both temporary or permanently.

There are many issues to resolve, both in hardware and software, but imagine a device that can recognise you and adapt its speed and sensitivity to your personal way of manipulating it, a device capable of waiting for slower users and speeding up for faster ones, that can distinguish between an intentional and an accidental trigger, even recognising that the swipe you performed, was intended to have a different outcome (e.g. the swipe right on android browser, that have the same effect of a swipe-left, go back. I personally do it all the time).

Of course, the user should be able to enable or disable this feature. The system would require some time to learn (storing information to compare and process the correct response). Perhaps the built-in front-facing camera could be used as a tracking device, and face recognition could help the system identifying whos's using the device in real time, so multiple people - e.g. father and son - can do something together and each one will have a personalised experience.

It should also work in low-light conditions, from an acceptable distance (for example, the face recognition built into some devices requires the user to come closer than 30cm to the sensor for recognition) and with people wearing sunglasses.

Privacy concerns must be taken in consideration, e.g. all data must be strictly stored in the device, in the same way biometric data are kept.

Nevertheless, I believe it’s an idea worth considering (if it's not been done already).