ETH Zürich Team Introduces A Novel Method To Decode Text From Accelerometer Signals Sensed At The User’s Wrist Using A Wearable Device

This Article Is Based On The Research 'TapType: Ten-finger text entry on everyday surfaces via Bayesian inference'. All Credit For This Research Goes To The Researchers of This Project 👏👏👏

Please Don't Forget To Join Our ML Subreddit

Many surveys show that despite the introduction of touchscreens, typing on physical keyboards remains the most efficient method of entering text since. This is because users have the scope of using all of their fingers over a full-size keyboard. Text input on mobile and wearable devices has compromised on full-size typing as users increasingly type on the go. 

New research by the Sensing, Interaction & Perception Lab at ETH Zürich present TapType, a mobile text entry system that allows full-size typing on passive surfaces without using a keyboard. Their paper “TapType: Ten-finger text entry on everyday surfaces via Bayesian inference” explains that two bracelets that makeup TapType detect vibrations caused by finger taps. It distinguishes itself by combining the finger probabilities from the Bayesian neural network classifier with the characters’ prior probability from an n-gram language model to forecast the most likely character sequences.

The hardware is a simple design based on the Dialog DA14695, a good Cortex M33-based Bluetooth Low Energy SoC that contains a “sensor node controller” block that can handle sensor devices connected to its interfaces independently of the main CPU. The Bosch BMA456 3-axis accelerometer was utilized, and it is renowned for its low power usage of only 150 A.

TapType’s processing pipeline is divided into three sections:

1. A tap detection algorithm that detects abrupt changes in IMU signals
2. Classification network that estimates probabilities across the five fingers and palm
3. A decoder converts the classifier’s output sequence with priors from an n-gram language model to the most likely character sequence. 

The team tested numerous architectures with different Bayesian layer placements to select effective probability distributions for the decoder. Their findings show that 2-Bayes produced the best accuracy and robustness. 

To collect, the team enlisted the help of many volunteers. They had to type sentences on an A3-sized piece of paper using a QWERTY keyboard. A capacitive touch sensor was inserted beneath the printed keyboard to provide ground-truth touch events and locations. The OptiTrack was used to record the fingertip motions as well as the IMU feeds from both TapType bracelets. 

On evaluation, the researchers found that after 30 minutes of training, participants averaged 19 words per minute with a character mistake rate of 0.6 percent. Furthermore, expert typists could attain more than 25 WPM while maintaining a similar error rate. 

The researchers state that TapType can be used in many applications, such as:

• Text entry in mobile environments such as around smartphones and tablets
• As a supplement to engagement in located Mixed Reality without visual control
• As an eyes-free mobile text input method with an aural feedback-only interface. 

Project: https://siplab.org/projects/TapType

Paper: https://siplab.org/papers/chi2022-taptype.pdf

Source: https://hackaday.com/2022/05/14/taptype-ai-assisted-hand-motion-tracking-using-only-accelerometers/