Innovative Bio-Inspired Sensor Detects Motion and Predicts Trajectories for Various Applications

Researchers at Aalto University have developed an innovative bio-inspired sensor that can detect moving objects in a single video frame and accurately predict their future movements. Described in a Nature Communications paper, this advanced sensor has numerous potential applications in fields such as dynamic vision sensing, automatic inspection, industrial process control, robotic guidance, and autonomous driving technology.

Traditional motion detection systems require numerous components and complex algorithms that perform frame-by-frame analyses, resulting in inefficiency and high energy consumption. To address these limitations, the Aalto University team looked to the human visual system for inspiration and created a neuromorphic vision technology that unifies sensing, memory, and processing into a single device capable of detecting motion and predicting trajectories.

Photomemristors: The Core of the New Technology

The researchers’ technology is built on an array of photomemristors, electrical devices that generate electric current in response to light. Photomemristors possess a unique characteristic: the current doesn’t stop immediately when the light is turned off, but decays gradually. This feature allows photomemristors to effectively “remember” their recent exposure to light, enabling a sensor composed of an array of these devices to capture not only instantaneous information about a scene but also a dynamic memory of preceding moments.

“The unique property of our technology is its ability to integrate a series of optical images in one frame,” explains Hongwei Tan, the research fellow who led the study. “The information of each image is embedded in the following images as hidden information. In other words, the final frame in a video also has information about all the previous frames. That lets us detect motion earlier in the video by analyzing only the final frame with a simple artificial neural network. The result is a compact and efficient sensing unit.”

Demonstrating the Technology’s Capabilities

To showcase their technology, the researchers used videos that displayed the letters of a word one at a time. Although all the words ended with the letter “E,” conventional vision sensors couldn’t discern whether the “E” on the screen had followed the other letters in “APPLE” or “GRAPE.” However, the photomemristor array could utilize hidden information in the final frame to deduce which letters had preceded it and predict the word with nearly 100% accuracy.

In another experiment, the team showed the sensor videos of a simulated person moving at three different speeds. The system could not only recognize motion by analyzing a single frame, but it also accurately predicted subsequent frames.

Implications for Autonomous Vehicles and Intelligent Transport

Accurate motion detection and trajectory prediction are crucial for self-driving technology and intelligent transport systems. Autonomous vehicles rely on precise predictions of how cars, bikes, pedestrians, and other objects will move in order to make informed decisions. By incorporating a machine learning system into the photomemristor array, the researchers demonstrated that their integrated system could predict future motion based on in-sensor processing of an all-informative frame.

“Motion recognition and prediction by our compact in-sensor memory and computing solution provides new opportunities in autonomous robotics and human-machine interactions,” says Professor Sebastiaan van Dijken. “The in-frame information that we attain in our system using photomemristors avoids redundant data flows, enabling energy-efficient decision-making in real time.”