Researchers From China Propose A Novel Lensless Opto-Electronic Neural Network Architecture Called ‘LOEN’ For Machine Vision Applications

This Article Is Based On The Research Paper 'LOEN: Lensless opto-electronic neural network empowered machine vision'. All Credit For This Research Goes To The Researchers 👏👏👏

Please Don't Forget To Join Our ML Subreddit

Due to improvements in the parallelism and processing power of modern graphics processing units (GPUs), deep learning based on convolutional neural networks (CNN) has increased. This has led to effective solutions for a wide range of problems in AI applications. But the vast amounts of data involved in vision processing mean that CNNs can only be used on portable hardware, uses little power and can process data quickly.

Several studies have been done in optical computing to find ways to make electrical neural networks work better. Optical computing has a lot of great benefits, like optical parallelism, which can speed up computing by a lot, and optical passivity, which can lower energy costs and cut down on latency. Optical neural networks (ONNs) are a way to speed up computing and get around the limitations of electrical units’ bandwidth. But ONNs can only work with a coherent laser as a light source, so they can’t be used with a mature machine vision system in scenes with natural light. So, people have come up with hybrid optoelectronic neural networks, in which the front end is optical, and the back end is electrical. These lens-based systems make it harder to use them in edge devices, like self-driving cars.

In a new paper published in Light Science & Application, researchers have developed a lensless optoelectronic neural network (LOEN) architecture for computer vision tasks that uses a passive mask inserted in the imaging light path to perform convolution operations in the optical domain. This reduces the amount of work needed to do calculations and the energy used throughout the whole pipeline. Also, the optical link, image signal processing, and back-end network work together smoothly to achieve joint optimization for specific tasks.

LOEN can work in natural light. A pre-trained optoelectronic neural network determines the mask structure and convolution layer weights. A lightweight network for real-time recognition is constructed for tasks like object classification. Masks are used for feature extraction, functional verification, and improving accuracy. For visual applications like face recognition, global convolution kernel selection and design methods are provided that achieve optical encryption without computing use. The end-to-end network has no private information, such as recognizable facial information, and user privacy can be preserved. LOEN’s lack of a lens structure decreases the system’s volume, and its basic internal design reduces production costs. The unique design cascades all task links and optimizes them with autonomous driving, smart homes, and intelligent security applications.

The geometrical optics theory says that light travels straight so that scenes can be thought of as groups of point light sources. In this paper, instead of the lenses used in traditional machine vision hardware, it is suggested that an optical mask close to the imaging sensor be used instead. The mask modifies the optical signal in space to do the shift and superposition operations on the image sensor. It has been proven that optical masks can replace the convolutional layers of neural networks for extracting features in the optical domain.

For tasks like recognizing handwritten numbers, which are part of object classification, a lightweight network for real-time recognition is built to test how well the optical convolution in the architecture works. With a single convolution kernel, recognition accuracy can be as high as 93.47 percent. When the multi-channel convolution operation is done by putting multiple kernels on the mask simultaneously, the accuracy of classification can be raised to 97.21 percent. It can save about 50 percent of the energy used by traditional machine vision links.

Also, by making the optical mask bigger, the image gets messed up in the optical domain, and the sensor picks up an aliased image that the human eye can’t see. This is a natural way to encrypt private information without using computing power. Face recognition was used to test how well optical encryption worked. Compared to the random MLS pattern, the jointly optimized mask by an end-to-end network was more accurate at recognizing faces by more than 6%. At the same time, encryption was used to protect privacy; it got the same recognition accuracy as methods that didn’t use encryption.

This work suggests a straightforward system for machine vision tasks that calculates the optoelectronic neural network in natural scenes and lets the whole optoelectronic link be optimized as a whole to get the best results for a particular vision task. The all-natural-light neural network will be made when the nonlinear materials are added to the mix. The new architecture could be used in real-world situations, such as self-driving cars, smart homes, and intelligent security.

Optical convolution and optical encryption are machine vision challenges. LOEN simplifies machine vision without imaging. The pipeline is optimized optically and electrically. The total power costs of the sensor and the ISP in an imaging pipeline are comparable. So, when capturing raw data (without ISP), the system saves almost half the energy of standard pipelines. An optimized optical mask replaces digital encryption and provides the same recognition accuracy as no-encryption approaches in privacy-protecting face recognition. Optical convolution encryption offers real-time facial recognition with privacy.

LOEN doesn’t use lenses to convert electrical to optical convolution calculations. Unlike DNNs, the focus is on actual scene visual tasks; hence the system must work with incoherent illumination. All task operations are combined. ISP can be tuned for specific functions to simplify the acquisition and reduce sensor power consumption. The method uses one convolution layer. ONNs are dynamic and nonlinear. When combined with nonlinear materials, such as saturation absorber, optical phase-change memory, and other innovative materials, the nonlinear layer can also be used in the light field. Multi-convolution layers allow for a closed-loop natural light neural network. The calculating speed is accelerated while the energy usage is reduced. When reconfigurable optical elements, such as LCMs or metasurfaces, are added to LOEN, convolution kernels can be programmed. Thus, convolution in space and time can be achieved while the structure can be reused. The technology enables a tiny, intelligent, and low-energy solution for smart devices’ visual duties.

Paper: https://www.nature.com/articles/s41377-022-00809-5

Source: https://www.eurekalert.org/news-releases/951916