Facebook AI Introduce ‘SaLinA’: A Lightweight Library To Implement Sequential Decision Models, Including Reinforcement Learning Algorithms

Deep Learning libraries are great for facilitating the implementation of complex differentiable functions. These functions typically have shapes like f(x) → y, where x is a set of input tensors, and y is output tensors produced by executing multiple computations over those inputs. In order to implement a new f function and create a new prototype, one will need to assemble various blocks (or modules) through composition operators. Despite of the easy process, this approach cannot handle the implementation of sequential decision methods. Classical platforms are well-suited for managing the acquisition, processing, and transformation of information in an efficient way.

When it comes to reinforcement learning (RL), these all implementations get critical. A classical deep-learning framework is not enough to capture the interaction of an agent with their environment. Still, extra code can be written that does not integrate well into these platforms. It has been considered to use multiple reinforcement learning (RL) frameworks for these tasks, but they still have two drawbacks:

• New abstractions are being created all the time in order to model more complex systems. However, these new ideas often have a high adoption cost and low flexibility, making them difficult for laypersons who may not be familiar with reinforcement learning techniques.
• The use cases for RL are as vast and varied as the problems it solves. For that reason, there is no one-size-fits all library available on these platforms because each platform has been designed to solve a specific type of problem with their unique features from model-based algorithms through batch processing or multiagent playback strategies, among other things – but they can’t do everything.

As a solution to the above two problems, Facebook researchers introduce ‘SaLinA’. SaLina works towards making the implementation of sequential decision processes, including reinforcement learning related, natural and simple for practitioners with a basic understanding of how neural networks can be implemented. SaLina proposes to solve any sequential decision problem by using simple ‘agents’ that process information sequentially. The targeted audience are not only RL researchers or computer vision researchers, but also NLP experts looking for a natural way of modelling conversations in their models, making them more intuitive and easy to understand than previous methods.

SaLinA is an extension of PyTorch. It has a core code that’s easy to understand and maintain with only hundred lines in total.

Key Advantages of SaLina:

• SaLina is simple to understand and use for models that are on a sequential decision making. There are no hidden mechanisms with this interface so that it will feel familiar in Pytorch. With SaLina you can create complex sequences of decisions easily without getting lost or confused.
• SaLinA allows you to build up complex agents by combining simpler ones with pre-defined containers.
• SaLinA is a very flexible AI framework. It comes with wrappers capturing openAI Gym environments as agents, DataLoader for when you want to develop complex models and Brax environments which allow one quickly implement many different types of architectures using replay buffers so your workspaces can be saved on disk without having them all in memory at once making batch RL much easier than it would otherwise be. 
• SaLinA provides an NRemoteAgent wrapper that can execute any agent over multiple processes, speeding up the computation of particular agents. Used in addition to having an algorithm running on your computer’s CPUs or GPUs with this library, it makes scaling easier by only making minor modifications when necessary.

Paper: https://arxiv.org/pdf/2110.07910.pdf

Github: https://github.com/facebookresearch/salina