Meet TextBox 2.0 – A Python Library, Based On PyTorch, For Applying Pre-Trained Language Models To Text Generation

Text generation models or casual language models are used to produce text on par with human-written text. Such related tasks are commonly referred to as “natural language generation.” Text generation is now used in numerous new applications due to recent technical advancements, including machine translation, text summarization, and dialogue systems. Pre-trained language models like BART, GPT, and other GAN-based techniques are some of the most cutting-edge techniques employed for text generation. As a result of numerous such advancements in the field of text creation, there has been an escalating need to develop and evaluate different text generation models in a more unified and trustworthy manner.

After carefully evaluating the speedy advancement of pre-trained language models for text generation, a group of researchers from the Renmin University of China, the University of Montreal, and Xidian University improved on an existing text generation package, TextBox 1.0, to develop TextBox 2.0. TextBox 2.0 significantly improves pre-trained text-generating models compared to its previous iteration. Its primary point of differentiation is that it uses over 45 models, covering 13 tasks and 83 datasets, to implement a unified framework to carry out research on text generation. Existing libraries failed to support the development of models in a unified manner as they did not maintain a comprehensive evaluation pipeline for text generation encompassing data loading, training, and evaluation. This is because they were only intended to handle a few text-generation tasks. 

The three primary additional elements introduced as a part of TextBox 2.0 to support pre-trained language models include generation tasks, generation models, and training strategies. TextBox 2.0 offers 83 datasets for 13 commonly studied text generation applications like text summarization, story generation, etc. To make it even more convenient for the users, the researchers took special care to reorganize the data in a common text-to-text format that the users could easily access using the command line or via a Python API. Moreover, 45 pre-trained language models are also included in the library, serving as an umbrella for diverse models like general, translation, Chinese, dialogue, and other lightweight models. The library offers a common method to compare various models and evaluate the generated text. The library also provides four rich and effective training methodologies and four pre-training objectives to help optimize pre-trained models for text generation. For research purposes, users can either pre-train a brand-new model from scratch or improve a pre-trained model. Such strategies increase the efficiency and dependability of text generation model optimization.

As a part of the evaluation process, the researchers thoroughly tested TextBox 2.0’s text generation capabilities through several experiments. Their tests showed that TextBox 2.0 performed remarkably in terms of computational efficiency in addition to precisely recreating results. This computational efficiency was attained by streamlining the training procedure by cutting back on time spent on unnecessary tasks. Supporting effective decoding techniques also made the library’s generation process noticeably faster.

To put it briefly, TextBox 2.0 is a comprehensive library that can be crucial for conducting further research on pre-trained language model-based text generation. The library includes 45 pre-trained language models in addition to 13 common text generation tasks and their 83 relevant datasets. Additionally, the library establishes unification by supporting the full research pipeline, from data loading through training and evaluation, and guaranteeing that each step is completed uniformly. The thorough testing conducted by the researchers concluded that TextBox 2.0 could generate outcomes comparable to and occasionally even better than the original implementations. In conclusion, the researchers believe that TextBox 2.0 will be a useful instrument for amateur researchers and those just starting out to learn more about and explore text generation models and encourage further research in this domain.  

Check out the Paper and Github. All Credit For This Research Goes To Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.