What is Fine Tuning and Best Methods for Large Language Model (LLM) Fine-Tuning

Large Language Models (LLMs) such as GPT, PaLM, and LLaMa have made major advancements in the field of Artificial Intelligence (AI) and Natural Language Processing (NLP) by enabling machines to comprehend and produce content that is similar to that of humans. These models possess an extensive comprehension of language and its subtleties, having been trained on massive amounts of data. However, their generalist character frequently proves inadequate when used for specialized activities or domains. This is where finetuning enters the picture, which is a crucial procedure that greatly improves the model’s performance.

What is Fine Tuning?

Finetuning is a way to modify a language model that has already been taught to perform well in a certain area. Even though LLMs have remarkable comprehension and production skills, they are not naturally suited to tackle specialized activities accurately. By retraining the model on a more manageable, domain-specific dataset, finetuning overcomes this constraint and enables the model to acquire the nuances and distinctive features of the intended field.

A pre-trained model with a broad grasp of language is the starting point for finetuning. This model is finetuned by subjecting it to a carefully selected dataset. The model modifies its internal parameters, such as weights and biases, through this exposure to better match the data’s characteristics. This specialized training phase greatly enhances the model’s performance on tasks linked to the domain, which helps the model understand the intricacies, vocabulary, and context.

Fine Tuning Approaches

1. Parameter Efficient Fine Tuning (PEFT) 

Reducing the trainable parameters in a neural network makes the training process more computationally efficient, and this is the main notion underlying PEFT. LoRA and QLoRA are a few prominent PEFT approaches.

a) LoRA 

Low-Rank Adaptation, or LoRA, is a PEFT method that operates as an adapter-based strategy. LoRA simply adds new parameters during the training phase, never permanently changing the model architecture. This method enables parameter-efficient finetuning without adding more parameters to the model overall.

LoRA divides the weight update matrix into two smaller matrices, A and B, each of which has a rank parameter ‘r.’ This allows for parameter efficiency. The rank parameter determines the size of these smaller matrices. The weight update matrix has the same size as the number of parameters that need to be updated during finetuning, and it basically represents the modifications learned through backpropagation. These smaller matrices help the model be trained using standard backpropagation. 

b) QLoRA

Quantized LoRA, often known as QLoRA, is an improvement on LoRA that combines low-precision storage with high-precision computation techniques. The goal of this combination is to maintain good accuracy and performance while keeping the model small.

To accomplish its objectives, QLoRA presents two crucial concepts,  i.e., Normal Float for 4 bits, in which numerical values are represented using a 4-bit normal float representation, and Double quantization, which includes quantizing both the learning rate and the model parameters. 

2. Supervised finetuning

Supervised finetuning is a method of optimizing LLMs using task-specific labeled datasets. The foundation of this approach is the idea that every input data point in these datasets is labeled with an accurate label or response, acting as a final manual for the model to follow during its learning phase. The model is motivated to modify its internal parameters in order to achieve high-accuracy label prediction through supervised fine-tuning. This uses the model’s huge knowledge base, which it gathered from large datasets during its initial pre-training phase, and refines it to the particulars and demands of the intended task. 

a) Basic Hyperparameter Tuning

Using this fundamental method, the model’s hyperparameters and important variables that control the training process, like learning rate, batch size, and number of training epochs, are carefully adjusted. The essence of basic hyperparameter tweaking is finding the ideal mix of these parameters that enables the model to learn from the task-specific data most effectively. This significantly increases learning efficacy, improving the model’s task-specific performance while reducing the likelihood of overfitting.

b) Transfer Learning

Transfer learning is particularly useful when there is a shortage of task-specific data. It begins with a pre-trained model on a large-scale, widely-used dataset. The smaller, task-specific dataset is then used to refine this model. Utilizing the model’s previously gained, broad information and tailoring it to the new task is the essence of transfer learning. In addition to saving time and training resources, this method frequently produces better outcomes than creating a model from scratch.

c) Few-shot learning

Few-shot learning enables a model to rapidly adjust to a new task using the least amount of task-specific data possible. By utilizing the model’s vast pre-trained knowledge base, it can understand the new task in a few instances. This approach is helpful when gathering a sizable labeled dataset for the new task is not feasible. The foundation of few-shot learning is the idea that a limited number of examples given during inference can successfully direct the model’s comprehension and execution of the novel job.

3. Reinforcement Learning from Human Feedback (RLHF) 

RLHF is an approach to language model training that integrates human evaluation skills and sophisticated comprehension into machine learning. This technology allows language models to be dynamically improved, resulting in outputs that are accurate, socially and contextually suitable. The key to RLHF is its capacity to combine the algorithmic learning powers of models with the subjective assessments of human feedback, allowing the models to develop more naturally and more responsively.

a) Reward modeling

By exposing the model to a range of possible reactions, reward modeling involves assessing the model’s performance through human evaluation. A variety of factors, such as appropriateness, coherence, and relevance, are taken into consideration by the evaluators when rating or ranking these outputs. The model is then trained as a reward function using human input as it learns to predict the rewards for various outputs depending on human evaluations. The model uses this learned reward function as a guide to modify its outputs over time to maximize these rewards from humans.

b) Proximal Policy Optimisation

Within the RLHF paradigm, Proximal Policy Optimisation is a more technical step that focuses on improving the model’s decision-making policy iteratively in order to improve the expected reward outcomes. The key to PPO’s effectiveness is its deliberate approach to policy updates, which attempts to make modifiable but cautiously incremental changes to the model’s policy to prevent dramatic shifts that can upset the learning trajectory. 

An objective function that has been created and incorporates a clipping method to control the policy update rate accomplishes this. By doing this, PPO guarantees that the policy updates retain a controlled and steady advancement in learning by not deviating too much from the prior policy iteration, even while they are still significant enough to contribute to learning. PPO’s constraint mechanism is essential to its effectiveness because it fosters a steady and balanced learning process that is less vulnerable to the dangers of unpredictable policy changes.

References 

1. https://www.turing.com/resources/finetuning-large-language-models
2. https://www.analyticsvidhya.com/blog/2023/08/lora-and-qlora/
3. https://medium.com/@sujathamudadla1213/difference-between-qlora-and-lora-for-fine-tuning-llms-0ea35a195535
4. https://www.analyticsvidhya.com/blog/2023/08/fine-tuning-large-language-models/
5. https://www.signalfire.com/blog/comparing-llm-fine-tuning-methods