5 Best Open Source LLMs (August 2023)

In the rapidly evolving world of artificial intelligence (AI), Large Language Models (LLMs) have emerged as a cornerstone, driving innovations and reshaping the way we interact with technology.

As these models become increasingly sophisticated, there’s a growing emphasis on democratizing access to them. Open-source models, in particular, are playing a pivotal role in this democratization, offering researchers, developers, and enthusiasts alike the opportunity to delve deep into their intricacies, fine-tune them for specific tasks, or even build upon their foundations.

In this blog, we’ll explore some of the top open-source LLMs that are making waves in the AI community, each bringing its unique strengths and capabilities to the table.

Meta’s Llama 2 is a groundbreaking addition to their AI model lineup. This isn’t just another model; it’s designed to fuel a range of state-of-the-art applications. Llama 2’s training data is vast and varied, making it a significant advancement over its predecessor. This diversity in training ensures that Llama 2 is not just an incremental improvement but a monumental step towards the future of AI-driven interactions.

The collaboration between Meta and Microsoft has expanded the horizons for Llama 2. The open-source model is now supported on platforms like Azure and Windows, aiming to provide developers and organizations with the tools to create generative AI-driven experiences. This partnership underscores both companies’ dedication to making AI more accessible and open to all.

Llama 2 is not just a successor to the original Llama model; it represents a paradigm shift in the chatbot arena. While the first Llama model was revolutionary in generating text and code, its availability was limited to prevent misuse. Llama 2, on the other hand, is set to reach a wider audience. It’s optimized for platforms like AWS, Azure, and Hugging Face’s AI model hosting platform. Moreover, with Meta’s collaboration with Microsoft, Llama 2 is poised to make its mark not only on Windows but also on devices powered by Qualcomm’s Snapdragon system-on-chip.

Safety is at the heart of Llama 2’s design. Recognizing the challenges faced by earlier large language models like GPT, which sometimes produced misleading or harmful content, Meta has taken extensive measures to ensure Llama 2’s reliability. The model has undergone rigorous training to minimize ‘hallucinations’, misinformation, and biases.

Top Features of LLaMa 2:

• Diverse Training Data: Llama 2’s training data is both extensive and varied, ensuring a comprehensive understanding and performance.
• Collaboration with Microsoft: Llama 2 is supported on platforms like Azure and Windows, broadening its application scope.
• Open Availability: Unlike its predecessor, Llama 2 is available for a wider audience, ready for fine-tuning on multiple platforms.
• Safety-Centric Design: Meta has emphasized safety, ensuring that Llama 2 produces accurate and reliable results while minimizing harmful outputs.
• Optimized Versions: Llama 2 comes in two main versions – Llama 2 and Llama 2-Chat, with the latter being specially designed for two-way conversations. These versions range in complexity from 7 billion to 70 billion parameters.
• Enhanced Training: Llama 2 was trained on two million tokens, a significant increase from the original Llama’s 1.4 trillion tokens.

Anthropic’s latest AI model, Claude 2, is not merely an upgrade but represents a significant advancement in the capabilities of AI models. With its enhanced performance metrics, Claude 2 is designed to provide users with extended and coherent responses. The accessibility of this model is broad, available both through an API and its dedicated beta website. User feedback indicates that interactions with Claude are intuitive, with the model offering detailed explanations and demonstrating an extended memory capacity.

In terms of academic and reasoning capabilities, Claude 2 has exhibited remarkable achievements. The model achieved a score of 76.5% in the multiple-choice section of the Bar exam, marking an improvement from the 73.0% achieved by Claude 1.3. When benchmarked against college students preparing for graduate programs, Claude 2 performed above the 90th percentile in the GRE reading and writing exams, indicating its proficiency in comprehending and generating intricate content.

The versatility of Claude 2 is another noteworthy feature. The model can process inputs of up to 100K tokens, enabling it to review extensive documents ranging from technical manuals to comprehensive books. Additionally, Claude 2 has the capability to produce extended documents, from official communications to detailed narratives, seamlessly. The model’s coding capabilities have also been enhanced, with Claude 2 achieving a score of 71.2% on the Codex HumanEval, a Python coding assessment, and 88.0% on GSM8k, a collection of grade-school math challenges.

Safety remains a paramount concern for Anthropic. Efforts have been concentrated on ensuring that Claude 2 is less susceptible to generating potentially harmful or inappropriate content. Through meticulous internal evaluations and the application of advanced safety methodologies, Claude 2 has demonstrated a significant improvement in producing benign responses when compared to its predecessor.

Claude 2: Key Features Overview

• Performance Enhancement: Claude 2 delivers faster response times and offers more detailed interactions.
• Multiple Access Points: The model can be accessed via an API or through its dedicated beta website, claude.ai.
• Academic Excellence: Claude 2 has showcased commendable results in academic evaluations, notably in the GRE reading and writing segments.
• Extended Input/Output Capabilities: Claude 2 can manage inputs of up to 100K tokens and is capable of producing extended documents in a single session.
• Advanced Coding Proficiency: The model’s coding skills have been refined, as evidenced by its scores in coding and mathematical evaluations.
• Safety Protocols: Rigorous evaluations and advanced safety techniques have been employed to ensure Claude 2 produces benign outputs.
• Expansion Plans: While Claude 2 is currently accessible in the US and UK, there are plans to expand its availability globally in the near future.

MosaicML Foundations has made a significant contribution to this space with the introduction of MPT-7B, their latest open-source LLM. MPT-7B, an acronym for MosaicML Pretrained Transformer, is a GPT-style, decoder-only transformer model. This model boasts several enhancements, including performance-optimized layer implementations and architectural changes that ensure greater training stability.

A standout feature of MPT-7B is its training on an extensive dataset comprising 1 trillion tokens of text and code. This rigorous training was executed on the MosaicML platform over a span of 9.5 days.

The open-source nature of MPT-7B positions it as a valuable tool for commercial applications. It holds the potential to significantly impact predictive analytics and the decision-making processes of businesses and organizations.

In addition to the base model, MosaicML Foundations is also releasing specialized models tailored for specific tasks, such as MPT-7B-Instruct for short-form instruction following, MPT-7B-Chat for dialogue generation, and MPT-7B-StoryWriter-65k+ for long-form story creation.

The development journey of MPT-7B was comprehensive, with the MosaicML team managing all stages from data preparation to deployment within a few weeks. The data was sourced from diverse repositories, and the team utilized tools like EleutherAI’s GPT-NeoX and the 20B tokenizer to ensure a varied and comprehensive training mix.

Key Features Overview of MPT-7B:

• Commercial Licensing: MPT-7B is licensed for commercial use, making it a valuable asset for businesses.
• Extensive Training Data: The model boasts training on a vast dataset of 1 trillion tokens.
• Long Input Handling: MPT-7B is designed to process extremely lengthy inputs without compromise.
• Speed and Efficiency: The model is optimized for swift training and inference, ensuring timely results.
• Open-Source Code: MPT-7B comes with efficient open-source training code, promoting transparency and ease of use.
• Comparative Excellence: MPT-7B has demonstrated superiority over other open-source models in the 7B-20B range, with its quality matching that of LLaMA-7B.

Falcon LLM, is a model that has swiftly ascended to the top of the LLM hierarchy. Falcon LLM, specifically Falcon-40B, is a foundational LLM equipped with 40 billion parameters and has been trained on an impressive one trillion tokens. It operates as an autoregressive decoder-only model, which essentially means it predicts the subsequent token in a sequence based on the preceding tokens. This architecture is reminiscent of the GPT model. Notably, Falcon’s architecture has demonstrated superior performance to GPT-3, achieving this feat with only 75% of the training compute budget and requiring significantly less compute during inference.

The team at the Technology Innovation Institute placed a strong emphasis on data quality during the development of Falcon. Recognizing the sensitivity of LLMs to training data quality, they constructed a data pipeline that scaled to tens of thousands of CPU cores. This allowed for rapid processing and the extraction of high-quality content from the web, achieved through extensive filtering and deduplication processes.

In addition to Falcon-40B, TII has also introduced other versions, including Falcon-7B, which possesses 7 billion parameters and has been trained on 1,500 billion tokens. There are also specialized models like Falcon-40B-Instruct and Falcon-7B-Instruct, tailored for specific tasks.

Training Falcon-40B was an extensive process. The model was trained on the RefinedWeb dataset, a massive English web dataset constructed by TII. This dataset was built on top of CommonCrawl and underwent rigorous filtering to ensure quality. Once the model was prepared, it was validated against several open-source benchmarks, including EAI Harness, HELM, and BigBench.

Key Features Overview of Falcon LLM:

• Extensive Parameters: Falcon-40B is equipped with 40 billion parameters, ensuring comprehensive learning and performance.
• Autoregressive Decoder-Only Model: This architecture allows Falcon to predict subsequent tokens based on preceding ones, similar to the GPT model.
• Superior Performance: Falcon outperforms GPT-3 while utilizing only 75% of the training compute budget.
• High-Quality Data Pipeline: TII’s data pipeline ensures the extraction of high-quality content from the web, crucial for the model’s training.
• Variety of Models: In addition to Falcon-40B, TII offers Falcon-7B and specialized models like Falcon-40B-Instruct and Falcon-7B-Instruct.
• Open-Source Availability: Falcon LLM has been open-sourced, promoting accessibility and inclusivity in the AI domain.

LMSYS ORG has made a significant mark in the realm of open-source LLMs with the introduction of Vicuna-13B. This open-source chatbot has been meticulously trained by fine-tuning LLaMA on user-shared conversations sourced from ShareGPT. Preliminary evaluations, with GPT-4 acting as the judge, indicate that Vicuna-13B achieves more than 90% quality of renowned models like OpenAI ChatGPT and Google Bard.

Impressively, Vicuna-13B outperforms other notable models such as LLaMA and Stanford Alpaca in over 90% of cases. The entire training process for Vicuna-13B was executed at a cost of approximately $300. For those interested in exploring its capabilities, the code, weights, and an online demo have been made publicly available for non-commercial purposes.

The Vicuna-13B model has been fine-tuned with 70K user-shared ChatGPT conversations, enabling it to generate more detailed and well-structured responses. The quality of these responses is comparable to ChatGPT. Evaluating chatbots, however, is a complex endeavor. With the advancements in GPT-4, there’s a growing curiosity about its potential to serve as an automated evaluation framework for benchmark generation and performance assessments. Initial findings suggest that GPT-4 can produce consistent ranks and detailed assessments when comparing chatbot responses. Preliminary evaluations based on GPT-4 show that Vicuna achieves 90% capability of models like Bard/ChatGPT.

Key Features Overview of Vicuna-13B:

• Open-Source Nature: Vicuna-13B is available for public access, promoting transparency and community involvement.
• Extensive Training Data: The model has been trained on 70K user-shared conversations, ensuring a comprehensive understanding of diverse interactions.
• Competitive Performance: Vicuna-13B’s performance is on par with industry leaders like ChatGPT and Google Bard.
• Cost-Effective Training: The entire training process for Vicuna-13B was executed at a low cost of around $300.
• Fine-Tuning on LLaMA: The model has been fine-tuned on LLaMA, ensuring enhanced performance and response quality.
• Online Demo Availability: An interactive online demo is available for users to test and experience the capabilities of Vicuna-13B.

The Expanding Realm of Large Language Models

The realm of Large Language Models is vast and ever-expanding, with each new model pushing the boundaries of what’s possible. The open-source nature of the LLMs discussed in this blog not only showcases the collaborative spirit of the AI community but also paves the way for future innovations.

These models, from Vicuna’s impressive chatbot capabilities to Falcon’s superior performance metrics, represent the pinnacle of current LLM technology. As we continue to witness rapid advancements in this field, it’s clear that open-source models will play a crucial role in shaping the future of AI.

Whether you’re a seasoned researcher, a budding AI enthusiast, or someone curious about the potential of these models, there’s no better time to dive in and explore the vast possibilities they offer.