This AI Paper Presents Find+Replace Transformers: A Family of Multi-Transformer Architectures that can Provably do Things no Single Transformer can and which Outperform GPT-4 on Several Tasks

In the annals of computational history, the journey from the initial mechanical calculators to Turing Complete machines has been revolutionary. While impressive, early computing devices, such as Babbage’s Difference Engine and the Harvard Mark I, lacked the Turing Completeness—a concept defining systems capable of performing any conceivable calculation given adequate time and resources. This limitation was not just theoretical; it delineated the boundary between simple automated calculators and fully-fledged computers capable of executing any computation task. Turing Complete systems, as conceptualized by Alan Turing and others, brought about a paradigm shift, enabling the development of complex, versatile, and composable software.

Fast forward to the present, the realm of Natural Language Processing (NLP) has been dominated by transformer models, celebrated for their prowess in understanding and generating human language. However, a lingering question has been their ability to achieve Turing Completeness. Specifically, could these sophisticated models, foundational to Large Language Models (LLMs), replicate the limitless computational potential of Turing Complete systems?

This paper aims to address this question, scrutinizing the transformer architecture’s computational boundaries and proposing an innovative pathway to transcend these limits. The core assertion is that while individual transformer models, as currently designed, fall short of Turing Completeness, a collaborative system of multiple transformers could cross this threshold.

The exploration begins with a dissection of computational complexity, a framework that categorizes problems based on the resources needed for their resolution. It’s a critical analysis as it lays bare the limitations of models confined to lower complexity classes—they cannot generalize beyond a certain scope of problems. This is vividly illustrated through the example of lookup tables, simple yet fundamentally constrained in their problem-solving capabilities.

Diving deeper, the paper highlights how transformers, despite their advanced capabilities, encounter a ceiling in their computational expressiveness. This is exemplified in their struggle with problems that exceed the REGULAR class within the Chomsky Hierarchy—a classification of language types based on their grammatical complexity. Such challenges underscore the inherent limitations of transformers when faced with tasks that demand a degree of computational flexibility they inherently lack.

However, the narrative takes a turn with the introduction of the Find+Replace Transformer model. This novel architecture reimagines the transformer’s role not as a solitary solver but as part of a dynamic duo (or more accurately, a team) where each member specializes in either identifying (Find) or transforming (Replace) segments of data. This collaborative approach not only sidesteps the computational bottlenecks faced by standalone models but also aligns closely with the principles of Turing Completeness.

The elegance of the Find+Replace model lies in its simplicity and its profound implications. By mirroring the reduction processes found in lambda calculus—a system foundational to functional programming and Turing Complete by nature—the model demonstrates a capability for unlimited computation. This is a significant leap forward, suggesting that transformers, when orchestrated in a multi-agent system, can indeed simulate any Turing machine, thereby achieving Turing Completeness.

Empirical evidence bolsters this theoretical advancement. Through rigorous testing, including challenges like the Tower of Hanoi and the FAITH and FATE tasks, the Find+Replace transformers consistently outperformed their single-transformer counterparts (e.g., GPT-3, GPT-3.5 and GPT-4). These results (shown in Table 1 and Table 2) validate the model’s theoretical underpinnings and showcase its practical superiority in tackling complex reasoning tasks that have traditionally impeded state-of-the-art transformers.

In conclusion, the finding that traditional transformers are not Turing-complete underscores their potential limitations. This work establishes Find+Replace transformers as a powerful alternative, pushing the boundaries of computational capability within language models. The attainment of Turing completeness lays the groundwork for AI agents designed to execute broader computational tasks, making them adaptable to solving increasingly diverse problems.

This work calls for continued exploration of innovative multi-transformer systems. In the future, more efficient versions of these models may offer a paradigm shift beyond single-transformer limitations. Turing-complete transformer architectures unlock vast potential, laying the path toward new frontiers in AI.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel