Researchers at Google AI Present a Machine Learning-based Approach to Teach Powerful LLMs How to Better Reason with Graph Information

Picture everything in your immediate vicinity, from your friends and family to the utensils in your kitchen and the components of your bicycle. Every one of them is related in some way. The word “graph” describes the relationships between entities in computer science. Nodes are the objects in a graph, whereas edges are the links between them that show their relationship. The very structure of the internet is a vast network of interconnected web pages. The information that search engines rely on is also structured like a graph.

A new Google study aims to train powerful LLMs to reason better with graph information. This is done since graphs are ubiquitous and LLM technology is advancing. While LLMs are often educated on ordinary text, graphs provide a more effective means of organizing information. The objective is to try several approaches to find the most effective ones and get real-world knowledge. Converting graphics into language that LLMs can comprehend is extremely intricate. The intricacy of multi-node graph structures with complex webs of edges connecting them is the root of the problem. This research focuses on methods for converting graphs into a language that LLMs can comprehend.

The researchers first created a benchmark named GraphQA to rigorously determine the optimum method for graph-to-text translation. The researchers rely on a single graph type to build an exhaustive and realistic LLM test; rather, they employ a variety of graphs to guarantee a large number of connections. Certain graph types make these kinds of problems easier or harder to solve. In this approach, GraphQA can reveal biases in an LLM’s analysis of the graphs, and the test becomes more representative of the real-world environment that LLMs may encounter.

Graph QA is concerned with elementary graph operations, such as verifying the existence of an edge, counting the number of edges or nodes, determining which nodes are connected to a given node, and detecting cycles in a graph. Despite their apparent simplicity, these activities necessitate familiarity with the connections between nodes and edges. To teach models how to evaluate graphs efficiently, GraphQA covers a wide range of tasks, from finding patterns to making new connections. More advanced reasoning on graphs, such as discovering communities or determining prominent nodes, relies on these foundational operations. In addition, GraphQA encompasses generating random graphs through several algorithms such as Erdős-Rényi, scale-free networks, the Barabasi-Albert model, and the stochastic block model. It also involves generating simpler graph structures such as routes, full graphs, and star graphs, offering varied data collection for training.

The team investigated various approaches to converting graphs into text that LLMs can process. They conducted three important experiments: one to evaluate LLMs’ performance on graph tasks and two to learn about the effects of LLM size and graph shape on performance. All of their experiments are conducted on GraphQA.

They evaluated the performance of pre-trained LLMs on graph tasks such as cycle detection, node degree estimation, and connection identification. The findings showed that a lot depends on encoding: There is a strong relationship between the graph’s textual representation and LLM performance. In a broad sense, the “incident” encoding performed exceptionally well across the board.

The team conducted this experiment to determine whether LLM performance improves with increasing LLM size (parameter count). To achieve this, they ran the identical graph jobs on four different PaLM 2 sizes: XXS, XS, S, and L. The findings are summarized here:

• When it came to graph reasoning tasks, larger models often performed better. The additional parameters seemed to allow them to learn more intricate patterns.
• Interestingly, the “edge existence” job, which involves determining whether two nodes in a graph are related, was less affected by size.
• When it came to the cycle check problem—determining whether a graph has a cycle—not even the largest LLM could reliably outperform a basic baseline solution. This demonstrates the potential for LLMs to excel in specific graph tasks.

The researchers also explored whether LLMs’ problem-solving abilities on a given graph are affected by its “shape”—the connections between its nodes. The study shows that the structure of graphs significantly affects LLM performance. For instance, LLMs performed admirably on graphs with many closely linked edges (where cycles are abundant) but poorly on path graphs (where cycles never occur) in an exercise testing for the existence of cycles. It was interesting to see how offering a few different instances helped it adjust. For cycle checks, for instance, they included both cycle-containing and cycle-free instances as few-shots in the prompt. 

Findings from this research provide light on the best practices for preparing graphics for LLMs. With the correct encoding methods, an LLM can enhance its accuracy on graph issues by a factor of five to sixty-plus. The researchers hope their new benchmark, GraphQA, will encourage more studies in this field.

Check out the Paper and Blog. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 38k+ ML SubReddit