Reliable Reasoning Beyond Natural Language
저자 Nasim Borazjanizadeh , Steven T. Piantadosi
출판 2024-07-16 목차
Abstract
“
Despite their linguistic competence, Large Language models (LLMs) often exhibit limitations in their ability to reason reliably and flexibly. To address this, we propose a neurosymbolic approach that prompts LLMs to extract and encode all relevant information from a problem statement as logical code statements, and then use a logic programming language (Prolog ) to conduct the iterative computations of explicit deductive reasoning. Our approach significantly enhances the performance of LLMs on the standard mathematical reasoning benchmark, GSM8k , and the Navigate dataset from the BIG-bench dataset. Additionally, we introduce a novel dataset, the Non-Linear Reasoning (NLR) dataset, consisting of 55 unique word problems that target the shortcomings of the next token prediction paradigm of LLMs and require complex non-linear reasoning but only basic arithmetic skills to solve. Our findings demonstrate that the integration of Prolog enables LLMs to achieve high performance on the NLR dataset, which even the most advanced language models (including GPT4) fail to solve using text only.
목차 관련 글 자연어 기반 AI-인간 인터렉션의 한계 이 글을 인용: “Reliable Reasoning Beyond Natural Language (Prolog를 LLM을 위한 정신어로 쓰기)” The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity “Recent generations of frontier language models have introduced Large Reasoning Models (LRMs) that generate deta…” Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens “With this lens, we dissect CoT reasoning via three dimensions: task, length, and format.” Large Language Models are zero-shot reasoners “Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly o…” Tensor Logic: The Language of AI “Most importantly, tensor logic makes new directions possible, such as sound reasoning in embedding space.” Large language models are human-level prompt engineers “Efficient score estimation: 효율을 높이기 위해 더 가능성 있는 후보에게 더 많은 계산 리소스를 할당하고 가능성 낮은 후보를 걸러내는 방식을 적용” 추론 모델 “Reasoning models, like OpenAI o1 and o3-mini, are new large language models trained with reinforcement learning…” Data statements for natural language processing “Data Statements for Natural language processing: Toward Mitigating System Bias and Enabling Better Science” Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for AGI and its Societal Impact “Despite the growing capabilities of models such as GPT-4.5, DeepSeek, Claude 3.5 Sonnet, Phi-4, and Grok 3, whi…” MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark “In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) …” A comprehensive taxonomy of hallucinations in Large Language Models “Large language models (LLMs) have revolutionized natural language processing, yet their propensity for hallucin…” Language Models Know More Than They Show: Exploring Hallucinations From the Model's Viewpoint “Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failur…” Investigating Intersectional Bias in LLMs using Confidence Disparities in Coreference Resolution “Prior work has laid a solid foundation for assessing bias in LLMs by evaluating demographic disparities in diff…” On the Effectiveness of Large Language Models in Writing Alloy Formulas “One, we employ LLMs to write complete Alloy formulas from given natural language descriptions (in English).” Thinking beyond the anthropomorphic paradigm benefits LLM research “For each assumption (e.g., that LLMs must use natural language for reasoning, or that they should be evaluated …” A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions “The emergence of large language models (LLMs) has marked a significant breakthrough in natural language process…” Text-to-LoRA: Instant Transformer Adaption “To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language mode…” Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs “Surprisingly, we observe a significant decline in LLMs reasoning abilities under format restrictions.” iText2KG: Incremental knowledge graphs construction using large language models “KGs also facilitate insights, inference, and reasoning.” Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation “We find incorrect conclusions in approximately 31% of hypotheses for state-of-the-art LLMs, and in half the hyp…”