Skip to content

Unifying Large Language Models and Knowledge Graphs (LLM-KG) for Question Answering (QA)

License

Notifications You must be signed in to change notification settings

machuangtao/LLM-KG4QA

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 

Repository files navigation

LLM-KG4QA: Unifying Large Language Models and Knowledge Graphs (LLM-KG) for Question Answering (QA)

Content


1. Unifying LLMs and KGs for QA

KGs as Background Knowledge

NO Title Venue Year Category Paper Link
1 Deep Bidirectional Language-Knowledge Graph pretraining NeurIPS 2022 Knowledge Integration and Fusion Link
2 GreaseLM: Graph REASoning Enhanced Language Models ICLR 2022 Knowledge Integration and Fusion Link
3 InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration LLM+KG@VLDB 2024 Knowledge Integration and Fusion Link
4 KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation NAACL 2024 Knowledge Integration and Fusion Link
5 KG-Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine-Tuning ACL Findlings 2024 Knowledge Integration and Fusion Link
6 A GAIL Fine-Tuned LLM Enhanced Framework for Low-Resource Knowledge Graph Question Answering CIKM 2024 Knowledge Integration and Fusion Link
7 Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts arXiv 2024 KG-Augmented Prompting Link
8 KnowGPT: Knowledge Graph based Prompting for Large Language Models arXiv 2024 KG-Augmented Prompting Link
9 Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering NLRSE 2023 KG-Augmented Prompting Link
10 Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering IJCKG 2023 KG-Augmented Prompting Link
11 Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation arXiv 2024 Retrieval Augmented Generation Link
12 GRAG: Graph Retrieval-Augmented Generation arXiv 2024 Retrieval Augmented Generation Link
13 KG-RAG: Bridging the Gap Between Knowledge and Creativity arXiv 2024 Retrieval Augmented Generation Link
14 Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering SIGIR 2024 Retrieval Augmented Generation Link
15 Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering CIKM 2024 Retrieval Augmented Generation Link
16 REnhancing Large Language Models with Knowledge Graphs for Robust Question Answering ICPADS 2024 Retrieval Augmented Generation Link
17 FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs arXiv 2025 Retrieval Augmented Generation Link
18 SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation arXiv 2025 Retrieval Augmented Generation Link

KGs as Reasoning Guideline

NO Title Venue Year Category Paper Link
1 Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answerings ACL 2022 Offline KG Guidelines Link
2 keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM arXiv 2023 Offline KG Guidelines Link
3 Explore then Determine: A GNN-LLM Synergy Framework for Reasoning over Knowledge Graph arXiv 2024 Offline KG Guidelines Link
4 Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models arXiv 2024 Offline KG Guidelines Link
5 Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering EMNLP 2022 Online KG Guildlines Link
6 Knowledge-Enhanced Iterative Instruction Generation and Reasoning for Knowledge Base Question Answering NLPCC 2022 Online KG Guildlines Link
7 Evaluating and Enhancing Large Language Models for Conversational Reasoning on Knowledge Graphs arXiv 2023 Online KG Guildlines Link
8 Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph ICLR 2024 Online KG Guildlines Link
9 Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation ICLR 2024 Online KG Guildlines Link
10 KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Grap arXiv 2024 Agent-based KG Guildlines Link
11 ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs ACL Findings 2024 Agent-based KG Guildlines Link

KGs as Refiner and Filter

NO Title Venue Year Category Paper Link
1 Answer Candidate Type Selection: Text-To-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs KONVENS 2023 KG-Driven Filtering and Validation Link
2 Answer Candidate Type Selection: Text-To-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs KONVENS 2023 KG-Driven Filtering and Validation Link
3 KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques BioNLP Workshop 2024 KG-Driven Filtering and Validation Link
4 Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting AAAI 2024 KG-Driven Filtering and Validation Link
5 Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering ariXv 2024 KG-Augmented Output Refinement Link
6 Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models ACL 2024 KG-Augmented Output Refinement Link
7 Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs arXiv 2024 KG-Augmented Output Refinement Link

2. Advanced Topics

Explainable QA

NO Title Venue Year Category Paper Link
1 Retrieval In Decoder benefits generative models for explainable complex question answering Neural Networks 2024 - Link
2 Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering ACL 2024 - Link
3 Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networks SIGIR 2023 - Link

Visual QA

NO Title Venue Year Category Paper Link
1 Lako: Knowledge-driven visual question answering via late knowledge-to text injection IJCKG 2022 - Link
2 Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering ACL 2024 - Link
3 Knowledge-Enhanced Visual Question Answering with Multi-modal Joint Guidance IJCKG 2024 - Link

Conversational and Multi-Hop QA

NO Title Venue Year Category Paper Link
1 Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networks SIGIR 2023 Conversational QA Link
2 Conversational Question Answering with Language Models Generated Reformulations over Knowledge Graph ACL Findings 2024 Conversational QA Link
3 LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments EMNLP 2024 Multi-Hop QA Link
4 Multi-Hop Question Answering with LLMs & Knowledge Graphs Blog 2023 Multi-Hop QA Link
5 GraphLLM: A General Framework for Multi-hop Question Answering over Knowledge Graphs Using Large Language Models NLPCC 2024 Multi-Hop QA Link
6 LLM-KGMQA: Large Language Model-Augmented Multi-Hop Question-Answering System based on Knowledge Graph in Medical Field KBS 2024 Multi-Hop QA Link

Optimization

NO Title Venue Year Category Paper Link
1 Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning arXiv 2023 Index-based Optimization Link
2 Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs ICLR 2024 Index-based Optimization Link
3 Prompting Is Programming: A Query Language for Large Language Models PLDL 2023 Prompting-based Optimization Link
4 LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs ACL Findings 2024 Prompting-based Optimization Link
5 LightRAG: Simple and Fast Retrieval-Augmented Generation arXiv 2024 Graph retrieval-based optimization Link
6 Clue-Guided Path Exploration: Optimizing Knowledge Graph Retrieval with Large Language Models to Address the Information Black Box Challenge arXiv 2024 Graph retrieval-based optimization Link
7 KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques BioNLP Workshop 2024 Ranking-based optimization Link
8 KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering arXiv 2024 Ranking-based optimization Link
9 RAG-based Question Answering over Heterogeneous Data and Text arXiv 2024 Ranking-based optimization Link
10 Cost-efficient Knowledge-based Question Answering with Large Language Models arXiv 2024 Cost-based optimization Link
11 KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs arXiv 2024 Cost-based optimization Link
12 Knowledge Graph-Enhanced Large Language Models via Path Selection ACL Findings 2024 Path-based optimization Link
13 LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration arXiv 2024 Path-based optimization Link
14 Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models arXiv 2024 Query-based optimization Link
15 A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models arXiv 2024 MapReduce-based optimization Link

Data Management

NO Title Venue Year Category Paper Link
1 Triple Augmented Generative Language Models for SPARQL Query Generation from Natural Language Questions arXiv 2024 NL2GQL Link
2 R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL ACL Findings 2024 NL2GQL Link
3 Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL CIKM 2024 NL2GQL Link
4 UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Models arXiv 2024 NL2GQL Link
5 NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language arXiv 2024 NL2GQL Link
6 CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era arXiv 2024 NL2GQL Link
7 SpCQL: A Semantic Parsing Dataset for Converting Natural Language into Cypher CIKM 2022 NL2GQL Link
8 Robust Text-to-Cypher Using Combination of BERT, GraphSAGE, and Transformer (CoBGT) Model Applied Sciences 2024 NL2GQL Link
9 Real-Time Text-to-Cypher Query Generation with Large Language Models for Graph Databases Future Internet 2024 NL2GQL Link
10 LLM4QA: Leveraging Large Language Model for Efficient Knowledge Graph Reasoning with SPARQL Query Journal of Advances in Information Technology 2024 NL2GQL Link
11 Text to Graph Query Using Filter Condition Attributes LSGDA@VLDB 2024 NL2GQL Link
12 Text-to-CQL Based on Large Language Model and Graph Pattern Enhancement PRML 2024 NL2GQL Link
13 Demystifying Natural Language to Cypher Conversion with OpenAI, Neo4j, LangChain, and LangSmith Blog 2024 NL2GQL Link
14 Text2Cypher, the beginning of the Graph + LLM stack Blog 2023 NL2GQL Link
15 Text2Cypher - Natural Language Queries Blog 2023 NL2GQL Link
16 LLaSA: Large Language and Structured Data Assistant arXiv 2024 Structured Data Assistant Link
17 GraphRAG and role of Graph Databases in Advancing AI International Journal of Research in Computer Applications and Information Technology 2024 Graph DB Link

3. Benchmark and Applications

Benchmark Dataset

NO Title Venue Year Dataset Category Paper Link
1 The Value of Semantic Parse Labeling for Knowledge Base Question Answering ACL 2016 WebQSP KBQA and KGQA Link
2 Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs arXiv 2024 CAQA KBQA and KGQA Link
3 G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering arXiv 2024 GraphQA KBQA and KGQA Link
4 Automatic Question-Answer Generation for Long-Tail Knowledge KnowledgeNL@KDD 2023 Long-tail QA KBQA and KGQA Link
5 CPAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering ACL Findings 2024 TemporalQA KBQA and KGQA Link
6 A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases GRADES-NDA 2024 ChatDB LLM and KGs for QA Link
7 XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs EMNLP 2024 XplainLLM LLM and KGs for QA Link
8 Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering SEMANTICS 2023 LLM-KG-Bench LLM and KGs for QA Link
9 Docugami Knowledge Graph Retrieval Augmented Generation (KG-RAG) Datasets - 2023 KG-RAG LLM and KGs for QA -
10 How Credible Is an Answer From Retrieval-Augmented LLMs? Investigation and Evaluation With Multi-Hop QA ACL ARR 2024 - LLM and KGs for QA Link
11 MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge arXiv 2024 MINTQA LLM and KGs for QA Link
12 SituatedQA: Incorporating Extra-Linguistic Contexts into QA EMNLP 2024 SituatedQA Open-retrieval QA Link
13 PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering ACL Findings 2024 PAQA Temporal QA Link
14 BioASQ-QA: A manually curated corpus for Biomedical Question Answering Scientific Data 2023 BioASQ-QA KBQA and KGQA Link
15 MiniRAG: Towards Extremely Simple Retrieval-Augmented Generation arXiv 2025 LiHua-World LLM and KGs for QA Link

Industrial Applications

NO Title Venue Year Github Category Paper Link
1 KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation arXiv 2024 KAG LLM and KGs for QA Link
2 From Local to Global: A Graph RAG Approach to Query-Focused Summarization arXiv 2024 GraphRAG LLM and KGs for QA Link
3 LightRAG: Simple and Fast Retrieval-Augmented Generatio arXiv 2024 LightRAG LLM and KGs for QA Link
4 Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs arXiv 2024 Fact Finder LLM and KGs for QA Link
5 Leveraging Large Language Models and Knowledge Graphs for Advanced Biomedical Question Answering Systems CSA 2024 2024 Cypher Translator LLM and KGs for QA Link
6 A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers arXiv 2024 - LLM and KGs for QA Link
7 Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education arXiv 2024 - LLM and KGs for QA Link

Demo

NO Name Description Source Github
1 GraphRAG-QA An industrial demo of GraphRAG integrating several query engine for augmenting QA, NLP2Cypher-based KG query engine, vector RAG query engine, and Graph vector RAG query engine. NebulaGraph GraphRAG-QA
2 Neo4jRAG-QA This sample application demonstrates how to implement a Large Language Model (LLM) and Retrieval Augmented Generation (RAG) system with a Neo4j Graph Database. Neo4j Graph Neo4j Graph RAG
3 BioGraphRAG This a platform to integrate biomedical knowledge graphs stored in NebulaGraph with LLMs via GraphRAG architecture. NebulaGraph BioGraphRAG
4 kotaemon An open-source clean & customizable RAG UI for chatting with your documents. Built with both end users and developers in mind. Cinnamon AI kotaemon

4. Related Survey

NO Title Venue Year Paper Link
1 Unifying Large Language Models and Knowledge Graphs: A Roadmap TKDE 2024 Link
2 Graph Retrieval-Augmented Generation: A Survey arXiv 2024 Link
3 Retrieval-Augmented Generation with Graphs (GraphRAG) arXiv 2024 Link
4 Multilingual Question Answering Systems for Knowledge Graphs—A Survey Semantic Web 2024 Link
5 Temporal Knowledge Graph Question Answering: A Survey arXiv 2024 Link

About

Unifying Large Language Models and Knowledge Graphs (LLM-KG) for Question Answering (QA)

Topics

Resources

License

Stars

Watchers

Forks