EVALUATION OF THE EFFECTIVENESS OF LARGE LANGUAGE MODELS IN THE TASK OF KNOWLEDGE GRAPH COMPLETION: METHODS AND PERSPECTIVES

Abstract

The automation of knowledge graph (KG) completion is a key task in artificial intelligence, with applications in dialogue systems, search engines, and analytical platforms. This study evaluates the effectiveness of large language models (LLMs) in knowledge graph completion (KGC) by analyzing three models: GPT-4o, GPT-3.5-Turbo-0125, and Mixtral-8x7b-Instruct-v0.1.The research assesses model performance in Zero-Shot and One-Shot scenarios and examines the impact of different prompting strategies, including in-context learning (ICL) and chain-of- thought (COT) reasoning. Two specialized datasets are utilized, containing both explicit and implicit entity relationships, enabling the assessment of models’ reasoning capabilities. The analysis employs a strict evaluation paradigm, requiring an exact match between predicted and reference triples, as well as a flexible paradigm, allowing for partial matches and post-processing adjustments. The findings demonstrate that LLMs can be effective in KGC tasks, yet their performance heavily depends on prompt quality, the presence of examples, and output formatting precision.Detailed prompts without examples do not consistently improve results, while Zero-Shot prompting proves less effective than One-Shot approaches. GPT models exhibit greater alignment with given instructions, whereas Mixtral-8x7b tends to include additional explanatory text, making its integration into structured KG systems more challenging. Despite advancements, LLMs still face limitations in output formatting, recognition of implicit relationships, and dependency on prompt formulation. Future research should focus on prompt optimization, refining learning methodologies, and integrating LLMs into more complex KG systems, enhancing the accuracy and efficiency of automated knowledge completion.