Assessing Large Language Models for SPARQL Query Generation in Scientific Question Answering

Meloni A.;reforgiato Recupero D.

;Osborne F.;Salatino A.;Motta E.;Vahadati S.;Lehmann J.

2025-01-01

---

Abstract

Scientific question answering remains a significant challenge for the current generation of large language models (LLMs) due to the requirement of engaging with highly specialised concepts. A promising solution is to integrate LLMs with knowledge graphs of research concepts, ensuring that responses are grounded in structured, verifiable information. One effective approach involves using LLMs to translate questions posed in natural language into SPARQL queries, enabling the retrieval of relevant data. In this paper, we analyse the performance of several LLMs on this task using two scientific question-answering benchmarks: SciQA and DBLP-QuAD. We explore both few-shot learning and fine-tuning strategies, investigate error patterns across different models, and propose directions for future research.