Large Language Models for Scientific Question Answering: An Extensive Analysis of the SciQA Benchmark

Lehmann J.;Meloni A.;Motta E.;Osborne F.;reforgiato Recupero D.

;Salatino A. A.;Vahdati S.

2024-01-01

---

Abstract

The SciQA benchmark for scientific question answering aims to represent a challenging task for next-generation question-answering systems on which vanilla large language models fail. In this article, we provide an analysis of the performance of language models on this benchmark including prompting and fine-tuning techniques to adapt them to the SciQA task. We show that both fine-tuning and prompting techniques with intelligent few-shot selection allow us to obtain excellent results on the SciQA benchmark. We discuss the valuable lessons and common error categories, and outline their implications on how to optimise large language models for question answering over knowledge graphs.