PhD Scientific Days 2026

Pharmaceutical Sciences and Health Technologies 1.

REFLECTIVE-TIAB: Agentic Reflective Prompt Evolution for Cost-Effective Large-Scale Title and Abstract Screening

Előadó neve

Dr. Imre, Attila

Neptune code

BHHASB

Előadó munkahelye

Center for Health Technology Assessment

Előadó telefonszáma

+36309599637

Előadó e-mail címe

imre.attila@stud.semmelweis.hu

Az előadás címe

REFLECTIVE-TIAB: Agentic Reflective Prompt Evolution for Cost-Effective Large-Scale Title and Abstract Screening

Szerző(k) neve és munkahelye

Attila Imre^1,2,3,4, Ákos Józwiak^1,2, Judit Hagymásy^1,2, Judit Tittmann¹, Ágnes Nagy¹, Sándor Kovács^1,2, Przemyslaw Kardas⁵, Job FM van Boven⁶, Irene Mommers⁶, Balázs Nagy^2,3,4, Tamás Ágh^1,2

1: Center for Health Technology Assessment and Pharmacoeconomic Research, University of Pecs, Hungary
2: Syreon Research Institute, Budapest, Hungary
3: Center for Health Technology Assessment, Semmelweis University, Budapest, Hungary
4: Center for Pharmacology and Drug Research & Development, Semmelweis University, Budapest, Hungary
5: Department of Family Medicine, Medical University of Lodz, Lodz, Poland
6: Department of Clinical Pharmacy & Pharmacology, Groningen Research Institute for Asthma and COPD (GRIAC), University Medical Center Groningen, University of Groningen, Groningen, the Netherlands

Bemutatás módja

Szóbeli

Szekció

Pharmaceutical Sciences and Health Technologies 1.

Language of the presentation

English

Preferred session

Pharmaceutical Sciences and Health Technologies

Összefoglaló szövege

Introduction: Title and abstract screening is a labour-intensive stage of systematic reviews. Large language models (LLMs) can automate this process, but performance depends heavily on prompt design and model selection, which is typically manual and time-consuming.
Aims: Our objective was to evaluate whether automated, reflection-driven prompt optimisation improves LLM performance during title and abstract screening.
Method: REFLECTIVE-TIAB uses the GEPA reflective prompt optimiser to improve prompts under an asymmetric loss penalising false negatives. Nine LLMs screened 8,520 de-duplicated records from a COPD exacerbation predictor search. A 100-abstract gold standard was constructed from inter-model disagreements and was expert-labelled. The prompt was optimised on Llama 3.3 70B via DSPy/GEPA and evaluated across all nine models.
Results: Optimisation improved recall across all LLMs (+3.7% to +37.1%). Gemini 3 Flash Preview achieved the highest performance (91% accuracy, F1 81.6%) while costing 25-fold less per abstract than GPT-5.2, which ranked among the lowest-performing models. A prompt optimised on a single open-source model generalised to all nine without retraining. Total optimisation cost was $6.36.
Conclusion: REFLECTIVE-TIAB provides automated, model-transferable prompt optimisation for literature screening at negligible cost. Model price did not predict screening performance. The framework could substantially reduce screening workload while preserving comprehensiveness.
Funding: This research is part of the COPD-ALERT project. The “COPD-ALERT - Prediction of COPD exacerbations through artificial intelligence based monitoring of medication adherence and other medical data” project is granted by the 2024-1.2.3-HU-RIZONT International Excellence Program (National Research, Development and Innovation Office – NKFIH). Supported by the 2025-2.1.1-EKÖP-2025-00014 University Research Scholarship Programme of the Ministry for Culture and Innovation from the source of the National Research, Development and Innovation Fund.

University

Semmelweis University

Supervisor

Dr. Balázs Nagy PhD

Publication of my abstract

I give consent to the publication of my abstract on the website of the congress.

phd.section.field

in doctoral studies after complex exam (PhD)

Kind

Szabad

Status

elfogadva

Accepted presentation method

szóbeli

Előadás fájl jóváhagyás

nem rendelkezett róla

Előadó

9681

Start

15:00

End

15:10