BACKGROUND: Systematic reviews (SR) are a time-consuming process, with mean estimated time to complete a project reported to be almost 70 weeks. The average aggregated time to collaborate with a librarian to formulate a search strategy is approximately 27 hours, which translates into several days of meetings and communications. This crucial step significantly delays the already lengthy process. Large language models hold promise in playing an influential role in search strategy creation, which relies on well-formulated steps and database-specific syntax. Herein, we report the first use of custom GPT-4 to create a useable search strategy that is sensitive to identifying the majority of relevant articles on previously conducted systematic reviews in urology.
METHODS: A customized version of ChatGPT-4 model from OpenAI (openai.com) was created to develop a search strategy for Ovid databases. The custom GPT was equipped with tailored instructions regarding search strategy creation steps and actions using an application programming interface (API) to search for synonyms/medical terminology, as well as Medical Subject Headings (MeSH) terms. It was also provided with database specific syntax for Scopus database (Figure 1). The authors used five systematic reviews that were previously published by our group to ensure accurate search strategies were available for qualitative comparison.
RESULTS: Search strategy was created using custom GPT-4 model for Ovid Medline and Ovid Embase, and converted to that of Scopus to search all three databases (Example: https://chatgpt.com/share/d94b2d45-b193-484f-b448-32390e017918?oai-dm=1). The sensitivity of search strategies identifying relevant articles included in SRs ranged from 50% to 100%, with an average of 75% (Table 1). Upon comparison of GPT-4 created search strategies and human-performed search strategies, there was less variability with GPT-4, specifically with synonyms of specific medical terms.
CONCLUSIONS:
Our custom GPT-4 model (https://chatgpt.com/g/g-9DTzHJL0V-search-sparrow) can generate a preliminary search strategy with acceptable, albeit less than perfect sensitivity. However, this may be improved in the future with review of the search strategy by a clinical expert to ensure all appropriate synonyms are included. Nonetheless, this investigation points to a promising step in lowering potential barriers to performing SRs to teams with limited resources or less experienced, and provides a good framework to modernize the development of SR.
