Can large language models help predict results from a complex behavioural science study?

Simple item page
dc.citation.issue9
dc.citation.volume11
dc.contributor.authorLippert S
dc.contributor.authorDreber A
dc.contributor.authorJohannesson M
dc.contributor.authorTierney W
dc.contributor.authorCyrus-Lai W
dc.contributor.authorUhlmann EL
dc.contributor.authorEmotion Expression Collaboration
dc.contributor.authorPfeiffer T
dc.coverage.spatialEngland
dc.date.accessioned2024-10-30T21:52:40Z
dc.date.available2024-10-30T21:52:40Z
dc.date.issued2024-09
dc.description.abstractWe tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate-at scale and minimal cost-which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human-AI collaboration in science.
dc.description.confidentialfalse
dc.edition.editionSep 2024
dc.format.pagination240682-
dc.identifier.author-urlhttps://www.ncbi.nlm.nih.gov/pubmed/39323554
dc.identifier.citationLippert S, Dreber A, Johannesson M, Tierney W, Cyrus-Lai W, Uhlmann EL, Emotion Expression Collaboration , Pfeiffer T. (2024). Can large language models help predict results from a complex behavioural science study?. R Soc Open Sci. 11. 9. (pp. 240682-).
dc.identifier.doi10.1098/rsos.240682
dc.identifier.eissn2054-5703
dc.identifier.elements-typejournal-article
dc.identifier.issn2054-5703
dc.identifier.piirsos240682
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/71876
dc.languageeng
dc.publisherThe Royal Society
dc.publisher.urihttps://royalsocietypublishing.org/doi/10.1098/rsos.240682
dc.relation.isPartOfR Soc Open Sci
dc.rights(c) 2024 The Author/s
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectforecasting
dc.subjectlarge language models
dc.subjectmeta-research
dc.titleCan large language models help predict results from a complex behavioural science study?
dc.typeJournal article
pubs.elements-id491662
pubs.organisational-groupCollege of Health
Files
Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
Published version.pdf
Size:
902.64 KB
Format:
Adobe Portable Document Format
Description:
491662 PDF.pdf
Download
Thumbnail Image
Name:
Evidence.docx
Size:
899.12 KB
Format:
Microsoft Word XML
Description:
rsos240682_si_001.docx
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
9.22 KB
Format:
Plain Text
Description:
Download
Collections
Journal Articles