Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Simple item page
dc.contributor.authorMarcoci A
dc.contributor.authorWilkinson DP
dc.contributor.authorVercammen A
dc.contributor.authorWintle BC
dc.contributor.authorAbatayo AL
dc.contributor.authorBaskin E
dc.contributor.authorBerkman H
dc.contributor.authorBuchanan EM
dc.contributor.authorCapitán S
dc.contributor.authorCapitán T
dc.contributor.authorChan G
dc.contributor.authorCheng KJG
dc.contributor.authorCoupé T
dc.contributor.authorDryhurst S
dc.contributor.authorDuan J
dc.contributor.authorEdlund JE
dc.contributor.authorErrington TM
dc.contributor.authorFedor A
dc.contributor.authorFidler F
dc.contributor.authorField JG
dc.contributor.authorFox N
dc.contributor.authorFraser H
dc.contributor.authorFreeman ALJ
dc.contributor.authorHanea A
dc.contributor.authorHolzmeister F
dc.contributor.authorHong S
dc.contributor.authorHuggins R
dc.contributor.authorHuntington-Klein N
dc.contributor.authorJohannesson M
dc.contributor.authorJones AM
dc.contributor.authorKapoor H
dc.contributor.authorKerr J
dc.contributor.authorKline Struhl M
dc.contributor.authorKołczyńska M
dc.contributor.authorLiu Y
dc.contributor.authorLoomas Z
dc.contributor.authorLuis B
dc.contributor.authorMéndez E
dc.contributor.authorMiske O
dc.contributor.authorMody F
dc.contributor.authorNast C
dc.contributor.authorNosek BA
dc.contributor.authorSimon Parsons E
dc.contributor.authorPfeiffer T
dc.contributor.authorReed WR
dc.contributor.authorRoozenbeek J
dc.contributor.authorSchlyfestone AR
dc.contributor.authorSchneider CR
dc.contributor.authorSoh A
dc.contributor.authorSong Z
dc.contributor.authorTagat A
dc.contributor.authorTutor M
dc.contributor.authorTyner AH
dc.contributor.authorUrbanska K
dc.contributor.authorvan der Linden S
dc.coverage.spatialEngland
dc.date.accessioned2025-02-14T01:37:07Z
dc.date.available2025-02-14T01:37:07Z
dc.date.issued2024-12-20
dc.description.abstractReplications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
dc.description.confidentialfalse
dc.edition.edition2024
dc.identifier.author-urlhttps://www.ncbi.nlm.nih.gov/pubmed/39706868
dc.identifier.citationMarcoci A, Wilkinson DP, Vercammen A, Wintle BC, Abatayo AL, Baskin E, Berkman H, Buchanan EM, Capitán S, Capitán T, Chan G, Cheng KJG, Coupé T, Dryhurst S, Duan J, Edlund JE, Errington TM, Fedor A, Fidler F, Field JG, Fox N, Fraser H, Freeman ALJ, Hanea A, Holzmeister F, Hong S, Huggins R, Huntington-Klein N, Johannesson M, Jones AM, Kapoor H, Kerr J, Kline Struhl M, Kołczyńska M, Liu Y, Loomas Z, Luis B, Méndez E, Miske O, Mody F, Nast C, Nosek BA, Simon Parsons E, Pfeiffer T, Reed WR, Roozenbeek J, Schlyfestone AR, Schneider CR, Soh A, Song Z, Tagat A, Tutor M, Tyner AH, Urbanska K, van der Linden S. (2024). Predicting the replicability of social and behavioural science claims in COVID-19 preprints.. Nat Hum Behav.
dc.identifier.doi10.1038/s41562-024-01961-1
dc.identifier.eissn2397-3374
dc.identifier.elements-typejournal-article
dc.identifier.issn2397-3374
dc.identifier.pii10.1038/s41562-024-01961-1
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/72495
dc.languageeng
dc.publisherSpringer Nature Limited
dc.publisher.urihttps://www.nature.com/articles/s41562-024-01961-1#Sec32
dc.relation.isPartOfNat Hum Behav
dc.rights(c) 2024 The Author/s
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.titlePredicting the replicability of social and behavioural science claims in COVID-19 preprints
dc.typeJournal article
pubs.elements-id492801
pubs.organisational-groupOther
Files
Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
492801 PDF.pdf
Size:
2.8 MB
Format:
Adobe Portable Document Format
Description:
Evidence
Download
Thumbnail Image
Name:
41562_2024_1961_MOESM1_ESM.pdf
Size:
3.4 MB
Format:
Adobe Portable Document Format
Description:
Evidence
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
9.22 KB
Format:
Plain Text
Description:
Download
Collections
Journal Articles