Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Simple item page
dc.citation.issue4
dc.citation.volume16
dc.contributor.authorGordon M
dc.contributor.authorViganola D
dc.contributor.authorDreber A
dc.contributor.authorJohannesson M
dc.contributor.authorPfeiffer T
dc.coverage.spatialUnited States
dc.date.accessioned2023-11-30T01:43:54Z
dc.date.accessioned2024-07-25T06:44:05Z
dc.date.available2021-04-14
dc.date.available2023-11-30T01:43:54Z
dc.date.available2024-07-25T06:44:05Z
dc.date.issued2021-04-14
dc.description.abstractThe reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package “pooledmaRket” and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.
dc.format.paginatione0248780-
dc.identifier.author-urlhttps://www.ncbi.nlm.nih.gov/pubmed/33852589
dc.identifier.citationGordon M, Viganola D, Dreber A, Johannesson M, Pfeiffer T. (2021). Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects.. PLoS One. 16. 4. (pp. e0248780-).
dc.identifier.doi10.1371/journal.pone.0248780
dc.identifier.eissn1932-6203
dc.identifier.elements-typejournal-article
dc.identifier.issn1932-6203
dc.identifier.numberARTN e0248780
dc.identifier.piiPONE-D-20-35960
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/70773
dc.languageeng
dc.publisherPublic Library of Science (PLoS)
dc.relation.isPartOfPLoS One
dc.rights(c) The Author/s 2021
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectForecasting
dc.subjectHumans
dc.subjectReproducibility of Results
dc.subjectResearch
dc.subjectSurveys and Questionnaires
dc.titlePredicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects
dc.typeJournal article
pubs.elements-id444761
pubs.organisational-groupOther
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Published version
Size:
1.16 MB
Format:
Adobe Portable Document Format
Description:
Download
Collections
Journal Articles