Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme
Loading...
Date
2021-07
Open Access Location
Journal Title
Journal ISSN
Volume Title
Publisher
The Royal Society
Rights
(c) The author/s
CC BY 4.0
CC BY 4.0
Abstract
There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
Description
Keywords
hypothesis, peer beliefs, prediction markets
Citation
Viganola D, Buckles G, Chen Y, Diego-Rosell P, Johannesson M, Nosek BA, Pfeiffer T, Siegel A, Dreber A. (2021). Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme.. R Soc Open Sci. 8. 7. (pp. 181308-).