Attenzione: i dati modificati non sono ancora stati salvati. Per confermare inserimenti o cancellazioni di voci è necessario confermare con il tasto SALVA/INSERISCI in fondo alla pagina
IRIS PubliCatt
To what extent are the results of research investigations influenced by subjective decisions that
scientists make as they design studies? Fifteen research teams independently designed studies to
answer five original research questions related to moral judgments, negotiations, and implicit
cognition. Participants from two separate, large samples (total N > 15,000) were then randomly
assigned to complete one version of each study. Effect sizes varied dramatically across different
sets of materials designed to test the same hypothesis: materials from different teams rendered
significant effects in opposite directions for four out of five hypotheses, with the narrowest range
in estimates being d = -0.37 to 0.26. Meta-analysis indicated a lack of overall support for two
original hypotheses, mixed support for one hypothesis, and significant support for two
hypotheses. Overall, none of the variability in effect sizes was attributable to the skill of the
research team in designing materials, while some variability was attributable to the hypothesis
being tested. In a forecasting survey, predictions of other scientists were strongly correlated with
study results, and average predictions were similar to observed outcomes. Crowdsourced testing
of research hypotheses helps reveal the true consistency of empirical support for a scientific
claim.
Landy, J. F., (liam) Jia, M., Ding, I., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., Ly, A., Van Den Bergh, D., Marsman, M., Wagenmakers, E., Bartels, D. M., Bauman, C. W., Brady, W., Cheung, F., Cimpian, A., Dohle, S., Brent Donnellan, M., Hahn, A., Hall, M., Jiménez-Leal, W., Johnson, D. J., Lucas, R. E., Monin, B., Montealegre, A., Mullen, E., Pang, J., Ray, J., Reinero, D. A., Reynolds, J., Sowden, W., Storage, D., Su, R., Tworek, C. M., Van Bavel, J. J., Walco, D., Wills, J., Xu, X., Chi Yam, K., Yang, X., Schweinsberg, M., Urwitz, M., Adamkovič, M., Alaei, R., Albers, C. J., Allard, A., Anderson, I. A., Andreychik, M. R., Babinčák, P., Baker, B. J., Baník, G., Baskin, E., Bavolar, J., Berkers, R. M. W. J., Białek, M., Blanke, J., Breuer, J., Brizi, A., Brown, S. E. V., Brühlmann, F., Bruns, H., Caldwell, L., Campourcy, J., Chan, E. Y., Chang, Y., Cheung, B. Y., Chin, A., Cho, K. W., Columbus, S., Conway, P., Corretti, C. A., Craig, A. W., Curran, P. G., Danvers, A. F., Dawson, I. G. J., Day, M. V., Dietl, E., Doerflinger, J. T., Dominici, A., Dranseika, V., Edelsbrunner, P. A., Edlund, J. E., Fisher, M., Fung, A., Genschow, O., Gnambs, T., Goldberg, M. H., Graf-Vlachy, L., Hafenbrack, A. C., Hafenbrädl, S., Hartanto, A., Heck, P. R., Heffner, J. P., Hilgard, J., Holzmeister, F., Horchak, O. V., Huang, T. S. -., Hüffmeier, J., Hughes, S., Hussey, I., Imhoff, R., Jaeger, B., Jamro, K., Johnson, S. G. B., Jones, A., Keller, L., Kombeiz, O., Krueger, L. E., Lantian, A., Laplante, J. P., Lazarevic, L. B., Leclerc, J., Legate, N., Leonhardt, J. M., Leung, D. W., Levitan, C. A., Lin, H., Liu, Q., Tullio Liuzza, M., Locke, K. D., Ly, A. L., Maceacheron, M. D., Madan, C. R., Manley, H., Mari, S., Martončik, M., Mclean, S. L., Mcphetres, J., Mercier, B. G., Michels, C., Mullarkey, M. C., Musser, E. D., Nalborczyk, L., Nilsonne, G., Otis, N. G., Otner, S. M. G., Otto, P. E., Oviedo-Trespalacios, O., Paruzel- Czachura, M., Pellegrini, F., Pereira, V. M. D., Perfecto, H., Pfuhl, G., Phillips, M. H., Plonsky, O., Pozzi, M., Purić, D. B., Raymond-Barker, B., Redman, D. E., Reynolds, C. J., Ropovik, I., Röseler, L., Ruessmann, J. K., Ryan, W. H., Sablaturova, N., Schuepfer, K. J., Schütz, A., Sirota, M., Stefan, M., Stocks, E. L., Strosser, G. L., Suchow, J. W., Szabelska, A., Tey, K. S., Tiokhin, L., Troian, J., Utesch, T., Vásquez-Echeverría, A., Ann Vaughn, L., Verschoor, M., Von Helversen, B., Wallisch, P., Weissgerber, S. C., Wichman, A. L., Woike, J. K., Žeželj, I., Zickfeld, J. H., Ahn, Y., Blaettchen, P. F., Kang, X., Jin Lee, Y., Parker, P. M., Parker, P. A., Song, J. S., Very, M., Wong, L., Uhlmann, E. L., Crowdsourcing hypothesis tests: Making transparent how design choices shape research results, <<PSYCHOLOGICAL BULLETIN>>, 5; 146 (5): 451-479. [doi:10.1037/bul0000220] [http://hdl.handle.net/10807/146268]
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results
Justin F. Landy;Miaolei (Liam) Jia;Isabel Ding;Domenico Viganola;Warren Tierney;Anna Dreber;Magnus Johannesson;Thomas Pfeiffer;Charles R. Ebersole;Quentin F. Gronau;Alexander Ly;Don van den Bergh;Maarten Marsman;Eric-Jan Wagenmakers;Daniel M. Bartels;Christopher W. Bauman;William Brady;Felix Cheung;Andrei Cimpian;Simone Dohle;M. Brent Donnellan;Adam Hahn;Michael Hall;William Jiménez-Leal;David J. Johnson;Richard E. Lucas;Benoît Monin;Andres Montealegre;Elizabeth Mullen;Jun Pang;Jennifer Ray;Diego A. Reinero;Jesse Reynolds;Walter Sowden;Daniel Storage;Runkun Su;Christina M. Tworek;Jay J. Van Bavel;Daniel Walco;Julian Wills;Xiaobing Xu;Kai Chi Yam;Xiaoyu Yang;Martin Schweinsberg;Molly Urwitz;Matúš Adamkovič;Ravin Alaei;Casper J. Albers;Aurélien Allard;Ian A. Anderson;Michael R. Andreychik;Peter Babinčák;Bradley J. Baker;Gabriel Baník;Ernest Baskin;Jozef Bavolar;Ruud M. W. J. Berkers;Michał Białek;Joel Blanke;Johannes Breuer;Ambra Brizi;Stephanie E. V. Brown;Florian Brühlmann;Hendrik Bruns;Leigh Caldwell;Jean-François Campourcy;Eugene Y. Chan;Yen-Ping Chang;Benjamin Y. Cheung;Alycia Chin;Kit W. Cho;Simon Columbus;Paul Conway;Conrad A. Corretti;Adam W. Craig;Paul G. Curran;Alexander F. Danvers;Ian G. J. Dawson;Martin V. Day;Erik Dietl;Johannes T. Doerflinger;Alice Dominici;Vilius Dranseika;Peter A. Edelsbrunner;John E. Edlund;Matthew Fisher;Anna Fung;Oliver Genschow;Timo Gnambs;Matthew H. Goldberg;Lorenz Graf-Vlachy;Andrew C. Hafenbrack;Sebastian Hafenbrädl;Andree Hartanto;Patrick R. Heck;Joseph P. Heffner;Joseph Hilgard;Felix Holzmeister;Oleksandr V. Horchak;Tina S. -T. Huang;Joachim Hüffmeier;Sean Hughes;Ian Hussey;Roland Imhoff;Bastian Jaeger;Konrad Jamro;Samuel G. B. Johnson;Andrew Jones;Lucas Keller;Olga Kombeiz;Lacy E. Krueger;Anthony Lantian;Justin P. Laplante;Ljiljana B. Lazarevic;Jonathan Leclerc;Nicole Legate;James M. Leonhardt;Desmond W. Leung;Carmel A. Levitan;Hause Lin;Qinglan Liu;Marco Tullio Liuzza;Kenneth D. Locke;Albert L. Ly;Melanie D. MacEacheron;Christopher R. Madan;Harry Manley;Silvia Mari;Marcel Martončik;Scott L. McLean;Jonathon McPhetres;Brett G. Mercier;Corinna Michels;Michael C. Mullarkey;Erica D. Musser;Ladislas Nalborczyk;Gustav Nilsonne;Nicholas G. Otis;Sarah M. G. Otner;Philipp E. Otto;Oscar Oviedo-Trespalacios;Mariola Paruzel- Czachura;Francesco Pellegrini;Vitor M. D. Pereira;Hannah Perfecto;Gerit Pfuhl;Mark H. Phillips;Ori Plonsky;Pozzi, Maura;Danka B. Purić;Brett Raymond-Barker;David E. Redman;Caleb J. Reynolds;Ivan Ropovik;Lukas Röseler;Janna K. Ruessmann;William H. Ryan;Nika Sablaturova;Kurt J. Schuepfer;Astrid Schütz;Miroslav Sirota;Matthias Stefan;Eric L. Stocks;Garrett L. Strosser;Jordan W. Suchow;Anna Szabelska;Kian-Siong S. Tey;Leonid Tiokhin;Jais Troian;Till Utesch;Alejandro Vásquez-Echeverría;Leigh Ann Vaughn;Mark Verschoor;Bettina von Helversen;Pascal Wallisch;Sophia C. Weissgerber;Aaron L. Wichman;Jan K. Woike;Iris Žeželj;Janis H. Zickfeld;Yeonsin Ahn;Philippe F. Blaettchen;Xi Kang;Yoo Jin Lee;Philip M. Parker;Paul A. Parker;Jamie S. Song;May-Anne Very;Lynn Wong;Eric L. Uhlmann
2020
Abstract
To what extent are the results of research investigations influenced by subjective decisions that
scientists make as they design studies? Fifteen research teams independently designed studies to
answer five original research questions related to moral judgments, negotiations, and implicit
cognition. Participants from two separate, large samples (total N > 15,000) were then randomly
assigned to complete one version of each study. Effect sizes varied dramatically across different
sets of materials designed to test the same hypothesis: materials from different teams rendered
significant effects in opposite directions for four out of five hypotheses, with the narrowest range
in estimates being d = -0.37 to 0.26. Meta-analysis indicated a lack of overall support for two
original hypotheses, mixed support for one hypothesis, and significant support for two
hypotheses. Overall, none of the variability in effect sizes was attributable to the skill of the
research team in designing materials, while some variability was attributable to the hypothesis
being tested. In a forecasting survey, predictions of other scientists were strongly correlated with
study results, and average predictions were similar to observed outcomes. Crowdsourced testing
of research hypotheses helps reveal the true consistency of empirical support for a scientific
claim.
Landy, J. F., (liam) Jia, M., Ding, I., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., Ly, A., Van Den Bergh, D., Marsman, M., Wagenmakers, E., Bartels, D. M., Bauman, C. W., Brady, W., Cheung, F., Cimpian, A., Dohle, S., Brent Donnellan, M., Hahn, A., Hall, M., Jiménez-Leal, W., Johnson, D. J., Lucas, R. E., Monin, B., Montealegre, A., Mullen, E., Pang, J., Ray, J., Reinero, D. A., Reynolds, J., Sowden, W., Storage, D., Su, R., Tworek, C. M., Van Bavel, J. J., Walco, D., Wills, J., Xu, X., Chi Yam, K., Yang, X., Schweinsberg, M., Urwitz, M., Adamkovič, M., Alaei, R., Albers, C. J., Allard, A., Anderson, I. A., Andreychik, M. R., Babinčák, P., Baker, B. J., Baník, G., Baskin, E., Bavolar, J., Berkers, R. M. W. J., Białek, M., Blanke, J., Breuer, J., Brizi, A., Brown, S. E. V., Brühlmann, F., Bruns, H., Caldwell, L., Campourcy, J., Chan, E. Y., Chang, Y., Cheung, B. Y., Chin, A., Cho, K. W., Columbus, S., Conway, P., Corretti, C. A., Craig, A. W., Curran, P. G., Danvers, A. F., Dawson, I. G. J., Day, M. V., Dietl, E., Doerflinger, J. T., Dominici, A., Dranseika, V., Edelsbrunner, P. A., Edlund, J. E., Fisher, M., Fung, A., Genschow, O., Gnambs, T., Goldberg, M. H., Graf-Vlachy, L., Hafenbrack, A. C., Hafenbrädl, S., Hartanto, A., Heck, P. R., Heffner, J. P., Hilgard, J., Holzmeister, F., Horchak, O. V., Huang, T. S. -., Hüffmeier, J., Hughes, S., Hussey, I., Imhoff, R., Jaeger, B., Jamro, K., Johnson, S. G. B., Jones, A., Keller, L., Kombeiz, O., Krueger, L. E., Lantian, A., Laplante, J. P., Lazarevic, L. B., Leclerc, J., Legate, N., Leonhardt, J. M., Leung, D. W., Levitan, C. A., Lin, H., Liu, Q., Tullio Liuzza, M., Locke, K. D., Ly, A. L., Maceacheron, M. D., Madan, C. R., Manley, H., Mari, S., Martončik, M., Mclean, S. L., Mcphetres, J., Mercier, B. G., Michels, C., Mullarkey, M. C., Musser, E. D., Nalborczyk, L., Nilsonne, G., Otis, N. G., Otner, S. M. G., Otto, P. E., Oviedo-Trespalacios, O., Paruzel- Czachura, M., Pellegrini, F., Pereira, V. M. D., Perfecto, H., Pfuhl, G., Phillips, M. H., Plonsky, O., Pozzi, M., Purić, D. B., Raymond-Barker, B., Redman, D. E., Reynolds, C. J., Ropovik, I., Röseler, L., Ruessmann, J. K., Ryan, W. H., Sablaturova, N., Schuepfer, K. J., Schütz, A., Sirota, M., Stefan, M., Stocks, E. L., Strosser, G. L., Suchow, J. W., Szabelska, A., Tey, K. S., Tiokhin, L., Troian, J., Utesch, T., Vásquez-Echeverría, A., Ann Vaughn, L., Verschoor, M., Von Helversen, B., Wallisch, P., Weissgerber, S. C., Wichman, A. L., Woike, J. K., Žeželj, I., Zickfeld, J. H., Ahn, Y., Blaettchen, P. F., Kang, X., Jin Lee, Y., Parker, P. M., Parker, P. A., Song, J. S., Very, M., Wong, L., Uhlmann, E. L., Crowdsourcing hypothesis tests: Making transparent how design choices shape research results, <<PSYCHOLOGICAL BULLETIN>>, 5; 146 (5): 451-479. [doi:10.1037/bul0000220] [http://hdl.handle.net/10807/146268]
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10807/146268
Citazioni
ND
98
84
social impact
Conferma cancellazione
Sei sicuro che questo prodotto debba essere cancellato?
simulazione ASN
Il report seguente simula gli indicatori relativi alla propria produzione scientifica in relazione alle soglie ASN 2023-2025 del proprio SC/SSD. Si ricorda che il superamento dei valori soglia (almeno 2 su 3) è requisito necessario ma non sufficiente al conseguimento dell'abilitazione. La simulazione si basa sui dati IRIS e sugli indicatori bibliometrici alla data indicata e non tiene conto di eventuali periodi di congedo obbligatorio, che in sede di domanda ASN danno diritto a incrementi percentuali dei valori. La simulazione può differire dall'esito di un’eventuale domanda ASN sia per errori di catalogazione e/o dati mancanti in IRIS, sia per la variabilità dei dati bibliometrici nel tempo. Si consideri che Anvur calcola i valori degli indicatori all'ultima data utile per la presentazione delle domande.
La presente simulazione è stata realizzata sulla base delle specifiche raccolte sul tavolo ER del Focus Group IRIS coordinato dall’Università di Modena e Reggio Emilia e delle regole riportate nel DM 589/2018 e allegata Tabella A. Cineca, l’Università di Modena e Reggio Emilia e il Focus Group IRIS non si assumono alcuna responsabilità in merito all’uso che il diretto interessato o terzi faranno della simulazione. Si specifica inoltre che la simulazione contiene calcoli effettuati con dati e algoritmi di pubblico dominio e deve quindi essere considerata come un mero ausilio al calcolo svolgibile manualmente o con strumenti equivalenti.