Mazar et al. (1) conducted a direct replication and six conceptual replications of the studies demonstrating experiment aversion (2, 3), the tendency to rate an experiment as less favorable than its treatment arms. Based on their findings, Mazar et al. conclude that experiment aversion “is neither generalizable nor robust” (1). We are questioning whether Mazar et al.’s findings justify this conclusion. First, in the direct replication the authors perfectly replicate experiment aversion, testifying to the replicability of experiment aversion. In that sense, experiment aversion constitutes a robust empirical finding. In the six conceptual replications testing the generalizability of experiment aversion, the authors changed the original scenarios in a number of ways. For example, the authors employed six rather one dependent variable to measure evaluations of hospitals that conduct experiments and instantiate treatment arms, asked participants to evaluate several hospitals rather than the decision of one doctor, described the hospitals’ actions in the past rather than the present tense, etc. Unfortunately, these deviations from the original studies (2, 3) were not systematically and orthogonally manipulated, so it is not clear which changes contributed to not finding evidence of experiment aversion. Even more troubling, evidence for experiment aversion was found on some of the authors’ measures but not on others. While we agree that these findings suggest that experiment aversion is influenced by a host of factors, it is unclear what these factors are, what impact they have, and whether they are theoretically or practically interesting (4). One uninteresting possibility could be that the lengthy descriptions of the different hospitals that participants were asked to evaluate on six dimensions each reduced participants’ attentiveness with the task, resulting in uninformative responses (5). Without systematically investigating the influence of each deviation from the original studies, it seems like little can be learned from the null findings. Concluding that experiment aversion “is neither generalizable nor robust” (1) is going far beyond Mazar et al.’s findings.

No evidence that experiment aversion is not a robust empirical phenomenon

Bas, Burcak
Membro del Collaboration Group
;
Vosgerau, Joachim
Membro del Collaboration Group
;
Ciulli, Rachele
Membro del Collaboration Group
2023

Abstract

Mazar et al. (1) conducted a direct replication and six conceptual replications of the studies demonstrating experiment aversion (2, 3), the tendency to rate an experiment as less favorable than its treatment arms. Based on their findings, Mazar et al. conclude that experiment aversion “is neither generalizable nor robust” (1). We are questioning whether Mazar et al.’s findings justify this conclusion. First, in the direct replication the authors perfectly replicate experiment aversion, testifying to the replicability of experiment aversion. In that sense, experiment aversion constitutes a robust empirical finding. In the six conceptual replications testing the generalizability of experiment aversion, the authors changed the original scenarios in a number of ways. For example, the authors employed six rather one dependent variable to measure evaluations of hospitals that conduct experiments and instantiate treatment arms, asked participants to evaluate several hospitals rather than the decision of one doctor, described the hospitals’ actions in the past rather than the present tense, etc. Unfortunately, these deviations from the original studies (2, 3) were not systematically and orthogonally manipulated, so it is not clear which changes contributed to not finding evidence of experiment aversion. Even more troubling, evidence for experiment aversion was found on some of the authors’ measures but not on others. While we agree that these findings suggest that experiment aversion is influenced by a host of factors, it is unclear what these factors are, what impact they have, and whether they are theoretically or practically interesting (4). One uninteresting possibility could be that the lengthy descriptions of the different hospitals that participants were asked to evaluate on six dimensions each reduced participants’ attentiveness with the task, resulting in uninformative responses (5). Without systematically investigating the influence of each deviation from the original studies, it seems like little can be learned from the null findings. Concluding that experiment aversion “is neither generalizable nor robust” (1) is going far beyond Mazar et al.’s findings.
2023
2023
Bas, Burcak; Vosgerau, Joachim; Ciulli, Rachele
File in questo prodotto:
File Dimensione Formato  
Bas et al. PNAS 2023.pdf

accesso aperto

Descrizione: letter
Tipologia: Pdf editoriale (Publisher's layout)
Licenza: Creative commons
Dimensione 111.29 kB
Formato Adobe PDF
111.29 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11565/4061020
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 2
social impact