A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short ``null results reports'' online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.

Learning from null effects: a bottom-up approach

Alrababa'h, Ala';Williamson, Scott;
2023

Abstract

A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short ``null results reports'' online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.
2023
2022
Alrababa'H, Ala'; Williamson, Scott; Dillon, Andrea; Hainmueller, Jens; Hangartner, Dominik; Hotard, Michael; Laitin, David D.; Lawrence, Duncan; Wein...espandi
File in questo prodotto:
File Dimensione Formato  
null_effects.pdf

non disponibili

Descrizione: Accepted Version of Article
Tipologia: Documento in Pre-print (Pre-print document)
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 74.19 kB
Formato Adobe PDF
74.19 kB Adobe PDF   Visualizza/Apri
PA_Acceptance_Email_Williamson.pdf

non disponibili

Descrizione: Acceptance Email from PA
Tipologia: Allegato per valutazione Bocconi (Attachment for Bocconi evaluation)
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 86.28 kB
Formato Adobe PDF
86.28 kB Adobe PDF   Visualizza/Apri
learning-from-null-effects-a-bottom-up-approach.pdf

accesso aperto

Tipologia: Pdf editoriale (Publisher's layout)
Licenza: Creative commons
Dimensione 267.42 kB
Formato Adobe PDF
267.42 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11565/4043183
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 6
  • ???jsp.display-item.citation.isi??? 4
social impact