Assessment of Evidential Value Requires More than a Single Data Point
- PDF / 548,701 Bytes
- 5 Pages / 595.276 x 790.866 pts Page_size
- 75 Downloads / 178 Views
COMMENTARY
Assessment of Evidential Value Requires More than a Single Data Point Roland Imhoff1 Received: 28 August 2020 / Revised: 3 September 2020 / Accepted: 4 September 2020 © The Author(s) 2020
In his contribution, Sakaluk (2020) provides an important lesson for sex research. Embracing and adopting practices of an open, transparent, and solid science constitutes an important developmental goal for sex research, and Sakaluk does a great job of not only correcting the scientific record (by qualifying the evidential value of a recently published paper), but by tutoring and explaining the different techniques to assess evidential value. Many of the described techniques are extremely helpful tools to not just evaluate empirical contributions in the peer-review process, but also after the fact: Whole research programs and effects can be scrutinized post-peer review for their robustness, for the available evidence of evidence. Importantly, these have to be research programs, not single studies. One critical aspect of the current state of sex research, I will thus argue, is glaringly missing from Sakaluk’s analysis, one that increasingly seems like the elephant in the room: the overreliance on single-study publications in sex research. Many fields of basic research (cognitive psychology, social psychology, personality psychology) reserve single-study papers for contributions whose empirical parts required such effort that it seems unreasonable to expect a second study (large-scale data, sample from hard-to-reach and underrepresented populations, extensive and expensive data collection as in biological studies). In contrast, single studies continue to be the norm rather than the exception in the flagship journal of sex research, the Archives of Sexual Behavior. Single-study empirical papers with quantitative analyses have remained the prototypical type of article over the past five years, whereas multi-study papers are a rare species and make up barely 12% of all quantitative papers (Table 1 for frequencies of different article types; Fig. 1 for a more visual impression; raw data with coding for each article available at https://osf.io/jqm3k/). This overreliance on single studies * Roland Imhoff roland.imhoff@uni‑mainz.de 1
Social and Legal Psychology, Department of Psychology, Johannes Gutenberg University Mainz, Binger Str. 14‑16, 55122 Mainz, Germany
is in my perspective one of the larger roadblocks on our way toward a more solid sexual science. I will briefly explain why I believe this to be the case. While it certainly is always nice to have more rather than less data, there are several more pressing reasons for adopting a policy that makes single-study papers the exception rather than the rule. I will briefly mention three downsides of single study papers that I see as most relevant. First, they maximize researchers’ degrees of freedom and the danger of false positives. Second, directly following up on Sakaluk (2020), single-study papers make many of the more formalized statistical tests for evidential
Data Loading...
