Skip to main content

Many analysts and few incentives

Religion, Brain & Behavior, 2023

Robert M. Ross, Justin Sulik, Jacek Buczny, Bruno Schivinski

Religion, Brain & Behavior, volume 13, issue 3, pages 336–339, 2023 (published online 6 July 2022)

  • Commentary
  • Many Analysts Religion Project
  • Metascience
  • Construct validity
  • Research incentives
Download the accepted manuscript (PDF)Published version

Free full text: the authors’ accepted manuscript (CC BY 4.0, as posted on PsyArXiv), re-typeset as a clean PDF. Please cite the published version.

Keywords
  • many analysts
  • Many Analysts Religion Project
  • MARP
  • metascience
  • research incentives
  • construct validity
  • measurement
  • questionable measurement practices
  • sum scores
  • measurement invariance
  • cross-cultural research
  • religiosity
  • multilevel modelling
  • pre-registration
  • open science

Many-analysts projects remove the pressure to find significant results, but they also remove the reward for doing the hard parts well. Reflecting on their own two analysis teams in the Many Analysts Religion Project, the authors show how they never checked whether their religiosity measure was valid, and suggest ways to reward careful work.

Key findings

  1. No incentive for significance can also mean no incentive for care. A core strength of many-analysts projects is that teams have no direct incentive to reach statistical significance, but this lack of incentives is a double-edged sword: a team’s submission is one among many and has little visible effect on the project’s success (Ross et al., 2023).
  2. Both authors’ teams ignored construct validity. Team 039 summed four of the nine religiosity items and Team 175 summed all nine, but neither team assessed the internal consistency, factor structure or cross-cultural measurement invariance of the religiosity score (Ross et al., 2023).
  3. Teams invest where they already have expertise. The teams put careful work into the multilevel models they were skilled at, while construct validation in cross-cultural research would have needed a large investment of time and effort, so it was left out (Ross et al., 2023).
  4. Reward the strongest submissions and build diverse teams. The authors predict that very few MARP teams assessed construct validity, and suggest rewarding the strongest submissions (author order, invited commentaries, awards or prizes) and forming teams that combine analytic skills with domain knowledge (Ross et al., 2023).

At a glance

Type
Commentary on the Many Analysts Religion Project (MARP), written by members of two of its analysis teams (Team 039 and Team 175)
Topic
How the incentive structure of many-analysts projects can lead teams to neglect key parts of an analysis, such as the construct validity of measures
Proposals
Code MARP submissions for attempts to assess construct validity; reward the strongest submissions; incentivise teams that bring together diverse analytic skills and domain knowledge

The question

Many-analysts projects ask many independent teams to analyse the same data, and teams have no direct incentive to reach a particular result. We asked whether that lack of incentives has a downside, using our own two submissions to the Many Analysts Religion Project as a case.

Implications

Practical implications

Organisers of many-analysts projects should consider ways to reward the quality of submissions, for example by identifying the strongest submissions and giving their authors more prominent author positions, invited commentaries, awards or prizes, and by encouraging teams that combine analytic and domain expertise.

Theoretical implications

The many-analysts approach does not by itself fix poor measurement practices (a “measurement schmeasurement” attitude) in the social sciences; construct validity, including measurement invariance across cultures, needs attention in multi-team as well as single-team research.

Limitations and future research

  • Based on the authors’ own two submissions; the prediction about other teams remains to be tested.
  • The authors are not confident that they have good introspective access to what motivated their analysis decisions.

Questions this paper answers

What is a many-analysts study?

A study in which many independent teams analyse the same data set to answer the same question, showing how analytic choices affect results (Silberzahn & Uhlmann, 2015; Silberzahn et al., 2018).

What is the downside of having no incentives in many-analysts projects?

Teams have no reason to chase significance, but also little reason to invest effort in parts of the analysis outside their expertise; in the Many Analysts Religion Project, the authors’ own teams did not assess the construct validity of their religiosity measure (Ross et al., 2023).

How could many-analysts projects reward careful work?

By identifying and rewarding the strongest submissions (more prominent author positions, invited commentaries, awards or prizes) and by encouraging teams that combine diverse analytic skills and domain knowledge (Ross et al., 2023).

Key concepts

Many-analysts approach
Independent teams analyse the same data to answer the same question, so that the effect of analytic choices on results can be seen.
Construct validity
Whether a measure captures the construct it is meant to; here, at least internal consistency, factor structure and measurement invariance across cultures could have been checked.
Measurement invariance
A measure works the same way across groups (e.g., cultures), so that scores can be compared between them.

Open science and materials

How to cite

Ross, R. M., Sulik, J., Buczny, J., & Schivinski, B. (2023). Many analysts and few incentives. Religion, Brain & Behavior, 13(3), 336–339. https://doi.org/10.1080/2153599X.2022.2070248

Related publications

All publications