Skip to main content

Mark Rubin

Durham University, Psychology, Faculty Member

Followers

685

Following

169

Co-authors

24

Public Views

I am a professor in psychology at Durham University. I received an MSc from the London School of Economics and a PhD from Cardiff University, UK. I have published over 100 research articles, mainly in the areas of social identity, stereotyping, and prejudice. My recent research has focused on social class, social exclusion, and mental health. I have also published several articles on issues connected with the replication crisis in science. For more information about my work, please visit: https://bit.ly/rubinpsyc
Supervisors: Miles Hewstone
Phone: +61 (0)2 4921 6706
Address: School of Psychology, The University of Newcastle, Callaghan, NSW 2308, Australia

less

InterestsView All (45)

Uploads

Papers by Mark Rubin

Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses

Methods in Psychology, 2024

During multiple testing, researchers often adjust their alpha level to control the familywise err... more During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., "H 1,1 or H 1,2 "). However, in some cases, they do not make this inference. Instead, they make separate inferences about each of the individual hypotheses that comprise the joint hypothesis (e.g., H 1,1 and H 1,2). For example, a researcher might use a Bonferroni correction to adjust their alpha level from the conventional level of 0.050 to 0.025 when testing H 1,1 and H 1,2 , find a significant result for H 1,1 (p < 0.025) and not for H 1,2 (p > 0.025), and so claim support for H 1,1 and not for H 1,2. However, these separate individual inferences do not require an alpha adjustment. Only a statistical inference about the union alternative hypothesis "H 1,1 or H 1,2 " requires an alpha adjustment because it is based on "at least one" significant result among the two tests, and so it refers to the familywise error rate. Hence, an inconsistent correction occurs when a researcher corrects their alpha level during multiple testing but does not make an inference about a union alternative hypothesis. In the present article, I discuss this inconsistent correction problem, including its reduction in statistical power for tests of individual hypotheses and its potential causes visà-vis error rate confusions and the alpha adjustment ritual. I also provide three illustrations of inconsistent corrections from recent psychology studies. I conclude that inconsistent corrections represent a symptom of statisticism, and I call for a more nuanced inference-based approach to multiple testing corrections. ☆ I am grateful to Vinay Tummarakota for discussions that led to my explanation of Confusion IV. I declare no funding sources. I declare no conflict of interest.

Type I Error Rates are Not Usually Inflated

Journal of Trial & Error, 2024

The inflation of Type I error rates is thought to be one of the causes of the replication crisis.... more The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

The Replication Crisis is Less of a “Crisis” in Lakatos’ Philosophy of Science

Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of... more Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures. In this article, I contrast Popper’s approach with Lakatos’ (1978) approach and a related approach called naïve methodological falsificationism (NMF; Lakatos, 1978). The Popperian approach is powerful because it is based on logical refutation, but its theories are noncausal and, therefore, lacking in scientific value. In contrast, the Lakatosian approach considers causal theories, but it concedes that these theories are not logically refutable. Finally, the NMF approach subjects Lakatosian causal theories to Popperian logical refutations. However, its approach of temporarily accepting a ceteris paribus clause during theory testing may be viewed as scientifically inappropriate, epistemically inconsistent, and “completely redundant” (Lakatos, 1978, p. 40). I conclude that the replication “crisis” makes the most sense in the context of the Popperian and NMF approaches because it is only in these two approaches that replication failures represent logical refutations of theories. In contrast, replication failures are less problematic in the Lakatosian approach because they do not logically refute theories. Indeed, in the Lakatosian approach, replication failures can be legitimately ignored or used to motivate theory development.

Questionable Metascience Practices

Journal of Trial and Error, 2023

Metascientists have studied questionable research practices in science. The present article consi... more Metascientists have studied questionable research practices in science. The present article considers the parallel concept of questionable metascience practices (QMPs). A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. The present article reviews ten QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast 'n' bropen scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is "too low" without specifying an "acceptable" rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as being more "tentative" than confirmatory results; (8) presuming that questionable research practices are problematic research practices; (9) focusing on knowledge accumulation; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.

Social Identity Explanations of System Justification: Misconceptions, Criticisms, and Clarifications -Supplementary Document

by Mark Rubin and Chuma Owuamalam

European Review of Social Psychology, 2023

This document supplements our reply to Jost et al.'s (2023) rejoinder to our initial article on t... more This document supplements our reply to Jost et al.'s (2023) rejoinder to our initial article on the social identity model of system attitudes (SIMSA; Rubin et al., 2023). We respond to issues that we did not address in our main reply. Specifically, we argue that: (1) like SIMSA, Tajfel (1981) considered general, non-ideological, psychological processes; (2) system justification theory also refers to general, non-ideological, social psychological processes; (3) Jost et al. confuse respecting people with respecting people's beliefs; (4) SIMSA proponents do not ignore system justification theory's three motives; and (5) SIMSA may help to explain Osborne et al.'s (2019) results. In addition, (6) we apologise if we mischaracterised some researchers' theoretical affiliations, and we respond to Jost et al.'s (7) questions about ideology; (8) criticisms of Owuamalam et al.'s (2016) work; and (9) criticisms about the validity of Owuamalam and Spears' (2020) pupil dilation measure.

Social identity explanations of system justification: Misconceptions, criticisms, and clarifications

European Review of Social Psychology, 2023

In this article, we reply to Jost et al.’s (2023) rejoinder to our article reviewing evidence for... more In this article, we reply to Jost et al.’s (2023) rejoinder to our article reviewing evidence for the social identity model of system attitudes (SIMSA; Rubin et al., 2023). We argue that (1) SIMSA treats system justification as the outcome of an interaction between general social psychological process and specific historical, political, cultural, and ideological environments; (2) it does not conflate perceived intergroup status differences with the perceived stability and legitimacy of those differences, (3) it is not fatalistic, because it assumes that people may engage in social change when they perceive an opportunity to do so; (4) it adopts a non-reductionist, social psychological explanation of system justification, rather than an individualist explanation based on individual differences; (5) it presupposes “existing social arrangements”, including their existing legitimacy and stability, and assumes that these social arrangements are either passively acknowledged or actively supported; and (6) it is not reliant on minimal group experiments in its evidence base.

A social identity model of system attitudes (SIMSA): Multiple explanations of system justification by the disadvantaged that do not depend on a separate system justification motive

European Review of Social Psychology, 2023

System justification theory (SJT) assumes that social identity theory (SIT) cannot fully account ... more System justification theory (SJT) assumes that social identity theory (SIT) cannot fully account for system justification by members of low-status (disadvantaged) groups. Contrary to this claim, we provide several elaborations of SIT that explain when and why members of low-status groups show system justification independent from any separate system justification motive. According to the social identity model of system attitudes (SIMSA), the needs for social accuracy and a positively distinct social identity fully account for system justification by members of low-status groups. In the present article, we (a) explain SIMSA's accounts of system justification, (b) develop associated hypotheses, (c) summarise evidence that supports each hypothesis, and (d) highlight issues to be addressed in future research. We conclude that SIMSA provides a more parsimonious explanation of system justification by the disadvantaged than SJT, because it does not refer to an additional separate system justification motive.

Sleep as a mediator of the relationship between social class and health in higher education students

by Mark Rubin and Romany Mcguffog

British Journal of Psychology, 2023

A substantial body of research indicates that higher education students from lower social class b... more A substantial body of research indicates that higher education students from lower social class backgrounds tend to have poorer health than those from higher social class backgrounds. To investigate sleep as a potential mediator of this relationship, online survey responses of students from five large Australian universities, one Irish university and one large Australian technical college were analysed in three studies (Study 1 N = 628; Study 2 N = 376; Study 3 N = 446). The results revealed that sleep quality, sleep duration, sleep disturbances, pre-sleep worries and sleep schedule variability mediated the relationship between social class and physical and mental health. Sleep remained a significant mediator when controlling for related variables and other mediators. Thus, the findings suggest that sleep partly explains social class differences in health. We discuss the importance of addressing sleep issues among students from lower social class backgrounds.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

Philosophical Psychology, 2022

Preregistration has been proposed as a useful method for making a publicly verifiable distinction... more Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences.

A First Class Measure: Evidence for a Comprehensive Social Class Scale in Higher Education Populations

Research in Higher Education, 2022

Despite social class being a burgeoning area of research in the higher education literature, ther... more Despite social class being a burgeoning area of research in the higher education literature, there is no single comprehensive measure of social class in university student populations. Most previous research has included objective single-item measures (e.g., parent education or occupation) to assess social class and then sorted students into distinct social class categories using these items. Such approaches do not adequately capture the complexity and nuance of class, and they ignore the subjective and social components involved. The present paper reports the development and validation of an 11-item Comprehensive Social Class Scale (CSCS) that uses a mix of objective and subjective items to assess multiple aspects of social class, including education level, occupational prestige, family affluence, social class identity and subjective social status. Across 12 samples (N = 4926), we provide evidence for a single factor structure of the CSCS and demonstrate aspects of its reliability and validity. We conclude by discussing some limitations and suggestions for use of the CSCS in higher education populations.

That's Not a Two-Sided Test! It's Two One-Sided Tests

Significance, 2022

When reporting tests of significance, researchers might claim to have conducted a twosided test w... more

Binge Drinkers Shouldn't Set Their Own Alcohol Reduction Goals

Alcohol Treatment Quarterly, 2022

The present research aimed to investigate the relative effectiveness of three types of alcohol re... more The present research aimed to investigate the relative effectiveness of three types of alcohol reduction intervention. Participants were 354 university students from an Australian university. After completing an initial survey, they were randomly assigned to either (a) follow national guidelines for alcohol consumption, (b) set their own personal alcohol consumption reduction goal, or (c) work with their peers to set a goal. Participants then recorded their alcohol consumption in a drinking diary over a period of four weeks. The results showed that participants in the personal goal condition reported significantly higher alcohol consumption than those in the national guidelines conditions, with participants in the group goal condition reporting mid-level alcohol consumption. This effect was moderated by binge drinking propensity. Personal goal setting was most likely to lead to higher alcohol consumption when participants scored relatively high on measures of binge drinking propensity. These findings highlight the point that different types of alcohol reduction interventions may be effective for different types of people. In particular, people who have a relatively high propensity for binge drinking should be encouraged to follow goals that have been set by authorities (e.g., national guidelines) rather than by themselves.

A cross-sectional investigation of parenting style and friendship as mediators of the relation between social class and mental health in a university community

Introduction: This study tested a novel explanation for the positive relation between social clas... more Introduction: This study tested a novel explanation for the positive relation between social class and mental health among university students. Students with a higher social class were expected to have experienced more authoritative and less authoritarian parenting styles; these parenting styles were expected to lead to greater friendship and social integration at university; and greater friendship and integration were expected to lead to better mental health. Method: To test this model, the researchers asked 397 Australian undergraduate students to complete an online survey. The research used a cross-sectional correlational design, and the data was analysed using bootstrapped multiple serial mediation tests. Results: Consistent with predictions, parenting style, general friendship and support, and social integration at university mediated the relation between social class and mental health. Conclusions: The present results suggest that working-class parenting styles may inhibit the...

The academic outcomes of first-in-family in an Australian university: An exploratory study

International Studies in Widening Participation

Although the first-generation and first-in-family status (FIF) of university students has been of... more Although the first-generation and first-in-family status (FIF) of university students has been of intense interest in the USA, it has received very little consideration in Australia. The present research redressed this imbalance by investigating the academic outcomes of FIF undergraduate students at a large, public, Australian university. Undergraduate students (N = 227) who were enrolled in education, nursing and liberal arts degrees completed an online survey. Data are representative of typical gender enrolment patterns for these degrees. In contrast to US research, there was no clear relationship between socioeconomic status and FIF status in this sample. Consistent with US research, FIF students had poorer academic outcomes than non-FIF students. However, this difference was only significant after the first-year of study when students were less likely to receive scaffolded learning support within courses. FIF students were more likely than non-FIF students to seek support from u...

There’s No Need to Lower the Significance Threshold When Conducting Single Tests of Multiple Individual Hypotheses

Academia Letters, 2021

During null hypothesis significance testing, a multiple testing problem occurs when researchers t... more During null hypothesis significance testing, a multiple testing problem occurs when researchers test a joint null hypothesis using a union-intersection testing approach (e.g., Kim et al., 2004; Parker & Weir, 2020; Roy, 1953). A joint null hypothesis comprises two or more constituent null hypotheses that can each be subjected to separate significance tests. For example, consider the joint null hypothesis that "male participants have neither higher academic self-esteem scores nor higher social self-esteem scores than female participants." In this case, a researcher could conduct a significance test on each of the two constituent null hypotheses: (a) "male participants do not have higher academic self-esteem scores than female participants," and (b) "male participants do not have higher social self-esteem scores than female participants." During union-intersection testing, it is logical to reject the overall joint null hypothesis if at least one of its constituent null hypotheses is rejected. So, for example, it is logical to reject the joint null hypothesis that "male participants have neither higher academic self-esteem scores nor higher social self-esteem scores than female participants" if we reject the constituent null hypothesis that "male participants do not have higher academic self-esteem scores than female participants." The multiple testing problem occurs because the greater the number of constituent null hypotheses that comprise the joint null hypothesis, the greater the number of opportunities that a researcher has to incorrectly reject at least one constituent null hypothesis during unionintersection testing, and so the greater the probability of incorrectly rejecting the overall joint null hypothesis. More formally, if a researcher uses a significance threshold (alpha level) of α for each of k independent tests of their joint null hypothesis, then the Type I error rate

When to Adjust Alpha During Multiple Testing: A Consideration of Disjunction, Conjunction, and Individual Testing

Synthese, 2021

Scientists often adjust their significance threshold (alpha level) during null hypothesis signifi... more Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.

Emotions in Intergroup Contact: Incidental and Integral Emotions' Effects on Interethnic Bias Are Moderated by Emotion Applicability and Subjective Agency

by Mark Rubin and Jake Harwood

Frontiers in Psychology, 2021

This research draws from three distinct lines of research on the link between emotions and interg... more This research draws from three distinct lines of research on the link between emotions and intergroup bias as springboard to integrative, new hypotheses. Past research suggests that emotions extrinsic to the outgroup (or “incidental”), and intrinsic to the outgroup (or “integral”), produce valence-congruent effects on intergroup bias when relevant or “applicable” to the outgroup (e.g., incidental/integral anger and ethnic outgroups). These emotions produce valence incongruent effects when irrelevant or “non-applicable” to the outgroup (e.g., incidental/integral sadness and happiness, and ethnic outgroups). Internally valid and ecologically sound tests of these contrasting effects are missing; hence we examined them experimentally in meaningful settings of interethnic contact. To this end, we hybridized established research paradigms in mood and intergroup contact research; this approach enabled us to use same materials and induction methods to instigate incidental and integral emotions in a single research design. In Experiment 1, White Australian students (N = 93) in in vivo real face-to-face contact with an ethnic tutor in their classroom displayed less interethnic bias when incidentally sad (vs. happy) or integrally happy (vs. sad). In Experiment 2, White American males' (N = 492) anti-Arab bias displayed divergent effects under incidental vs. integral (non-applicable) sadness/happiness and similar effects under incidental vs. integral (applicable) anger. The role of perceptions of agency in the emotion-inducing situation is also explored, tested, and explained drawing from mainstream emotion theory. As expected, integral and incidental applicable emotions caused valence congruent effects, at the opposite sides of the subjective agency spectrum, by encouraging the generalization of dislike from the outgroup contact partner to the outgroup as a whole. On the other hand, incidental-non-applicable emotions caused valence-incongruent effects on bias, under high agency conditions, by encouraging (non-partner-centered) heuristic processing. Because of the improved methodology, these effects can be regarded as genuine and not the byproduct of methodological artifacts. This theory-driven and empirically sound analysis of the interplay between emotion source, emotion applicability and subjective agency in intergroup contact can increase the precision of emotion-based bias reduction strategies by deepening understanding of the emotion conditions that lead to intergroup bias attenuation vs. exacerbation.

Explaining the association between subjective social status and mental health among university students using an impact ratings approach

SN Social Sciences, 2021

Previous research has found a positive association between social class and mental health among u... more Previous research has found a positive association between social class and mental health among university students. Various mediators of this association have been proposed. However, the extent to which students perceive these mediators as having an impact on their mental health has not been investigated. It is important to investigate this issue because students who do not perceive issues as having an impact on their mental health may not be motivated to address those issues. In the current study, 402 first-year undergraduate psychology students from a large Australian university indicated the extent to which 32 issues had a negative impact on their mental health over the past six months. Students rated lack of money, time management, coursework assessment items, lack of sleep, and course marks as having the largest impact on their mental health. Lack of money and time management mediated the positive association between subjective social status and mental health over (a) the past week and (b) the past month. Coursework assessment items and course marks mediated the positive association between subjective social status and satisfaction with the university experience. It is concluded that interventions should focus on these relatively high impact issues in order to address social class differences in students’ mental health and university satisfaction.

"Repeated Sampling from the Same Population?" A Critique of Neyman and Pearson's Responses to Fisher

European Journal for Philosophy of Science, 2020

Fisher (1945a, 1945b, 1955, 1956, 1960) criticised the Neyman-Pearson approach to hypothesis test... more Fisher (1945a, 1945b, 1955, 1956, 1960) criticised the Neyman-Pearson approach to hypothesis testing by arguing that it relies on the assumption of "repeated sampling from the same population." The present article considers the responses to this criticism provided by Pearson (1947) and Neyman (1977). Pearson interpreted alpha levels in relation to imaginary replications of the original test. This interpretation is appropriate when test users are sure that their replications will be equivalent to one another. However, by definition, scientific researchers do not possess sufficient knowledge about the relevant and irrelevant aspects of their tests and populations to be sure that their replications will be equivalent to one another. Pearson also interpreted the alpha level as a personal rule that guides researchers' behavior during hypothesis testing. However, this interpretation fails to acknowledge that the same researcher may use different alpha levels in different testing situations. Addressing this problem, Neyman proposed that the average alpha level adopted by a particular researcher can be viewed as an indicator of that researcher's typical Type I error rate. Researchers' average alpha levels may be informative from a metascientific perspective. However, they are not useful from a scientific perspective. Scientists are more concerned with the error rates of specific tests of specific hypotheses, rather than the error rates of their colleagues. It is concluded that neither Neyman nor Pearson adequately rebutted Fisher's "repeated sampling" criticism. Fisher's significance testing approach is briefly considered as an alternative to the Neyman-Pearson approach.

Does preregistration improve the credibility of research findings?

The Quantitative Methods for Psychology, 2020

Preregistration entails researchers registering their planned research hypotheses, methods , and ... more Preregistration entails researchers registering their planned research hypotheses, methods , and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers' biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration's historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses

Methods in Psychology, 2024

During multiple testing, researchers often adjust their alpha level to control the familywise err... more During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., "H 1,1 or H 1,2 "). However, in some cases, they do not make this inference. Instead, they make separate inferences about each of the individual hypotheses that comprise the joint hypothesis (e.g., H 1,1 and H 1,2). For example, a researcher might use a Bonferroni correction to adjust their alpha level from the conventional level of 0.050 to 0.025 when testing H 1,1 and H 1,2 , find a significant result for H 1,1 (p < 0.025) and not for H 1,2 (p > 0.025), and so claim support for H 1,1 and not for H 1,2. However, these separate individual inferences do not require an alpha adjustment. Only a statistical inference about the union alternative hypothesis "H 1,1 or H 1,2 " requires an alpha adjustment because it is based on "at least one" significant result among the two tests, and so it refers to the familywise error rate. Hence, an inconsistent correction occurs when a researcher corrects their alpha level during multiple testing but does not make an inference about a union alternative hypothesis. In the present article, I discuss this inconsistent correction problem, including its reduction in statistical power for tests of individual hypotheses and its potential causes visà-vis error rate confusions and the alpha adjustment ritual. I also provide three illustrations of inconsistent corrections from recent psychology studies. I conclude that inconsistent corrections represent a symptom of statisticism, and I call for a more nuanced inference-based approach to multiple testing corrections. ☆ I am grateful to Vinay Tummarakota for discussions that led to my explanation of Confusion IV. I declare no funding sources. I declare no conflict of interest.

Type I Error Rates are Not Usually Inflated

Journal of Trial & Error, 2024

The inflation of Type I error rates is thought to be one of the causes of the replication crisis.... more The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

The Replication Crisis is Less of a “Crisis” in Lakatos’ Philosophy of Science

Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of... more Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures. In this article, I contrast Popper’s approach with Lakatos’ (1978) approach and a related approach called naïve methodological falsificationism (NMF; Lakatos, 1978). The Popperian approach is powerful because it is based on logical refutation, but its theories are noncausal and, therefore, lacking in scientific value. In contrast, the Lakatosian approach considers causal theories, but it concedes that these theories are not logically refutable. Finally, the NMF approach subjects Lakatosian causal theories to Popperian logical refutations. However, its approach of temporarily accepting a ceteris paribus clause during theory testing may be viewed as scientifically inappropriate, epistemically inconsistent, and “completely redundant” (Lakatos, 1978, p. 40). I conclude that the replication “crisis” makes the most sense in the context of the Popperian and NMF approaches because it is only in these two approaches that replication failures represent logical refutations of theories. In contrast, replication failures are less problematic in the Lakatosian approach because they do not logically refute theories. Indeed, in the Lakatosian approach, replication failures can be legitimately ignored or used to motivate theory development.

Questionable Metascience Practices

Journal of Trial and Error, 2023

Metascientists have studied questionable research practices in science. The present article consi... more Metascientists have studied questionable research practices in science. The present article considers the parallel concept of questionable metascience practices (QMPs). A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. The present article reviews ten QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast 'n' bropen scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is "too low" without specifying an "acceptable" rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as being more "tentative" than confirmatory results; (8) presuming that questionable research practices are problematic research practices; (9) focusing on knowledge accumulation; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.

Social Identity Explanations of System Justification: Misconceptions, Criticisms, and Clarifications -Supplementary Document

by Mark Rubin and Chuma Owuamalam

European Review of Social Psychology, 2023

This document supplements our reply to Jost et al.'s (2023) rejoinder to our initial article on t... more This document supplements our reply to Jost et al.'s (2023) rejoinder to our initial article on the social identity model of system attitudes (SIMSA; Rubin et al., 2023). We respond to issues that we did not address in our main reply. Specifically, we argue that: (1) like SIMSA, Tajfel (1981) considered general, non-ideological, psychological processes; (2) system justification theory also refers to general, non-ideological, social psychological processes; (3) Jost et al. confuse respecting people with respecting people's beliefs; (4) SIMSA proponents do not ignore system justification theory's three motives; and (5) SIMSA may help to explain Osborne et al.'s (2019) results. In addition, (6) we apologise if we mischaracterised some researchers' theoretical affiliations, and we respond to Jost et al.'s (7) questions about ideology; (8) criticisms of Owuamalam et al.'s (2016) work; and (9) criticisms about the validity of Owuamalam and Spears' (2020) pupil dilation measure.

Social identity explanations of system justification: Misconceptions, criticisms, and clarifications

European Review of Social Psychology, 2023

In this article, we reply to Jost et al.’s (2023) rejoinder to our article reviewing evidence for... more In this article, we reply to Jost et al.’s (2023) rejoinder to our article reviewing evidence for the social identity model of system attitudes (SIMSA; Rubin et al., 2023). We argue that (1) SIMSA treats system justification as the outcome of an interaction between general social psychological process and specific historical, political, cultural, and ideological environments; (2) it does not conflate perceived intergroup status differences with the perceived stability and legitimacy of those differences, (3) it is not fatalistic, because it assumes that people may engage in social change when they perceive an opportunity to do so; (4) it adopts a non-reductionist, social psychological explanation of system justification, rather than an individualist explanation based on individual differences; (5) it presupposes “existing social arrangements”, including their existing legitimacy and stability, and assumes that these social arrangements are either passively acknowledged or actively supported; and (6) it is not reliant on minimal group experiments in its evidence base.

A social identity model of system attitudes (SIMSA): Multiple explanations of system justification by the disadvantaged that do not depend on a separate system justification motive

European Review of Social Psychology, 2023

System justification theory (SJT) assumes that social identity theory (SIT) cannot fully account ... more System justification theory (SJT) assumes that social identity theory (SIT) cannot fully account for system justification by members of low-status (disadvantaged) groups. Contrary to this claim, we provide several elaborations of SIT that explain when and why members of low-status groups show system justification independent from any separate system justification motive. According to the social identity model of system attitudes (SIMSA), the needs for social accuracy and a positively distinct social identity fully account for system justification by members of low-status groups. In the present article, we (a) explain SIMSA's accounts of system justification, (b) develop associated hypotheses, (c) summarise evidence that supports each hypothesis, and (d) highlight issues to be addressed in future research. We conclude that SIMSA provides a more parsimonious explanation of system justification by the disadvantaged than SJT, because it does not refer to an additional separate system justification motive.

Sleep as a mediator of the relationship between social class and health in higher education students

by Mark Rubin and Romany Mcguffog

British Journal of Psychology, 2023

A substantial body of research indicates that higher education students from lower social class b... more A substantial body of research indicates that higher education students from lower social class backgrounds tend to have poorer health than those from higher social class backgrounds. To investigate sleep as a potential mediator of this relationship, online survey responses of students from five large Australian universities, one Irish university and one large Australian technical college were analysed in three studies (Study 1 N = 628; Study 2 N = 376; Study 3 N = 446). The results revealed that sleep quality, sleep duration, sleep disturbances, pre-sleep worries and sleep schedule variability mediated the relationship between social class and physical and mental health. Sleep remained a significant mediator when controlling for related variables and other mediators. Thus, the findings suggest that sleep partly explains social class differences in health. We discuss the importance of addressing sleep issues among students from lower social class backgrounds.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

Philosophical Psychology, 2022

Preregistration has been proposed as a useful method for making a publicly verifiable distinction... more Preregistration has been proposed as a useful method for making a publicly verifiable distinction between confirmatory hypothesis tests, which involve planned tests of ante hoc hypotheses, and exploratory hypothesis tests, which involve unplanned tests of post hoc hypotheses. This distinction is thought to be important because it has been proposed that confirmatory hypothesis tests provide more compelling results (less uncertain, less tentative, less open to bias) than exploratory hypothesis tests. In this article, we challenge this proposition and argue that there are several advantages of exploratory hypothesis tests that can make their results more compelling than those of confirmatory hypothesis tests. We also consider some potential disadvantages of exploratory hypothesis tests and conclude that their advantages can outweigh the disadvantages. We conclude that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, are more appropriate in the context of unplanned deviations, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage. In contrast, confirmatory hypothesis tests may lead to an inappropriate level of confidence in research conclusions, less appropriate analyses in the context of unplanned deviations, and greater bias and errors in theoretical inferences.

A First Class Measure: Evidence for a Comprehensive Social Class Scale in Higher Education Populations

Research in Higher Education, 2022

Despite social class being a burgeoning area of research in the higher education literature, ther... more Despite social class being a burgeoning area of research in the higher education literature, there is no single comprehensive measure of social class in university student populations. Most previous research has included objective single-item measures (e.g., parent education or occupation) to assess social class and then sorted students into distinct social class categories using these items. Such approaches do not adequately capture the complexity and nuance of class, and they ignore the subjective and social components involved. The present paper reports the development and validation of an 11-item Comprehensive Social Class Scale (CSCS) that uses a mix of objective and subjective items to assess multiple aspects of social class, including education level, occupational prestige, family affluence, social class identity and subjective social status. Across 12 samples (N = 4926), we provide evidence for a single factor structure of the CSCS and demonstrate aspects of its reliability and validity. We conclude by discussing some limitations and suggestions for use of the CSCS in higher education populations.

That's Not a Two-Sided Test! It's Two One-Sided Tests

Significance, 2022

When reporting tests of significance, researchers might claim to have conducted a twosided test w... more

Binge Drinkers Shouldn't Set Their Own Alcohol Reduction Goals

Alcohol Treatment Quarterly, 2022

The present research aimed to investigate the relative effectiveness of three types of alcohol re... more The present research aimed to investigate the relative effectiveness of three types of alcohol reduction intervention. Participants were 354 university students from an Australian university. After completing an initial survey, they were randomly assigned to either (a) follow national guidelines for alcohol consumption, (b) set their own personal alcohol consumption reduction goal, or (c) work with their peers to set a goal. Participants then recorded their alcohol consumption in a drinking diary over a period of four weeks. The results showed that participants in the personal goal condition reported significantly higher alcohol consumption than those in the national guidelines conditions, with participants in the group goal condition reporting mid-level alcohol consumption. This effect was moderated by binge drinking propensity. Personal goal setting was most likely to lead to higher alcohol consumption when participants scored relatively high on measures of binge drinking propensity. These findings highlight the point that different types of alcohol reduction interventions may be effective for different types of people. In particular, people who have a relatively high propensity for binge drinking should be encouraged to follow goals that have been set by authorities (e.g., national guidelines) rather than by themselves.

A cross-sectional investigation of parenting style and friendship as mediators of the relation between social class and mental health in a university community

Introduction: This study tested a novel explanation for the positive relation between social clas... more Introduction: This study tested a novel explanation for the positive relation between social class and mental health among university students. Students with a higher social class were expected to have experienced more authoritative and less authoritarian parenting styles; these parenting styles were expected to lead to greater friendship and social integration at university; and greater friendship and integration were expected to lead to better mental health. Method: To test this model, the researchers asked 397 Australian undergraduate students to complete an online survey. The research used a cross-sectional correlational design, and the data was analysed using bootstrapped multiple serial mediation tests. Results: Consistent with predictions, parenting style, general friendship and support, and social integration at university mediated the relation between social class and mental health. Conclusions: The present results suggest that working-class parenting styles may inhibit the...

The academic outcomes of first-in-family in an Australian university: An exploratory study

International Studies in Widening Participation

Although the first-generation and first-in-family status (FIF) of university students has been of... more Although the first-generation and first-in-family status (FIF) of university students has been of intense interest in the USA, it has received very little consideration in Australia. The present research redressed this imbalance by investigating the academic outcomes of FIF undergraduate students at a large, public, Australian university. Undergraduate students (N = 227) who were enrolled in education, nursing and liberal arts degrees completed an online survey. Data are representative of typical gender enrolment patterns for these degrees. In contrast to US research, there was no clear relationship between socioeconomic status and FIF status in this sample. Consistent with US research, FIF students had poorer academic outcomes than non-FIF students. However, this difference was only significant after the first-year of study when students were less likely to receive scaffolded learning support within courses. FIF students were more likely than non-FIF students to seek support from u...

There’s No Need to Lower the Significance Threshold When Conducting Single Tests of Multiple Individual Hypotheses

Academia Letters, 2021

During null hypothesis significance testing, a multiple testing problem occurs when researchers t... more During null hypothesis significance testing, a multiple testing problem occurs when researchers test a joint null hypothesis using a union-intersection testing approach (e.g., Kim et al., 2004; Parker & Weir, 2020; Roy, 1953). A joint null hypothesis comprises two or more constituent null hypotheses that can each be subjected to separate significance tests. For example, consider the joint null hypothesis that "male participants have neither higher academic self-esteem scores nor higher social self-esteem scores than female participants." In this case, a researcher could conduct a significance test on each of the two constituent null hypotheses: (a) "male participants do not have higher academic self-esteem scores than female participants," and (b) "male participants do not have higher social self-esteem scores than female participants." During union-intersection testing, it is logical to reject the overall joint null hypothesis if at least one of its constituent null hypotheses is rejected. So, for example, it is logical to reject the joint null hypothesis that "male participants have neither higher academic self-esteem scores nor higher social self-esteem scores than female participants" if we reject the constituent null hypothesis that "male participants do not have higher academic self-esteem scores than female participants." The multiple testing problem occurs because the greater the number of constituent null hypotheses that comprise the joint null hypothesis, the greater the number of opportunities that a researcher has to incorrectly reject at least one constituent null hypothesis during unionintersection testing, and so the greater the probability of incorrectly rejecting the overall joint null hypothesis. More formally, if a researcher uses a significance threshold (alpha level) of α for each of k independent tests of their joint null hypothesis, then the Type I error rate

When to Adjust Alpha During Multiple Testing: A Consideration of Disjunction, Conjunction, and Individual Testing

Synthese, 2021

Scientists often adjust their significance threshold (alpha level) during null hypothesis signifi... more Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.

Emotions in Intergroup Contact: Incidental and Integral Emotions' Effects on Interethnic Bias Are Moderated by Emotion Applicability and Subjective Agency

by Mark Rubin and Jake Harwood

Frontiers in Psychology, 2021

This research draws from three distinct lines of research on the link between emotions and interg... more This research draws from three distinct lines of research on the link between emotions and intergroup bias as springboard to integrative, new hypotheses. Past research suggests that emotions extrinsic to the outgroup (or “incidental”), and intrinsic to the outgroup (or “integral”), produce valence-congruent effects on intergroup bias when relevant or “applicable” to the outgroup (e.g., incidental/integral anger and ethnic outgroups). These emotions produce valence incongruent effects when irrelevant or “non-applicable” to the outgroup (e.g., incidental/integral sadness and happiness, and ethnic outgroups). Internally valid and ecologically sound tests of these contrasting effects are missing; hence we examined them experimentally in meaningful settings of interethnic contact. To this end, we hybridized established research paradigms in mood and intergroup contact research; this approach enabled us to use same materials and induction methods to instigate incidental and integral emotions in a single research design. In Experiment 1, White Australian students (N = 93) in in vivo real face-to-face contact with an ethnic tutor in their classroom displayed less interethnic bias when incidentally sad (vs. happy) or integrally happy (vs. sad). In Experiment 2, White American males' (N = 492) anti-Arab bias displayed divergent effects under incidental vs. integral (non-applicable) sadness/happiness and similar effects under incidental vs. integral (applicable) anger. The role of perceptions of agency in the emotion-inducing situation is also explored, tested, and explained drawing from mainstream emotion theory. As expected, integral and incidental applicable emotions caused valence congruent effects, at the opposite sides of the subjective agency spectrum, by encouraging the generalization of dislike from the outgroup contact partner to the outgroup as a whole. On the other hand, incidental-non-applicable emotions caused valence-incongruent effects on bias, under high agency conditions, by encouraging (non-partner-centered) heuristic processing. Because of the improved methodology, these effects can be regarded as genuine and not the byproduct of methodological artifacts. This theory-driven and empirically sound analysis of the interplay between emotion source, emotion applicability and subjective agency in intergroup contact can increase the precision of emotion-based bias reduction strategies by deepening understanding of the emotion conditions that lead to intergroup bias attenuation vs. exacerbation.

Explaining the association between subjective social status and mental health among university students using an impact ratings approach

SN Social Sciences, 2021

Previous research has found a positive association between social class and mental health among u... more Previous research has found a positive association between social class and mental health among university students. Various mediators of this association have been proposed. However, the extent to which students perceive these mediators as having an impact on their mental health has not been investigated. It is important to investigate this issue because students who do not perceive issues as having an impact on their mental health may not be motivated to address those issues. In the current study, 402 first-year undergraduate psychology students from a large Australian university indicated the extent to which 32 issues had a negative impact on their mental health over the past six months. Students rated lack of money, time management, coursework assessment items, lack of sleep, and course marks as having the largest impact on their mental health. Lack of money and time management mediated the positive association between subjective social status and mental health over (a) the past week and (b) the past month. Coursework assessment items and course marks mediated the positive association between subjective social status and satisfaction with the university experience. It is concluded that interventions should focus on these relatively high impact issues in order to address social class differences in students’ mental health and university satisfaction.

"Repeated Sampling from the Same Population?" A Critique of Neyman and Pearson's Responses to Fisher

European Journal for Philosophy of Science, 2020

Fisher (1945a, 1945b, 1955, 1956, 1960) criticised the Neyman-Pearson approach to hypothesis test... more Fisher (1945a, 1945b, 1955, 1956, 1960) criticised the Neyman-Pearson approach to hypothesis testing by arguing that it relies on the assumption of "repeated sampling from the same population." The present article considers the responses to this criticism provided by Pearson (1947) and Neyman (1977). Pearson interpreted alpha levels in relation to imaginary replications of the original test. This interpretation is appropriate when test users are sure that their replications will be equivalent to one another. However, by definition, scientific researchers do not possess sufficient knowledge about the relevant and irrelevant aspects of their tests and populations to be sure that their replications will be equivalent to one another. Pearson also interpreted the alpha level as a personal rule that guides researchers' behavior during hypothesis testing. However, this interpretation fails to acknowledge that the same researcher may use different alpha levels in different testing situations. Addressing this problem, Neyman proposed that the average alpha level adopted by a particular researcher can be viewed as an indicator of that researcher's typical Type I error rate. Researchers' average alpha levels may be informative from a metascientific perspective. However, they are not useful from a scientific perspective. Scientists are more concerned with the error rates of specific tests of specific hypotheses, rather than the error rates of their colleagues. It is concluded that neither Neyman nor Pearson adequately rebutted Fisher's "repeated sampling" criticism. Fisher's significance testing approach is briefly considered as an alternative to the Neyman-Pearson approach.

Does preregistration improve the credibility of research findings?

The Quantitative Methods for Psychology, 2020

Preregistration entails researchers registering their planned research hypotheses, methods , and ... more Preregistration entails researchers registering their planned research hypotheses, methods , and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers' biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration's historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.