Linking departmental climate to the sense of belonging of chemistry graduate students and postdocs: evaluation and insights from the DCaDEI survey

Abstract

Students from historically marginalized communities are not well represented in Science, Technology, Engineering, and Mathematics (STEM) fields. While efforts have been taken to increase their participation in STEM through a top-down administrative approach, these efforts often overlook the unique climate within individual STEM departments, leading to ineffective interventions for advancing diversity, equity, and inclusion (DEI) within each distinct community. The Department of Chemistry at the University of California, Berkeley, created a survey to measure climate around DEI, which we refer to as the Departmental Climate around Diversity, Equity, and Inclusion (DCaDEI) survey. We evaluated the DCaDEI survey following the Standards for Educational and Psychological Testing and found that it can provide valid and reliable data for interpretation. Examination of the DCaDEI data revealed that graduate students and postdocs within the same chemistry department hold varying views about their DEI climate, ranging from slightly negative to very positive. It is very likely that these perceptions became more positive with continuous implementation of various DEI initiatives and interventions within the department. Furthermore, we found that a more positive departmental DEI climate leads to a higher sense of belonging among graduate students and postdocs, which is essential for their retention and success. The demonstrated validity and reliability of the data collected with the DCaDEI survey position it as a valuable instrument for assessing and longitudinally monitoring DEI climates in other chemistry departments. Leveraging DCaDEI data to inform data-driven DEI initiatives can help foster more inclusive academic chemistry environments that support the persistence and well-being of all students.

---

Introduction

Challenges faced by marginalized graduate students and postdocs and the need for climate assessments

The President's Council of Advisors on Science and Technology points out that women and students from historically marginalized groups are underrepresented in Science, Technology, Engineering, and Mathematics (STEM) fields (Olson and Riordan, 2012). For example, while racially marginalized communities represent about 31% of the United States (US) population, their presence in STEM disciplines and careers is disproportionately low. In 2014, only 21% of Bachelor's and 13% of PhD degrees were awarded to members of racially marginalized communities (Rivers, 2017). These numbers are not surprising, considering the various injustices and barriers marginalized students encounter in higher education, including racism, microaggressions, and hostile environments (Colwell et al., 2020; Ramos and Yi, 2020; Stachl et al., 2021a). These inequities have negative effects on the well-being, academic performance, and opportunities of marginalized students (Yosso, 2005; Johnson-Ahorlu, 2012). Furthermore, members of racially marginalized communities comprise only 11% of STEM professionals and 4% of faculty in research-intensive universities (Weatherton and Schussler, 2021). These data illustrate how racially marginalized communities are also less represented in the STEM workforce. Consequently, these findings have spurred initiatives aimed at identifying and addressing the disparities in STEM degrees awarded to racially marginalized students. While efforts have been taken to increase the proportion of women and historically racially marginalized groups in STEM departments through top-down administrative approaches (Colwell et al., 2020), these efforts often overlook the unique climates within individual departments, resulting in ineffective interventions for promoting diversity, equity, and inclusion (DEI) within each distinct community (Puritty et al., 2017; Colwell et al., 2020). Additionally, while we emphasize gender and race, it should be noted that any interventions that are not specifically tailored to address the unique challenges of the members of specific departments will likely continue to marginalize individuals who are not cisgender white men.

Several studies have demonstrated that creating a supportive academic climate can increase students’ sense of belonging. Sense of belonging is defined as the extent to which a person believes that they are accepted and included as a legitimate member of a community and that their presence and contributions to that community are valued (Freeman et al., 2007; Walton and Cohen, 2011; Good et al., 2012). Importantly, sense of belonging is a predictor of success and retention in academia. For example, Buckley and colleagues (2022) surveyed 75 000 undergraduate students from 82 college campuses in the US. They found that negative experiences, such as instances of discrimination, diminish the sense of belonging among students. A study by Crowe (2021) focusing on undergraduate participants in a three-year STEM scholarship program which promoted professional development, research mentorship, and faculty relationships, demonstrated that such a supportive environment resulted in heightened feelings of belonging and increased satisfaction with their chosen majors. Furthermore, Locks et al. (2008) surveyed more than 2000 undergraduate students across three different campuses, and found that when students have positive interactions with diverse peers, they experience an increased sense of belonging on their campuses.

Limited research examined the experiences of graduate students and postdocs, particularly within their respective departments. The focus on graduate students and postdocs is particularly warranted, given that their responsibilities and experiences in academia drastically differ from those of undergraduate students. Not only do graduate students and postdocs have different responsibilities, but they also conduct most of the research that sustains universities, including generating the data that secures funding through grants. Recent research has begun to address the gaps in understanding the challenges faced by marginalized graduate students in chemistry. A study by Stockard and colleagues (2021) surveyed 2544 chemistry master's and doctoral students and found that marginalized students encounter more negative experiences with their advisors and peers and often lack adequate financial support. A study by Jones and colleagues (2024) interviewed 29 marginalized domestic and international women who pursued graduate degrees in chemistry. They found that women who received predominantly positive recognition from their academic community, largely based on their research accomplishments, developed research, teaching, and altruistic science identities. In contrast, women who received primarily negative recognition, often due to their gender, race, or ethnicity, developed disrupted science identities. A disrupted science identity refers to an individual's struggle to gain recognition as a scientist from meaningful scientific others, despite their dedication and often ultimate success in the field (Carlone and Johnson, 2007). As a result, individuals with disrupted science identities may face more challenging trajectories, encountering exclusion or bias that complicates their professional and academic experiences. Another study surveyed 130 women pursuing graduate degrees in chemistry to understand why so few women seek faculty positions (Howe et al., 2022). These women reported low expectations for finding a department that values mental health and diversity and supports its community members. These findings highlight the importance of focusing on the need for departments to make fundamental changes regarding what they tangibly value and reward. To improve the experiences of graduate students and postdocs in academia, it is necessary to collect valid and reliable data to develop tailored evidence-based initiatives that align with the goals and needs of each individual departmental community.

Development of the departmental climate around diversity, equity, and inclusion (DCaDEI) survey

Recognizing these issues, the Department of Chemistry at the University of California, Berkeley (UC Berkeley), has been leading an effort since 2018 to improve its departmental climate around DEI through a comprehensive array of initiatives (Stachl et al., 2019; Brauer et al., 2021; Stachl et al., 2021a,b). These efforts include (1) developing an instrument to measure the departmental climate around DEI, which we refer to as the Departmental Climate around Diversity, Equity, and Inclusion (DCaDEI) survey (Stachl et al., 2019), (2) holding an annual department town hall to engage all community members in planning new DEI initiatives based on the DCaDEI survey data, (3) conducting monthly meetings to provide platforms for open communication on topics related to identity, belonging, current events and more, (4) prioritizing the mental health and wellness of graduate students while challenging associated social stigmas, (5) integrating student involvement in faculty hiring to increase student agency, (6) institutionalizing structures to support and reward graduate students who serve as agents of change within the department, and (7) collecting DCaDEI survey data annually to monitor the success of these change efforts. This survey and the ensuing actions emerged from a grassroots initiative led by students and community members. The DCaDEI's development involved soliciting input from stakeholders across all levels of the department. For example, before its dissemination, the survey underwent refinement based on feedback from the department chair, faculty, staff, and graduate students, improving its length, language, and content (Stachl et al., 2019).

The UC Berkeley's decision to develop a new survey was driven by the fact that existing survey instruments used to measure climate around DEI did not focus on the departmental level (Theofanos et al., 2021; University of Michigan, 2022; Hendrickson et al., 2023). However, research in related areas provided valuable insights that informed the DCaDEI survey's focus on several key factors described in more detail in the paragraphs below: advisor support, departmental support, and perceived departmental progress toward DEI.

Advisor-student relationships play a critical role in graduate education (Qu and Harshman, 2022). The American Chemical Society (ACS) Graduate Student Survey (2021) found that advisor support directly impacts student retention, sense of inclusion, and career aspirations. Other studies (Schlosser et al., 2011; Schniederjans et al., 2012; Harshman, 2021; Qu and Harshman, 2022) reinforce that the apprenticeship model of graduate education makes advisors highly influential in shaping students' academic and professional development. A department-specific climate survey must account for advisor's influence to provide a more comprehensive understanding of the graduate students and postdocs' experience.

Beyond advisors, overall departmental/organizational culture plays a major role in member's well-being and success. The American Association of Physicists in Medicine (AAPM) survey assessed workplace and organizational climate, instances of discrimination and harassment, and access to equitable opportunities (Hendrickson et al., 2023). The ACS Graduate Student Survey (2021) also identified harassment and discrimination as persistent challenges in graduate education, negatively impacting students' mental health, academic performance, and retention (Hendrickson et al., 2023). These findings emphasize the need for a departmental-level assessment to better understand the local climate and the specific challenges students face within their academic units.

Institutional efforts to advance DEI significantly affect students’ experiences. The University of Michigan Campus Climate Survey assessed participation in DEI initiatives, perceptions of DEI progress, and institutional commitment. The findings helped evaluate the effectiveness of DEI efforts and identify areas for improvement. Similarly, the National Institute of Standards and Technology (NIST) survey on Gender, Equity, and Inclusion examined organizational commitment to DEI and gendered experiences in the workplace (Theofanos et al., 2021). However, these surveys do not assess how individual departments implement and support DEI initiatives, leaving gaps in understanding how DEI progress is perceived at the departmental level.

These studies and reports collectively highlight the importance of advisor support, departmental support, and perceived departmental progress toward DEI in fostering an inclusive and equitable academic environment. The DCaDEI survey was developed with these insights in mind to provide a relevant and appropriately framed instrument. For more details about the DCaDEI's development, see the “Methods” section.

Study goals and research questions

While the DCaDEI survey has been used consistently by UC Berkeley Chemistry over the years (and now by other STEM departments at and outside of UC Berkeley), there remains a need for additional evidence to support the validity and reliability of the data it produces. Therefore, this study aims to achieve two goals: to evaluate the DCaDEI survey for broader use in other STEM departments by providing the validity and reliability evidence of its data for interpretation, and to delve into the insights offered by the DCaDEI data, particularly when analyzed in conjunction with data on students’ sense of belonging. To warrant its continued use within the Department of Chemistry at UC Berkeley and other departments, this study answers two research questions (RQs):

1. What validity and reliability evidence supports the use of the DCaDEI survey?

2. To what extent does the departmental climate around DEI relate to the sense of belonging among graduate students and postdocs within the Department of Chemistry at UC Berkeley?

Since the DCaDEI instrument focuses on evaluating departmental climate around diversity, equity, and inclusion, it is important to define the term “climate.” Organizational climate refers to the shared perceptions of policies, practices, and procedures within a workplace (Schneider et al., 2013; Shi and Stains, 2021). This climate reflects how individuals experience and interpret the behaviors that are encouraged, rewarded, or expected in their environment. In an academic department, climate encompasses faculty, staff, and students’ perceptions of inclusivity, fairness, and support, among other things. For example, it includes how individuals view mentorship practices, the distribution of resources, and the level of support for historically marginalized groups. Climate is often assessed using surveys and quantitative methods to capture trends across different levels of an organization and track progress across the years (Schneider et al., 2013). In this study, we use the DCaDEI survey to examine the DEI climate within an academic department, specifically focusing on perceptions of graduate students and postdocs across several years. By collecting these data, we both evaluate the data produced by the DCaDEI survey (RQ1), examine the DEI climate within the target department, and offer some insights into ways to create a more inclusive and supportive academic environment. Additionally, we examine the relationship between departmental climate around DEI and the sense of belonging among graduate students and postdocs. In doing so, we provide additional validity evidence for the DCaDEI survey's data by assessing its relations to other relevant variables (RQ2).

Positionality

We want to address our team's positionality, as it is essential to acknowledge how our identities impact our interpretation of data and the findings from this study (Tuhiwai-Smith, 1999; Russo-Tait, 2022). The motivation for this work was to evaluate the DCaDEI survey to provide the STEM community with a quality tool that could be used to evaluate the climate around DEI and, when applicable, inform initiatives to improve the said climate. The chemistry context was chosen as the authors are part of the chemistry community and are passionate about making science, and chemistry specifically, more inclusive and welcoming.

All authors identify as women. One of the authors was born and raised in the U.S. (C. N. S.), and two are international scholars who have lived in the U.S. for over five (L. S.) and over ten (M. P.) years. The team is also racially and ethnically diverse, with one Asian (L. S.), one Latina (C. N. S.), and one White (M. P.) researcher. All authors are at different stages of their professional trajectories, within and outside of academia. Two of the authors are affiliated with the University of North Carolina at Greensboro and include a postdoctoral scholar (L. S.) and associate professor (M. P.). C. N. S. is a founder of a company, Reflecting Equity (2022), which provides consultation services for organizations seeking to transform their workplace culture to move toward a more inclusive and equitable future. The diverse authors’ team brought different perspectives to the interpretation of the findings, improving the rigor of this work. At the same time, while we have each experienced marginalization based on our identities throughout our career paths, we recognize that we will never experience all the ways in which other individuals can be and are marginalized by oppressive social norms and structures. We hope this study brings to light the importance of community knowledge in designing efforts to assess and change department climate, to address the needs of those in our community who have lived experiences different from our own.

The study design was carried out by M. P. and L. S. The data were collected by C. N. S. when she was a graduate student at UC Berkeley, leading the grassroots efforts described above. C. N. S. was also one of the DCaDEI developers. The data were primarily analyzed by L. S., who also wrote the first draft of this manuscript. M. P. and C. N. S. revised this manuscript. Importantly, while C. N. S. was directly involved in the DCaDEI creation and improving the climate in the Department of Chemistry at UC Berkeley, L. S. and M. P. have no association with this department and institution and acted as independent evaluators of the DCaDEI's data.

Methods

The following section describes the development and administration of the DCaDEI survey in 2018–2020. We also outline our approach to handling missing data and detail the steps taken to ascertain the survey's data internal structure validity and reliability (RQ1). Additionally, we present the validity evidence based on relations to other variables by examining the relationship between DCaDEI survey and the Sense of Belonging (SoB) survey (Stachl and Baranger, 2020) results in 2019 and 2020 (RQ2). While the SoB survey is not the primary focus of this study, we provide background on its development to help contextualize the findings related to its relationship with the DCaDEI survey data.

DCaDEI and SoB surveys’ development, administration, and handling of missing data

The DCaDEI survey was designed with attention to content validity to improve its effectiveness in capturing the department's climate and DEI-related experiences (Stachl et al., 2019). First, DCaDEI survey was designed using expert input from faculty in Sociology and Demography at UC Berkeley. Additionally, incorporating insights from established climate surveys at UC Berkeley and the University of Michigan's DEI Strategic Plan (University of Michigan, 2022) further reinforced the survey's alignment with widely recognized best practices in assessing academic climate. Second, content validity was bolstered through a rigorous review process before administration. The DCaDEI survey was examined by faculty, staff, graduate students, and the department chair of the Department of Chemistry at UC Berkeley, who provided feedback to refine the instrument. This diverse review focused on improving the items so that they are clear, relevant, and appropriately framed for the department's context. However, while some important steps were taken to ascertain the content validity of the DCaDEI's data, additional measures are needed to gather evidence of the internal structure validity and reliability of the data, as well as validity based on relations to other variables.

The DCaDEI survey used for our analysis consists of 21 items selected from the original survey (Stachl et al., 2021a) and can be found in SI 1 (ESI†). Six items were excluded from the analysis for two main reasons. First, the excluded items had different response scales. The selected 21 items use a 5-point Likert scale ranging from “strongly agree” to “strongly disagree,” with an additional option for “prefer not to answer”. In contrast, the excluded items use a 3-point Likert scale ranging from “very important” to “not important”. Second, upon initial analysis of the responses, excluded items had no correlation with the rest of the data set. Since the items were unrelated, excluding any of them does not affect participants' interpretation of the remaining items. For further analysis, all responses were positively coded, with “5” indicating “strongly agree” and “1” indicating “strongly disagree”.

The DCaDEI survey was administered to graduate students and postdocs at UC Berkeley in February 2018, 2019, and 2020 (Stachl et al., 2021a). The survey was distributed through Qualtrics. Participants were offered a chance to win one of two $100 raffle prizes upon completing the survey. Additionally, a coffee and snack event was promised to be hosted by the department if participation exceeded 40%. Both incentives were advertised via email and flyers. In total, 646 responses were collected over the three years. The total response rates were 43.1% in 2018, 35.7% in 2019, and 39.4% in 2020. After excluding participants who left more than 50% of the survey items unanswered, which accounted for 46 participants (7% of the total), the analysis included responses from 600 participants (n = 210 in 2018, n = 205 in 2019, and n = 185 in 2020, first row in Table 1). In 2018, the demographic questions did not distinguish between graduate students and postdocs, so we do not know the proportion of each group that participated in the survey that year. In 2019 and 2020, the response rates from postdocs were much lower compared to graduate students, which aligns with the department's demographics: 28 postdocs responded in 2019 (∼14%) and 21 in 2020 (∼11%). To protect participants' identities, data on race, ethnicity, and academic standing (graduate student vs. postdoc) are not publicly available.

Table 1 Number of participants who completed DCaDEI and SoB surveys across the years

	n in 2018	n in 2019	n in 2020	Total N
DCaDEI participants	210	205	185	600
SoB participants	—	184	159	343
DCaDEI and SoB participants	—	184	159	343

---

A closer examination of the missing data in the 600-participant DCaDEI data set revealed that most survey items had less than 5% of missing responses. However, two items had higher non-response rates, with one item missing 6% of responses and another missing 12%. To maximize the number of participants included in further analyses, mean imputation was employed to address these missing responses (Baraldi and Enders, 2010).

To ascertain the validity evidence based on relations to other variables, we also examined the relationship between the DCaDEI and the SoB survey data. The SoB (Stachl and Baranger, 2020) survey was designed by UC Berkeley Chemistry to assess graduate students and postdocs' sense of belonging at the departmental level, using a visual narrative format items, where each item is depicted as a picture to help respondents better visualize the described scenarios (SI 2, ESI†). Multiple forms of validity evidence were gathered to ascertain the quality of the data produced by this instrument. First, content validity was established through a collaborative development process, incorporating input from graduate students, postdocs, and faculty members from the Department of Chemistry at UC Berkeley during roundtable discussions. This review focused on improving the items so that they are relevant, clear, and meaningful to the target population. Second, response process validity was supported through pilot testing and think-aloud interviews with 32 graduate students. These steps helped confirm that respondents understood the survey items as intended and that the format effectively captured their experiences. Third, the internal structure validity was demonstrated using a Wright map based on item response theory (IRT). The results showed that the logit values of survey items spanned the full range of respondent logit values, indicating that the instrument effectively measured sense of belonging across varying levels of respondent perceptions. Finally, reliability evidence was established using a partial credit model analysis, which yielded a reliability coefficient of 0.799. This indicates an acceptable level of internal consistency, meaning that the survey items reliably measure sense of belonging. Given the robust psychometric evidence supporting the SoB survey's data, no additional validation was conducted before analyzing the collected data and its relationship with the DCaDEI data.

The SoB survey uses a 5-tiered scale, with response options ranging from “do not relate” to “always relate,” and includes a “prefer not to respond” option. For analysis, responses were positively coded with “5” indicating “always relate” and “1” indicating “do not relate”. To streamline the analysis process, all SoB items were positively worded (SI 3, ESI†), and response options were reverse coded as necessary.

Since the SoB survey was developed a year later than the DCaDEI survey (Stachl and Baranger, 2020), we are able to examine the relationship between the DCaDEI and SoB data only in 2019 and 2020. Overall, 184 graduate students and postdocs completed both of the surveys in 2019 and 159 in 2020 (Table 1).

In the subsequent section, our evaluation of the DCaDEI survey's effectiveness in capturing and characterizing DEI climate adheres to the Standards for Educational and Psychological Testing (1999). This evaluation builds upon previously established evidence of effectiveness, such as expert evaluation, and encompasses three additional key areas:

1. Implementing factor analysis to ascertain the internal structure validity of the data, providing evidence that the items are related to their intended constructs (RQ1).

2. Using reliability coefficient to evaluate internal consistency, confirming that the survey items consistently measure the same underlying constructs (RQ1).

3. Examining validity evidence based on relation to other variables, specifically investigating how the DEI climate within the department correlates with the sense of belonging among graduate students and postdocs (RQ2). This analysis provides evidence of the survey's ability to capture meaningful aspects of the DEI climate that directly impact individuals' experiences and perceptions.

The sections below provide more information about each of these analyses.

Internal structure validity

To establish the internal structure validity of the DCaDEI survey's data, we used a two-step process involving Exploratory Factor Analysis (EFA) followed by Confirmatory Factor Analysis (CFA). This approach was necessary because the survey was not designed based on a specific framework that could have given insight into its internal structure.

We performed EFA on half of the responses (n = 300) to identify the underlying factor structure. Before conducting the EFA, we checked the suitability of our data by performing the Bartlett test and the Kaiser–Meyer–Olkin (KMO) test, which confirmed the homogeneity and adequacy of our sample for analysis. The number of factors to be extracted was determined using both the Scree test and parallel analysis (SI 4, ESI†). Due to the non-normal distribution of item responses and high correlations among most item variables, we used principal axis factor extraction and oblique rotation (Williams et al., 2010).

For the subsequent CFA, we applied it to the other half of the dataset (n = 300) to confirm the structure proposed by the EFA. Given the categorical nature of the Likert scale responses, we used the Weighted Least Square Mean and Variance (WLSMV) adjusted estimator for the CFA. The sample size for CFA exceeds the recommended standard of ten respondents per item (Brown, 2015). We selected the Tau equivalent indicator for CFA, which assumes equal importance of items within each factor and maintains identical factor loadings, thereby enabling the use of item mean to represent each factor (Komperda et al., 2018). Two model fit indices independent of sample size were selected for the CFA analysis: (1) Comparative Fit Index (CFI), where a value above 0.9 indicates an adequate fit (Hu and Bentler, 1999), and (2) Root Mean Square Error of Approximation (RMSEA), where values less than 0.08 indicate a good fit, while values between 0.08 and 0.1 suggest a marginal fit (Browne and Cudeck, 1992).

Internal consistency

To evaluate the internal consistency of the DCaDEI survey data, which aligns with the Tau equivalent model, we calculated Cronbach's alpha coefficient (Komperda et al., 2018). A Cronbach's alpha value above 0.7 is considered indicative of sufficient reliability for data interpretation (Streiner, 2003). This measure provides evidence that the survey items consistently reflect the underlying constructs they are intended to measure.

Validity evidence based on relations to other variables

Besides longitudinally administering the DCaDEI survey, the Department of Chemistry at UC Berkeley also collected data about the sense of belonging of their graduate students and postdocs in 2019 (n = 184) and 2020 (n = 159) using the Sense of Belonging (SoB) survey (Stachl and Baranger, 2020).

To gather the validity evidence based on relations to other variables, we explored the relationship between the SoB survey data and the DCaDEI survey data from 2019 and 2020 (third row in Table 1). To do so, Mixture Model Clustering (MMC) analysis was conducted to group graduate students and postdocs with similar perceptions of their departmental climate (Fraley and Raftery, 1998). The Bayesian Information Criterion (BIC) was used to identify the class enumeration. To assess the variations in SoB responses across the identified groups, we employed the Kruskal–Wallis H test (Ostertagova et al., 2014). Subsequently, we calculated the percentage of participants in each group, providing insights into how SoB responses were distributed across different items for each group identified through the mixture model clustering analysis.

All statistical analyses were conducted using Mplus version 7.4 (Muthén and Muthén, 2017) or R version 4.2.0 (2013). As this investigation was based on a published de-identified dataset, no IRB approval was required to conduct the analyses described in this article. The Results section below is structured around reporting our results about DCaDEI's data validity and reliability. When reporting results, to eliminate confusion, all survey item names include not only the question numbers but also the name of the corresponding surveys (e.g., DCaDEI 8 vs. SoB 8).

Results

Internal structure validity of the DCaDEI survey data

Bartlett's test indicated a significance level of p < 0.001, affirming the suitability of the data for EFA. Furthermore, the KMO test yielded a result of 0.89, well above the minimum threshold of 0.7, confirming the sample's adequacy for analysis.

Both the Scree test and parallel analysis (SI 4, ESI†) suggested the consideration of three to five factors. After evaluating various EFA models, the four-factor model was chosen as the most appropriate for interpreting and explaining our data. In this model (Table 2), nearly all DCaDEI items align well with their respective factors, exhibiting loadings above 0.30, with the exception of one item (DCaDEI 10). Additionally, one item (DCaDEI 21) demonstrated a cross-loading above 0.30.

Table 2 Factor loadings from the EFA of the DCaDEI survey data (n = 300)

Item #	Item statement	Factor
		1	2	3	4
Advisor support: orientation toward research (ASR)
DCaDEI 1	I feel that my research advisor(s): is/are easy to talk to about my research	0.79	0.05	−0.01	−0.06
DCaDEI 2	I feel that my research advisor(s): is/are available when I need advice concerning my research	0.73	−0.02	−0.05	−0.05
DCaDEI 3	I feel that my research advisor(s): provide(s) constructive feedback on my research project	0.78	0.05	0.02	−0.05
DCaDEI 4	I feel that my research advisor(s): treat(s) my ideas with respect	0.62	0.08	0.09	0.09
DCaDEI 5	I feel that my research advisor(s): encourage(s) me to attend and present at conferences	0.42	0.11	−0.10	0.20
DCaDEI 6	I feel that my research advisor(s): advocate(s) for me when appropriate	0.62	0.01	0.00	0.13
DCaDEI 7	I feel that my research advisor(s): Foster(s) a collaborative environment with minimal competition between group members	0.55	0.05	0.07	0.09
Advisor support: orientation toward well-being (ASW)
DCaDEI 8	I feel that my research advisor(s): provide(s) emotional support when necessary	0.02	0.90	−0.01	−0.02
DCaDEI 9	I feel that my research advisor(s): provide(s) non-research advice when necessary	0.06	0.86	0.03	−0.05
DCaDEI 10	I feel comfortable: speaking with my advisor(s) about non-academic career paths	0.25	0.28	0.06	0.18
DCaDEI 11	I feel comfortable: disclosing mental and/or physical health conditions that may impact my work to my advisor(s)	0.00	0.64	0.01	0.19
Departmental progress toward diversity, equity, and inclusion (DDEI)
DCaDEI 12	In my opinion, in the Department of Chemistry, I believe that: exclusionary or offensive behavior is not tolerated	0.06	−0.06	0.81	−0.03
DCaDEI 13	In my opinion, in the Department of Chemistry, I believe that: harassment of any kind is not tolerated	0.09	0.00	0.80	−0.08
DCaDEI 14	In my opinion, in the Department of Chemistry, I believe that: there is sufficient discussion of equity and inclusion	−0.04	0.04	0.75	0.00
DCaDEI 15	In my opinion, in the Department of Chemistry, I believe that: there is sufficient action toward improving equity and inclusion	−0.1	0.06	0.82	0.01
DCaDEI 16	In my opinion, as a member of the Department of Chemistry, I feel that: members of the department that identify as minorities feel valued and are included	−0.03	0.05	0.65	0.16
Departmental support (DS)
DCaDEI 17	I feel comfortable: seeking feedback and/or advice on my work from other faculty	0.11	−0.04	−0.01	0.72
DCaDEI 18	I feel comfortable: attending and participating in social events hosted by the Chemistry Graduate Life Committee (CGLC)	0.21	−0.14	0.08	0.37
DCaDEI 19	In my opinion, as a member of the Department of Chemistry, I feel that: there are faculty members other than my research advisor(s) who are available to me when I need advice	−0.09	0.09	−0.05	0.75
DCaDEI 20	In my opinion, as a member of the Department of Chemistry, I feel that: I know whom to talk with about any concerns regarding the departmental climate	0.01	0.15	0.13	0.47
DCaDEI 21	In my opinion, as a member of the Department of Chemistry, I feel that: I am valued and included as a member of the department	0.21	−0.05	0.40	0.46

---

To determine the best-fitting model for our data, we tested several CFA models, including options in which we: (1) removed item DCaDEI 10, (2) removed item DCaDEI 21, (3) assigned item DCaDEI 21 to factor 4 while retaining item DCaDEI 10, (4) assigned item DCaDEI 21 to factor 3 while retaining item DCaDEI 10 and (5) removed item DCaDEI10 and DCaDEI 21. These models were evaluated based on their CFI and RMSEA results (SI 5, ESI†). Ultimately, model 3 was selected as the best model, exhibiting a CFI of 0.92, which is above the 0.9 benchmark, and an RMSEA of 0.10, indicating a marginal fit. All factor loadings exceeded 0.7 (Table 3). The correlation among factors ranged from 0.35 to 0.71 (Table 4), suggesting moderate relationships. No correlations exceeded the threshold of 0.85 (Brown, 2015), supporting the discriminant validity of the constructs. These results indicate that the DCaDEI survey data align well with the established standards of goodness-of-fit.

Table 3 Standardized factor loadings from the CFA of the DCaDEI survey data (n = 300)

Factors	Item #	Factor loading^a
a Each item within the same factor has an equal factor loading due to the Tau equivalent indicator, with the factor loading representing the value for each item in a given factor.
Advisor support: orientation toward research (ASR)	DCaDEI 1–7	0.77
Advisor support: orientation toward well-being (ASW)	DCaDEI 8–11	0.82
Departmental progress toward diversity, equity, and inclusion (DDEI)	DCaDEI 12–16	0.86
Departmental support (DS)	DCaDEI 17–21	0.73

---

Table 4 Correlation matrix from the CFA of the DCaDEI survey data (n = 300)

Factors	ASR	ASW	DDEI	DS
ASR	1	0.71	0.42	0.54
ASW		1	0.35	0.61
DDEI			1	0.60
DS				1

---

The interpretation of the DCaDEI survey items within each factor yielded four subcategories:

1. A̲dvisor s̲upport: orientation toward r̲esearch (ASR): this factor focuses on the advisor's role in supporting students' research endeavors. It includes aspects such as providing feedback on research, appreciating students' ideas, encouraging participation in academic activities, and fostering a collaborative academic environment.

2. A̲dvisor s̲upport: orientation toward w̲ell-being (ASW): this factor reflects the advisor's attention to students' well-being. It encompasses mental and physical health support, career guidance, and life advice beyond research activities.

3. D̲epartmental progress toward d̲iversity, e̲quity, and i̲nclusion (DDEI): this factor captures students' perceptions of their department's commitment to creating an inclusive environment, promoting diversity, and enforcing a zero-tolerance policy for any form of exclusion or harassment.

4. D̲epartmental s̲upport (DS): this factor reflects students’ perceptions of whether the department provides a supportive environment. It includes aspects such as the ability to seek advice from other faculty members, feeling comfortable attending departmental events, feeling safe expressing concerns about the departmental climate, and feeling included as part of the department.

Internal consistency of the DCaDEI survey data

The internal consistency of each factor was evaluated using Cronbach's alpha. The Cronbach's alpha values ranged from 0.79 to 0.88 (Table 5), indicating that the survey data was sufficiently reliable for interpretation.

Table 5 Cronbach's alpha values, means, and standard deviation for each DCaDEI survey factor (n = 600)

Factors	Cronbach's α	Mean	SD
Advisor support: orientation toward research (ASR)	0.87	3.97	0.81
Advisor support: orientation toward well-being (ASW)	0.84	3.28	1.03
Departmental progress toward diversity, equity, and inclusion (DDEI)	0.88	3.35	0.90
Departmental support (DS)	0.79	3.25	0.89

---

Types of departmental climate around DEI and changes in the climate over time

To establish validity evidence based on relations to other variables, we explored the relationship between the SoB survey data (n = 343) and the DCaDEI survey data (n = 343) from 2019 and 2020. As a preliminary step to this analysis, we examined the DCaDEI data for the different types of departmental climates experienced around DEI from 2018 to 2020 (n = 553). Note that while we had access to three years of DCaDEI data (2018–2020), there are only two years of SoB data (2019–2020) because this instrument was designed and administered for the first time in 2019. Additionally, some participants only responded to the DCaDEI survey, which led to a decrease in total responses available for the analysis. To avoid counting participants multiple times, and because we do not have access to each participant's demographic information, we grouped participants and compared them within the same year.

To gain deeper insights into participants' perceptions of the departmental climate around DEI, we conducted separate mixture model clustering analyses for the years 2018, 2019, and 2020, using the mean scores of each subcategory. For model selection, we relied on BIC and BIC differences (Table 6). The models, denoted by three letters (e.g., VEE), vary in their structure, with the accompanying number indicating the number of clusters (Scrucca et al., 2016).

Table 6 Mixture model clustering (MMC) class enumeration for the 2018–2020 data (n = 553)

	Model type and number of classes for the 2018 data (n = 210)
	EEE, 1	EEV, 1	EVE, 1	VEE, 3
BIC	−2067.318	−2067.318	−2067.318	−2073.6
BIC difference	0	0	0	−6.282

---

	Model type and number of classes for the 2019 data (n = 184)
	VEE, 2	VVE, 2	VEE, 3
BIC	−1752.3	−1761.6	−1764.5
BIC difference	0.0	−9.3	−12.2

---

	Model type and number of classes for the 2020 data (n = 159)
	EEE, 2	EEE, 1	EEV, 1	VEE, 3
BIC	−1472.3	−1480.3	−1480.2	−1486.1
BIC difference	0.0	−8.0	−7.9	−13.8

---

In 2018, the suggested model had the lowest absolute BIC value, resulting in one cluster (Table 5), limiting further data analysis. The analysis of the 2019 data indicated that the “VEE,2” model had the lowest absolute BIC value (Table 6). This resulted in only two clusters, limiting the depth of data interpretation. The second-best fit model “VVE,2” also yielded two clusters. Therefore, we opted for the “VEE,3” model, which divided participants into three clusters allowing for a more nuanced analysis. In 2020, the BIC and BIC difference suggested either the “EEE” or “EEV” models as the best fit (Table 5). However, these models resulted in only one or two clusters, which restricted the depth of interpretation. To maintain consistency with the 2019 results, we selected the three-cluster “VEE” model for the 2018 and 2020 data analysis, allowing for a more detailed examination of the data. The careful examination of the three clusters revealed three climate categories each year representing distinct groups with notable differences across the years. In 2018, three clusters explained the data – those holding slightly negative, neutral, or positive perceptions of their climate around DEI, whereas in 2019 and 2020 these clusters shifted to group those holding neutral, positive, or very positive perceptions. Table 7 presents the estimated model proportions of these climate categories, as well as the calculated mean scores for each subcategory across the three years.

Table 7 Estimated model proportions and estimated means for the selected model across 2018–2020 DCaDEI data. The numbers in the table represent the average level of agreement on items within each factor, ranging from 1 (“strongly disagree”) to 5 (“strongly agree”)

Year	Type of climate (estimated model proportion)
	2018 (n = 210)			2019 (n = 184)			2020 (n = 159)
Factor	Slightly negative (18%)	Neutral (67%)	Positive (15%)	Neutral (55%)	Positive (36%)	Very positive (10%)	Neutral (20%)	Positive (64%)	Very positive (16%)
ASR	2.7	4.1	4.6	3.7	4.4	4.7	3.2	4.3	4.6
ASW	2.4	3.3	3.8	2.7	3.8	4.6	2.6	3.4	4.5
DDEI	2.6	3.1	4.1	3.3	3.9	4.7	3.0	3.8	4.3
DS	2.9	3.3	4.4	2.8	3.7	4.5	2.9	3.4	4.3

---

As shown in Table 7, in 2018, around 18% of graduate students and postdocs had a slightly negative stance across most of the DCaDEI scales. Participants in this group reported minimal support from their research advisors regarding well-being (M = 2.4) and only slightly more positive experiences in terms of research support (M = 2.7). They also perceived the departmental support (M = 2.9) and effort towards DEI (M = 2.6) as somewhat insufficient and unsatisfactory. The majority of participants, 67%, had an overall neutral stance on the climate around DEI. This group acknowledged substantial research support from their advisors (M = 4.1) and some level of care for their well-being (M = 3.3). They also noticed some department's efforts to create an inclusive environment (M = 3.1) and felt moderately supported as members of the department (M = 3.3). The last group, comprising 15% of the participants, held a positive view of the climate around DEI. This group reported higher mean scores, ranging from 3.8 to 4.6, indicating strong support from their research advisors in both research and well-being aspects. These participants also valued the department's efforts in creating a supportive and inclusive climate, feeling valued and well-supported as members of the department.

Compared to 2018, in 2019, no participants were categorized into the slightly negative group. Most of the graduate students and postdocs (55%) had a neutral stance across most of the DCaDEI subcategories. Participants in this group reported moderate support from their research advisors regarding well-being (M = 2.7) and a more positive experience in terms of research support (M = 3.7). They also recognized receiving a moderate level of departmental support (M = 2.8) and noticed some effort toward DEI (M = 3.3). The next group, which held an overall positive view of the climate around DEI, included 35% of the 2019 participants. This group acknowledged substantial research support from their advisors (M = 4.4) and a considerable level of care for their well-being (M = 3.8). They also recognized the department's efforts to create an inclusive environment (M = 3.9) and felt supported as members of the department (M = 3.7). A smaller group, comprising 10% of the participants, exhibited a very positive view of the climate around DEI. This group reported high mean scores ranging from 4.5 to 4.7, indicating significant support from research advisors in both research and well-being aspects. Additionally, this group highly valued the department's efforts in creating a supportive and inclusive climate and felt very valued and supported as members of the department.

In 2020, the distribution of participants across the DEI climate perception groups shifted notably compared to 2019. Only 20% of participants were in the neutral group. Conversely, 64% of participants held positive perceptions and 16% held very positive perceptions of their department's climate around DEI. This shift indicates a marked increase in positive perceptions, with 80% of participants viewing the climate more favorably in 2020 (as positive or very positive), compared to 45% in 2019 and 15% in 2018 (Fig. 1). This change suggests that the efforts made by UC Berkeley's chemistry department to enhance the DEI climate were likely effective, leading to a positive shift in student perceptions and recognition for creating a more supportive and inclusive environment. More research is needed to determine the exact factors driving these positive changes.

Fig. 1 Changes in graduate students and postdocs’ perceptions of departmental climate around DEI for 2018 (n = 210), 2019 (n = 184), and 2020 data (n = 159).

Validity evidence based on relations to other variables—the relationship between the perceived departmental climate around DEI and graduate students’ and postdocs’ sense of belonging

Sense of belonging is a critical factor in student retention, mental well-being, academic success, and overall satisfaction with their educational experience (Inzlicht and Good, 2006; Walton and Cohen, 2011; Good et al., 2012). Here, we examine how students' perceptions of their departmental climate around DEI relate to their sense of belonging. The SoB survey was administered alongside the DCaDEI survey in both 2019 and 2020. After calculating the correlations between each SoB item, 92% of the correlation coefficients were smaller than 0.4 (SI 6, ESI†), indicating a weak to negligible correlation (Schober et al., 2018). Therefore, we analyzed responses to each item separately. Due to the categorical nature of the response options in the SoB survey, we used the Kruskal−Wallis H test to compare the responses within each type of 2019 and 2020 DCaDEI climate group (neutral, positive, and very positive) across each SoB survey item. Our analysis revealed significant differences (p < 0.05) in eight out of the thirteen SoB items (SoB2, SoB3, SoB5, SoB7, SoB9, SoB11, SoB12, and SoB13) across the three DEI climate groups (Fig. 2). The non-significant items revealed either the same trends as the eight significant items (SoB1 and SoB10) or no clear trends in relationship to the experienced departmental climate (SoB4, SoB6, and SoB8) (Fig. 3 and SI 7, ESI†).

Fig. 2 Distribution of Sense of Belonging (SoB) survey responses across different types of department climate perceptions for (a) 2019 (n = 184) and (b) 2020 (n = 159), showing eight items with significant differences (p < 0.05) according to the Kruskal–Wallis H test.

Fig. 3 Distribution of Sense of Belonging (SoB) survey responses across different types of department climate perceptions for (a) 2019 (n = 184) and (b) 2020 (n = 159), showing five items with non-significant differences (p > 0.05) according to the Kruskal–Wallis H test.

Fig. 2 indicates that graduate students and postdocs who experienced a “very positive” climate in both 2019 and 2020 were significantly more likely to respond with “always relate” to the eight items compared to those who experienced a “positive” or “neutral” climate. Conversely, participants in the other two groups were less likely to answer “don’t relate” to the same eight items. These results indicate that participants who held a more positive view of the departmental climate around DEI were more likely to also feel an increased sense of belonging in various aspects. Specifically, those with a more favorable view of the departmental climate reported higher affinity to scenarios such as feeling smart enough to be in the department (SoB2), having faculty in the department whom they could talk to about their hardships (SoB3), feeling happy and accepted by the department (SoB5), feeling they are productive and scientifically successful in their research (SoB7), having research group members whom they could ask for help with research (SoB9), and not feeling like outsiders in the department (SoB11). They also reported feeling that their ideas are valued by their advisor (SoB12) and feeling like independent, confident scientists (SoB13).

The analysis of items from the Kruskal–Wallis H test items that showed no significant difference (Fig. 3) revealed either the same trends as the significant items discussed above or no clear trends in relationship to the experienced departmental climate. Specifically, although not statistically significant, two items (SoB1 and SoB10) demonstrated the same trends that graduate students and postdocs in the “very positive” climate groups in 2019 and 2020 tended to more frequently relate to the statements: (1) my classmates do not get better grades than I do (SoB1), and (2) I have peers in other research groups that I could ask for help with research (SoB10).

Conversely, three other items (SoB4, SoB6, and SoB8) did not exhibit any distinct trends across different departmental climate groups. These items relate to having a supportive social network (SoB4), comfort in discussing teaching with peers (SoB6), and being perceived as a serious scholar by others (SoB8). In exploring why graduate students' perceptions of the departmental climate around DEI did not result in statistically different responses to the SoB4 and SoB8 items, it is important to consider the complex nature of the sense of belonging and its relationship with departmental climate. An intriguing observation emerged from the responses to SoB6, which asked whether participants felt comfortable discussing teaching with their peers. Most respondents (∼80%), irrespective of their climate group, indicated that they “don’t relate” or “rarely relate” to this statement. This is particularly noteworthy considering the contrasting positive responses to discussing research with peers (SoB9) but not teaching (SoB6). This discrepancy suggests that, within this department, graduate students and postdocs may either undervalue teaching compared to research or not view their peers as valuable resources for teaching advice. The lack of variation in responses to the SoB6 item could be attributed to a culture within the academic environment that places greater emphasis on research over teaching. Additionally, this finding could stem from a lack of formal training related to teaching, making it a less common topic of discussion among peers. Overall, this finding points to a potential gap in the academic culture where teaching is undervalued or insufficiently supported, suggesting a need for more initiatives that encourage open conversations and professional development around teaching within the department. Further investigation is needed to understand why graduate students and postdocs in this department readily discuss research with each other but not teaching.

Discussion

This study had two primary goals. The first was to evaluate the DCaDEI survey for broader use in other STEM departments by assessing the validity and reliability of its data for interpretation (RQ1). The second was to explore the insights provided by the DCaDEI data, particularly when analyzed alongside the SoB survey data (RQ2). The DCaDEI instrument was originally developed by UC Berkeley's Chemistry Department (Stachl et al., 2019) to assess how graduate students and postdocs in STEM academic departments at research-intensive institutions perceive DEI in their departments. The survey categorizes respondents into those who experience climate on a continuum from slightly negative to very positive. Establishing validity and reliability for the DCaDEI's survey data can support its broader use in other departments (RQ1). The second goal extended our investigation by examining how departmental climate around DEI interacts with graduate students’ and postdocs' sense of belonging. Analyzing these relationships not only deepened our understanding of climate and belonging but also strengthened the survey's validity by providing evidence based on its relationship with other relevant constructs (RQ2). This analysis aligns with best practices for survey validation, as outlined in the Standards for Educational and Psychological Testing (1999).

Concerning our first goal, we employed several methods to add to the existing evidence (Stachl et al., 2019) around the validity and reliability of the DCaDEI survey data: factor analysis to establish the internal structure validity, reliability coefficient to establish internal consistency of the data, and mixture model clustering as well as the Kruskal–Wallis H test to establish validity evidence based on relations to other variables. The factor analysis results indicated that a four-factor model best fits our dataset, with items loading onto four latent subcategories: (1) advisor support: orientation toward Research (ASR), (2) advisor support: orientation toward Well-being (ASW), (3) departmental progress toward Diversity, Equity, and Inclusion (DDEI), and (4) departmental support (DS). The ASR subcategory highlights advisors' role in fostering students' research progress and success, including providing feedback, valuing ideas, and encouraging participation in academic activities. The ASW subcategory emphasizes advisors' focus on students' overall well-being, offering guidance on mental and physical health and providing support for life beyond academia. The DDEI subcategory relates to students' views on their department's efforts to create a welcoming and equitable environment, including upholding policies against discrimination and harassment. Finally, the DS subcategory concerns students’ perception of whether the department provides a supportive environment, including access to advice from other faculty within the department, the ability to express concerns about the departmental climate, and feeling included as part of the department. The internal consistency of each factor was evaluated using Cronbach's alpha. The Cronbach's alpha values ranged from 0.79 to 0.88, indicating that the survey data was sufficiently reliable for interpretation.

We found that the graduate students and postdocs experienced either slightly negative, neutral, positive, or very positive departmental climate around DEI across the years (2018–2020). The variation in perceptions, from slightly negative to very positive, suggests that there is no uniform experience of DEI within the department. Should other departments choose to administer the DCaDEI survey, the existence of slightly negative and neutral perceptions could highlight areas where the departments should focus their efforts. For instance, targeted interventions could be developed to support those who feel less included or supported, such as enhanced mentorship programs, peer support groups, or specific resources aimed at addressing the needs of underrepresented groups.

Notably, the positive shift in our data over the years implies that the Department of Chemistry at UC Berkeley was likely successful at implementing policies and practices that promote an inclusive and equitable climate. Specifically, starting in 2018, the department engaged in numerous initiatives, including developing the DCaDEI survey (Stachl et al., 2019) and administering it annually, hosting an annual department town hall to engage all community members in planning new DEI initiatives based on survey data, and holding monthly meetings to provide platforms for open communication on topics such as identity and belonging. The department also focused on supporting the mental health and wellness of graduate students, integrated student involvement in faculty hiring to increase student agency, and institutionalized support and recognition for graduate students serving as agents of change within the department. Evaluating graduate students’ and postdocs’ perceptions of their department's DEI climate using the DCaDEI survey over a span of multiple years (2018–2020) offered insights into the effectiveness of the initiatives described above. Importantly, the absence of participants falling into the slightly negative climate category in 2019 and 2020 and the substantial decrease in participants falling into the neutral category in 2020 is an encouraging and noteworthy finding as it contrasts with many studies that often report significant challenges and negative experiences in academic environments, especially in STEM fields (Johnson-Ahorlu, 2012; Colwell et al., 2020; Ramos and Yi, 2020). These findings reinforce the conclusion that the DCaDEI survey is an effective tool for measuring, monitoring, and improving departmental climate around DEI in STEM, with its main strengths including reasonable length for completion and a focus on department-level issues in research-intensive STEM academic spaces.

Concerning our second goal, our exploration into the relationship between DEI climate and sense of belonging underscored that graduate students and postdocs tend to experience a stronger sense of belonging if they perceive a more supportive departmental climate. Specifically, those who experienced a more supportive environment felt more confident and productive in their scientific work (e.g., SoB2, SoB7, SoB13), better supported by their research group (e.g., SoB9, SoB12), and more accepted within the department (e.g., SoB3, SoB5, SoB11). These findings align with previous research. For instance, Curtin et al. (2013) surveyed more than 800 graduate students who began their university graduate programs between 1998 and 2003. They found that both domestic and international doctoral students reported a strong sense of belonging when they maintained a positive relationship with their advisors (which relates to DCaDEI's factors ASR and ASW). Similarly, Solem et al. (2011) analyzed logs from 53 geography graduate students across nine institutions to explore their feelings of belonging, the challenges they encountered, and the strategies they used to overcome these challenges. The study found that most students perceived their departmental working environment as unfriendly and difficult (which relates to DCaDEI's factors DS and DDEI), which negatively impacted their sense of belonging. The authors recommended incorporating a variety of events, such as social activities related to mental health, to enhance students' sense of belonging. Our study not only corroborates these findings but also emphasizes that graduate students and postdocs are more likely to feel a strong sense of belonging as scientists when they experience a supportive DEI climate.

At the same time, while the experienced departmental climate around DEI significantly impacts graduate students and postdocs’ sense of belonging, certain aspects—such as social support (SoB4), scholarly recognition (SoB8), and discussions about teaching with peers (SoB6)—may be influenced more by individual experiences and communal values, which can vary independently of broader DEI efforts. This suggests that while a supportive DEI climate is important, these specific aspects of belonging might require targeted approaches that address personal and broader community dynamics. For example, Wao et al. (2010) found that students' perceptions of departmental climate do influence their sense of belonging, social support, and scholarly recognition, but these perceptions can be distinct from the broader departmental initiatives around DEI. Additionally, Renick and Reich (2020) highlight that secondary students' perceptions of safety and relationships with others are predictive of their sense of belonging. Specifically, approximately half of the graduate students and postdocs tended not to think that others see them as serious scholars (SoB8), regardless of which climate group they represented. This finding could potentially suggest that these participants experienced an imposter phenomenon where individuals, despite their achievements, feel inadequate and doubt their abilities (Clance and Imes, 1978). This pervasive self-doubt can lead graduate students and postdocs to deny their identity as serious scholars, even when they find themselves in supportive environments. Moreover, the high standards and competitive nature of academia may lead graduate students and postdocs to feel that they have not yet “earned” the label of a serious scholar, especially the graduate students who are still in the process of obtaining their degrees. They might perceive their accomplishments as insufficient compared to those of more established academics. Moreover, the ingrained academic norms that define what success looks like often leave graduate students and postdocs feeling like they are not serious scholars if they do not receive validation or recognition from mentors, peers, and/or the broader academic community (Carlone and Johnson, 2007; Avraamidou, 2020, 2022; Jones et al., 2024; Pfeifer et al., 2024). This lack of affirmation could be a contributing factor, regardless of the perceived DEI climate. This finding suggests that departmental climate alone may not be sufficient to address deep-seated feelings of inadequacy or self-doubt. Targeted interventions, such as mentoring, workshops on self-efficacy, peer support groups, or destigmatizing counseling and therapy services, may be necessary to help graduate students and postdocs overcome imposter phenomenon and embrace their scholarly identities (Cokley et al., 2013; Mount and Tardanico, 2014; Parkman, 2016). Additionally, Departments and advisors might need to foster a culture that values progress and growth as much as final outcomes and provide mentees with more explicit and regular recognition of their scholarly contributions to help graduate students and postdocs see themselves as serious scholars at all stages of their careers.

Lastly, most participants (∼80%) reported that they do not discuss teaching with their peers. This could be attributed to a culture within certain academic environments, especially in research-intensive institutions, that places greater emphasis on research over teaching. Addressing this issue could involve creating spaces where teaching is openly valued and intentionally discussed, such as teaching-focused seminars or informal teaching groups, to normalize and encourage conversations and reflections about teaching. Encouraging more dialogue about teaching, establishing a culture of periodic teaching peer observations, and rewarding quality teaching through explicit recognition and awards could enhance teaching skills (Lane et al., 2018; Jones et al., 2024). Further research is needed to understand why graduate students and postdocs in this department readily discuss research with each other but not teaching.

Limitations

While our study provides valuable insights into the departmental climate around DEI and its impact on the sense of belonging among graduate students and postdocs, several limitations should be acknowledged. First, the study's findings are based on data from a single chemistry department at a research-intensive university. This specificity may limit the generalizability of our results to other departments, disciplines, or institutions with different cultures and practices. However, given the psychometric analyses conducted with the DCaDEI survey showing the evidence of the validity and reliability of the data it generates, we recommend the use of the DCaDEI survey in other similar contexts. When using the DCaDEI survey, we recommend conducting validity and reliability checks with the collected dataset, including cognitive interview data (Peterson et al., 2017) that can also serve as a valuable resource for triangulation with the survey data, allowing for a more in-depth understanding of the departmental climate around DEI. Second, due to concerns with confidentiality, the scope of the survey, and the IRB restrictions, detailed demographic information such as age, gender, race, nationality, or academic standing (e.g., graduate student vs. postdoc) was not considered in the analysis. Cultural and social factors may influence how individuals perceive themselves in the academic community. This limitation restricted our ability to analyze how different demographic factors might intersect with perceptions of the departmental climate. For example, graduate students might be more likely to not think that others see them as serious scholars (SoB8) as they are in an earlier stage in their academic and career trajectory compared to postdocs. However, we are not able to compare the responses of graduate students and postdocs. Third, as with any survey-based research, our study relies on self-reported data, which may be subject to biases. Participants' responses could be influenced by their mood at the time of completing the survey or their desire to respond in socially acceptable ways (Saleh and Bista, 2017). Fourth, there is a possibility that those who chose to participate in the survey differ from those who did not, potentially leading to non-response bias. As a reminder, the total response rates were 43.1% in 2018, 35.7% in 2019, and 39.4% in 2020 of all graduate students and postdocs in the department. Fifth, we combined survey responses across three years (2018–2020) to assess the internal structure of the DCaDEI survey. This approach was necessary to ensure adequate sample size for factor analysis. At the same time, the survey instrument did not change across these years, meaning that the same items and in the same order were administered each year. Finally, while the quantitative nature of the survey provided large data and sample size, it may not capture the depth and nuances of participants' experiences and perceptions that qualitative methods might reveal. These limitations highlight areas for future research and suggest caution in applying our findings beyond the specific context of this study.

Conclusions

The psychometric analyses conducted in this study demonstrate that the Departmental Climate around Diversity, Equity, and Inclusion (DCaDEI) survey provides valid and reliable data for interpretation. We found that the graduate students and postdocs experienced either slightly negative, neutral, positive, or very positive departmental climate around DEI across the years (2018–2020). Notably, these perceptions shifted positively over time, indicating that the Department of Chemistry at UC Berkeley has successfully implemented policies and practices that promoted a more inclusive and supportive climate.

The study also explored the relationship between perceptions of the departmental climate around DEI and the sense of belonging among graduate students and postdocs. The results indicate that a more positive perception of the climate around DEI is associated with a higher sense of belonging. Our study adds to the existing body of research by focusing on the context of a large chemistry department at a research-intensive university. By zeroing in on this specific context, our research addresses a gap in the current understanding of how DEI climate influences the sense of belonging and the overall experiences of graduate students and postdocs in STEM fields, with a particular emphasis on chemistry. At the same time, while the experienced departmental climate around DEI significantly impacts graduate students and postdocs’ sense of belonging, the specific aspects of social support, scholarly recognition, and discussions about teaching may be influenced more by individual experiences and/or communal values, which can vary independently of broader DEI efforts.

For departments looking to use the DCaDEI survey, we recommend conducting validity and reliability checks to ensure the instrument produces accurate data within their specific institutional context, provided they have a sufficient sample size. Additionally, interviews with graduate students and postdocs are essential for gathering qualitative insights into how participants interpret survey items. These interviews can also help triangulate findings, providing a deeper understanding of how individuals experience DEI within their department. This is particularly relevant given that our findings highlight that perceptions of DEI vary, ranging from slightly negative to very positive, indicating that there is no single, uniform experience of departmental climate. If other departments choose to administer the DCaDEI survey, identifying areas where some individuals feel less included or supported can help target specific interventions, such as mentorship programs, peer support groups, or tailored resources. For departments with smaller sample sizes or limited experience with mixture model clustering analysis, a simpler approach is to calculate the average scores for items within each factor identified in our factor analysis. These averages can then be compared to the mean values in Table 7, allowing departments to categorize participants into different groups without requiring complex statistical modeling. As STEM departments work toward building more inclusive and equitable environments, using data can ensure that DEI efforts are informed, targeted, and impactful.

Ethical considerations

The data analyzed in this study is from several deidentified, publicly available datasets published elsewhere (Stachl et al., 2021a). Collection of this data has been approved by the University of California, Berkeley's Institutional Review Board.

Authors' contributions

M. P. conceptualized the study, and the study design was carried out by M. P. and L. S. C. N. S. collected and published the DCaDEI (Stachl et al., 2021a) and SoB (Stachl and Baranger, 2020) data. The data were primarily analyzed by L. S., who also wrote the first draft of this manuscript. M. P. and C. N. S. revised this manuscript. All authors read and approved the final manuscript.

Data availability

The data analyzed in this study is from several deidentified, publicly available datasets published elsewhere (Stachl et al., 2021a; Stachl and Baranger, 2020).

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

We want to thank all the participants who voluntarily provided their time in filling out the surveys. We would also like to thank the members of the Popova Research Group for their valuable feedback on this manuscript.

References

1. American Chemical Society, (2021), 2019 ACS Graduate Student Survey Report, Washington, DC DOI:10.1021/acs.edu.2019gssrpt.
2. American Educational Research Association, American Psychological Association and National Council on Measurement in Education, (1999), Standards for educational and psychological testing.
3. Avraamidou L., (2020), Science identity as a landscape of becoming: rethinking recognition and emotions through an intersectionality lens, Cult. Stud. Sci. Educ., 15(2), 323–345.
4. Avraamidou L., (2022), Identities in/out of physics and the politics of recognition, J. Res. Sci. Teach., 59(1), 58–94.
5. Baraldi A. N. and Enders C. K., (2010), An introduction to modern missing data analyses, J. School Psychol., 48(1), 5–37.
6. Brauer D. D., Mizuno H., Stachl C. N., Gleason J. M., Bumann S., Yates B. and Baranger A. M., (2021), Mismatch in perceptions of success: investigating academic values among faculty and doctoral students, J. Chem. Educ., 99(1), 338–345.
7. Brown T. A., (2015), Confirmatory factor analysis for applied research, Guilford publications.
8. Browne M. W. and Cudeck R., (1992), Alternative ways of assessing model fit, Sociol. Methods Res., 21(2), 230–258.
9. Buckley J. B., George C., Colston J. and Burrell A., (2022), Campus Climate and Sense of Belonging across Gender in ROTC, J. Student Affairs Res. Pract., 1–15.
10. Carlone H. B. and Johnson A., (2007), Understanding the science experiences of successful women of color: science identity as an analytic lens, J. Res. Sci. Teach., 44(8), 1187–1218.
11. Clance P. R. and Imes S. A., (1978), The imposter phenomenon in high achieving women: dynamics and therapeutic intervention, Psychotherapy, 15(3), 241.
12. Cokley K., McClain S., Enciso A. and Martinez M., (2013), An examination of the impact of minority status stress and impostor feelings on the mental health of diverse ethnic minority college students, J. Multicult. Couns. Dev., 41(2), 82–95.
13. Colwell R., Bear A. and Helman A., (2020), Promising practices for addressing the underrepresentation of women in science, engineering, and medicine: Opening doors, The National Academies Press.
14. Crowe J. A., (2021), Creating a departmental climate that increases a student's sense of belonging, perceived faculty support, and satisfaction with the major, Innov. High. Educ., 46(1), 95–109.
15. Curtin N., Stewart A. J. and Ostrove J. M., (2013), Fostering academic self-concept: advisor support and sense of belonging among international and domestic graduate students, Am. Educ. Res. J., 50(1), 108–137.
16. Fraley C. and Raftery A. E., (1998), How many clusters? Which clustering method? Answers via model-based cluster analysis, Comput. J., 41(8), 578–588.
17. Freeman T. M., Anderman L. H. and Jensen J. M., (2007), Sense of belonging in college freshmen at the classroom and campus levels, J. Exp. Educ., 75(3), 203–220.
18. Good C., Rattan A. and Dweck C. S., (2012), Why do women opt out? Sense of belonging and women's representation in mathematics, J. Personality Soc. Psychol., 102(4), 700.
19. Harshman J., (2021), Review of the challenges that face doctoral education in chemistry, J. Chem. Educ., 98(2), 259–269.
20. Hendrickson K. R., Avery S. M., Castillo R., Cervino L., Cetnar A., Gagne N. L. and Octave N., (2023), 2021 AAPM equity, diversity, and inclusion climate survey executive summary, Int. J. Radiat. Oncol. Biol. Phys., 116(2), 295–304.
21. Howe M. E., Kim M. M. and Pazicni S., (2022), Graduate Student Women's Perceptions of Faculty Careers: The Critical Role of Departmental Values and Support in Career Choice, JACS Au, 2(6), 1443–1456.
22. Hu L. and Bentler P. M., (1999), Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives, Struct. Equ. Modeling, 6(1), 1–55.
23. Inzlicht M. and Good C., (2006), How environments can threaten academic performance, self-knowledge, and sense of belonging, Stigma and group inequality, Psychology Press, pp. 143–164.
24. Johnson-Ahorlu R. N., (2012), The academic opportunity gap: How racism and stereotypes disrupt the education of African American undergraduates, Race Ethn. Educ., 15(5), 633–652.
25. Jones T., Ahmed R., Cieza E., Pratt J. M. and Popova M., (2024), Experiences of marginalized women pursuing doctoral degrees in chemistry: the critical role of recognition, J. Res. Sci. Teach., 62(2), 576–609.
26. Komperda R., Pentecost T. C. and Barbera J., (2018), Moving beyond alpha: a primer on alternative sources of single-administration reliability evidence for quantitative chemistry education research, J. Chem. Educ., 95(9), 1477–1491.
27. Lane A. K., Hardison C., Simon A. and Andrews T. C., (2018), A model of the factors influencing teaching identity among life sciences doctoral students, J. Res. Sci. Teach., 56(2), 141–162.
28. Locks A. M., Hurtado S., Bowman N. A. and Oseguera L., (2008), Extending notions of campus climate and diversity to students' transition to college, Rev. High. Educ., 31(3), 257–285.
29. Mount P. and Tardanico S., (2014), Beating the impostor syndrome, Center for Creative Leadership.
30. Muthén L. K. and Muthén B., (2017), Mplus user's guide: Statistical analysis with latent variables, user's guide, Muthén & Muthén.
31. Olson S. and Riordan D. G., (2012), Engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Report to the president, Executive Office President, pp. 1–130.
32. Ostertagova E., Ostertag O. and Kováč J., (2014), Methodology and application of the Kruskal-Wallis test, Appl. Mech. Mater., 611, 115–120.
33. Parkman A., (2016), The imposter phenomenon in higher education: incidence and impact, J. High. Educ. Theory Pract., 16(1), 51–60.
34. Peterson C. H., Peterson N. A. and Powell K. G., (2017), Cognitive interviewing for item development: validity evidence based on content and response processes, Measur. Evalu. Counseling Development, 50(4), 217–223.
35. Pfeifer M. A., Zajic C., Isaacs J. M., Erickson O. A. and Dolan E. L., (2024), Beyond performance, competence, and recognition: forging a science researcher identity in the context of research training, Int. J. STEM Educ., 11(1), 19.
36. Puritty C., Strickland L. R., Alia E., Blonder B., Klein E., Kohl M. T. and Tellman B., (2017), Without inclusion, diversity initiatives may not be enough, Science, 357(6356), 1101–1102.
37. Ramos D. M. and Yi V., (2020), Doctoral women of color coping with racism and sexism in the academy, Int. J. Doctoral Stud., 15, 135–158.
38. softwareR Core Team L., (2013), R: A language and environment for statistical computing, R Foundation for Statistical Computing.
39. Reflecting Equity, (2022), https://www.reflectingequity.com/.
40. Renick J. and Reich S. M., (2020), Best friends, bad food, and bullying: How students’ school perceptions relate to sense of school belonging, J. Commun. Psychol., 49(2), 447–467.
41. Rivers E. B., (2017), Women, minorities, and persons with disabilities in science and engineering, Natl. Sci. Found., 1–62.
42. Russo-Tait T., (2022), Color-blind or racially conscious? How college science faculty make sense of racial/ethnic underrepresentation in STEM, J. Res. Sci. Teach., 59(10), 1822–1852.
43. Saleh A. and Bista K., (2017), Examining factors impacting online survey response rates in educational research: perceptions of graduate students, J. MultiDisciplinary Eval., 13(2), 63–74.
44. Schlosser L. Z., Lyons H. Z., Talleyrand R. M., Kim B. S. and Johnson W. B., (2011), Advisor-advisee relationships in graduate training programs, J. Career Development, 38(1), 3–18.
45. Schneider B., Ehrhart M. G. and Macey W. H., (2013), Organizational climate and culture, Annu. Rev. Psychol., 64(1), 361–388.
46. Schniederjans D. G., Schniederjans M. and Levy Y., (2012), Equity theory based strategies for students on overcoming problems in PhD dissertation committees, Int. J. Doctoral Stud., 7, 221.
47. Schober P., Boer C. and Schwarte L. A., (2018), Correlation coefficients: appropriate use and interpretation, Anesth. Analg., 126(5), 1763–1768.
48. Scrucca L., Fop M., Murphy T. B. and Raftery A. E., (2016), mclust 5: clustering, classification and density estimation using Gaussian finite mixture models, R J., 8(1), 289.
49. Shi L. and Stains M., (2021), Development of the Departmental Climate around Taching (DCaT) survey: neither psychological collective climate nor departmental collective climate predicts STEM faculty's instructional practices, Int. J. STEM Educ., 8, 1–20.
50. Solem M., Hopwood N. and Schlemper M. B., (2011), Experiencing graduate school: a comparative analysis of students in geography programs, Prof. Geogr., 63(1), 1–17.
51. Stachl C. N. and Baranger A. M., (2020), Sense of belonging within the graduate community of a research-focused STEM department: quantitative assessment using a visual narrative and item response theory, PLoS One, 15(5), 1–27.
52. Stachl C. N., Brauer D. D., Mizuno H., Gleason J. M., Francis M. B. and Baranger A. M., (2021a), Improving the academic climate of an R1 STEM department: quantified positive shifts in perception, ACS Omega, 6(22), 14410–14419.
53. Stachl C. N., Brauer D. D., Mizuno H., Gleason J. M., Rorrer J. E., Francis M. B. and Baranger A. M., (2021b), Shaping the future of higher education: Practical, community-driven initiatives to improve academic climate, ACS Publications, vol. 7, pp. 910–916.
54. Stachl C. N., Hartman E. C., Wemmer D. E. and Francis M. B., (2019), Grassroots efforts to quantify and improve the academic climate of an R1 STEM department: using evidence-based discussions to foster community, J. Chem. Educ., 96(10), 2149–2157.
55. Stockard J., Rohlfing C. M. and Richmond G. L., (2021), Equity for women and underrepresented minorities in STEM: Graduate experiences and career plans in chemistry, Proc. Natl. Acad. Sci. U. S. A., 118(4), e2020508118.
56. Streiner D. L., (2003), Starting at the beginning: an introduction to coefficient alpha and internal consistency, J. Pers. Assess., 80(1), 99–103.
57. Theofanos M. F., Theofanos M. F., Evans J., Zwolak J. P. and Prettyman S. S., (2021), Survey on Gender, Equity and Inclusion, US Department of Commerce, National Institute of Standards and Technology.
58. Tuhiwai-Smith L., (1999), Decolonizing methodologies: research and indigenous peoples.
59. Qu T. and Harshman J., (2022), Situational interview based investigation of advisor-advisee conflict communication in U.S. chemistry graduate education, J. Chem. Educ., 99(3), 1400–1409.
60. University of Michigan, (2022), Climate Survey: diversity, equity & inclusion.
61. Walton G. M. and Cohen G. L., (2011), A brief social-belonging intervention improves academic and health outcomes of minority students, Science, 331(6023), 1447–1451.
62. Wao H. O., Lee R. S. and Borman K. M., (2010), Climate for retention to graduation: a mixed methods investigation of student perceptions of engineering departments and programs, J. Women Minor. Sci. Eng., 16(4), 293–317.
63. Weatherton M. and Schussler E., (2021), Success for all? A call to re-examine how student success is defined in higher education, CBE—Life Sci. Educ., 20(1), es3.
64. Williams B., Onsman A. and Brown T., (2010), Exploratory factor analysis: a five-step guide for novices, Australasian J. Paramed., 8, 1–13.
65. Yosso T. J., (2005), Whose culture has capital? A critical race theory discussion of community cultural wealth, Race Ethn. Educ., 8(1), 69–91.

---

Footnote

† Electronic supplementary information (ESI) available. See DOI: https://doi.org/10.1039/d4rp00322e

---