Enhancing the accessibility of chemistry assessments for multilingual learners: understanding challenging features in assessment items

Abstract

Multilingual learners face significant challenges when navigating the linguistic complexities of chemistry assessments. This study, employing the Equitable Framework for Classroom Assessment, identified these specific challenging features in general chemistry assessment items on the topics of limiting reactant and percent yield. Through in-depth, semi-structured interviews with multilingual students, we discovered critical barriers to comprehension: lack of metacognitive support, complex vocabulary and syntax, dense text layout, and extraneous information. These findings emphasize the need to better understand and alleviate these types of linguistic features in assessment items to more accurately measure chemistry knowledge, rather than linguistic proficiency. By addressing these challenges, instructors can design more accessible assessment items for a diverse group of students. The results also offer valuable insights and practical guidance for writing equitable assessment items.

---

Introduction

Over the past decade, the linguistic composition of students in the U.S. education system has witnessed a significant shift. According to the National Center for Education Statistics in 2022, one out of every five students in K-12 classrooms now comes from a household where a language other than English is spoken. These students collectively represent more than 350 diverse language groups, contributing rich cultural diversity and global perspectives to their educational environments (National Academies of Sciences, Engineering, and Medicine [NASEM], 2018). This diverse group encompasses recent immigrants, individuals classified as Generation 1.5 (those who arrived in the U.S. as youth or teenagers), and international students. In U.S. colleges and universities, there is a substantial presence of international students, nearing the one million mark, with approximately half of them pursuing undergraduate studies (The Open Doors Report, 2022/23). Within this international student population, a significant contingent of over 382 000 students is enrolled in Science, Technology, Engineering, and Mathematics (STEM) degree programs across the country.

Recently, there has been a growing emphasis on STEM education because of its pivotal role in driving innovation, problem-solving, and global competitiveness (Honey et al., 2020). As such, broadening access to STEM for a diverse range of students, including those who are linguistically diverse, is crucial for ensuring equitable learning opportunities. Given that general chemistry acts as a foundational course for aspiring STEM students, understanding the diverse learning experiences within its context is essential. While efforts have emerged within chemistry education to foster inclusivity and equity, prompted by rising awareness of systematic barriers and discrimination (Wilson-Kennedy et al., 2022), there's a pressing need for deeper exploration of language's role in enhancing accessibility. Multilingual learners often confront language barriers that place them at a disadvantage as they navigate the dual challenges of mastering both English and the discipline-specific language of chemistry, particularly in the context of rigorous, high-stakes assessments (Lee and Fradd, 1998).

Research focusing on how multilingual learners engage with STEM content has seen a recent surge. However, exploration into assessing multilingual learners remains notably limited, primarily situated within K-12 education contexts (NASEM, 2018; Pun et al., 2023). This study is part of a broader project investigating the perspectives of multilingual students about (1) typical assessment items used in their general chemistry courses and (2) modified assessment items designed to reduce linguistic complexity while maintaining academic rigor. Previously, we discussed features of assessment items identified as supportive by multilingual learners (Lee and Orgill, 2021). Here, we report on features of the assessment items that the multilingual learners found particularly challenging.

Literature review

Underrepresentation of multilingual learners in STEM fields

Multiple terms have been used to refer to this group of students over the years, including Limited English Proficient (LEP), English Learners (EL) English Language Learners (ELL), English Speakers of Other Languages (ESOL), emergent bilinguals, and multilingual learners. We adopt an asset-based approach and intentionally use the term multilingual learners to acknowledge the rich linguistic and cultural resources that these students bring to their studies of STEM content (Bergey et al., 2018; NASEM, 2018; Garcia et al., 2021; Buxton and Lee, 2023; Grapin, 2023; Grapin et al., 2023). We recognize that multilingual learner students are learning STEM content while they are developing proficiency in another language and, therefore, are potentially engaging in harder tasks than other students in a classroom. We also recognize that multilingual learners experience educational tasks in a context where instruction and instructional materials are not often presented as equitably as possible for them (NASEM, 2018).

Despite their being a significant—and growing—proportion of learners in the U.S., multilingual learners are underrepresented in STEM fields of study and careers (NASEM, 2018). Increasing multilingual learner students’ access to STEM content, fields of study, and jobs is crucial for multiple reasons. First, STEM knowledge is essential for understanding and participating in the world around us, as well as for addressing the global challenges we face as humans. According to the U.S. National Science Foundation publication STEM Education for the Future: A Visioning Report, “Whether or not they become scientists or engineers, all Americans should have the access, opportunity, encouragement, and tools to participate in the innovation economy, and to succeed amid technological progress and change” (Honey et al. 2020, p. 8). Thus, it is essential that all have access to quality STEM education, including multilingual learners. Second, the requirement for STEM workers has grown significantly, outpacing all other job sectors (NASEM, 2018; Honey et al., 2020). Multilingual learners can be a valuable resource for jobs in STEM-related fields. Third, because of their cultural and linguistic resources, multilingual learners can bring unique insights, approaches, perspectives, and foci to STEM research and innovation that can address the needs of their communities (Honey et al., 2020) and benefit society and research as a whole (NASEM, 2018).

“STEM subjects offer the potential for membership in the communities of mathematicians, scientists, engineers, and other technical experts—communities with their own ways of conceptualizing, representing, evaluating, and engaging with the world. In turn, STEM students from a wide range of backgrounds bring the potential to contribute to shaping STEM fields in critical ways that transform and remake focal topics, practices, and contributions (NASEM, 2018, p. 56).”

Finally, and most importantly, because of their economic value to society, STEM-related jobs generally have a high earning potential. By not increasing multilingual students’ access to STEM-related content, fields of study, and jobs, we reproduce and perpetuate social inequities (Fine and Furtak, 2020a). “Opening avenues to success in STEM for the nation's [multilingual learners] offers a path to improved earning potential, income security, and economic opportunity for these students and their families” (NASEM, 2018, p. 9).

Multilingual learners, language acquisition, and STEM courses

Just as there have been recent shifts in the terminology used to refer to multilingual learners, there have also been shifts in thinking about the interaction between language acquisition and learning in STEM-related courses. For example, it was thought previously that multilingual learners would benefit by learning language first and then learning STEM content. More recent research suggests that multilingual students acquire language, including discipline-specific language, best by engaging in authentic STEM practices (NASEM, 2018; Buxton and Lee, 2023; Grapin et al., 2023). According to a National Academies of Science, Engineering and Medicine report that focused on the multilingual learners in STEM, “language proficiency is not a prerequisite for content learning but an outcome of effective content learning” (NASEM, 2018, p. 10).

Another shift has occurred in the way that researchers view language acquisition itself. Previously, researchers and scholars identified two types of language proficiencies that multilingual learners needed to negotiate in order to succeed in STEM classes: basic interpersonal communication skills and cognitive academic language (Cummins, 2000). Cognitive academic language was seen as much more difficult, requiring seven to ten years for a typical multilingual learner to develop (Collier, 1995). As a consequence of this dichotomous view of language acquisition, instructors might have seen multilingual learners as lacking in STEM content knowledge if they did not present their understanding in the expected form of cognitive academic language (NASEM, 2018; Garcia et al., 2021; Deng and Flynn, 2023; Grapin et al., 2023). Now, however, researchers recognize that there is not a dichotomy of language proficiencies that multilingual learners develop, but, rather, that students develop different language proficiencies in multiple areas (e.g., reading, writing, speaking, listening, etc.) along a continuum of registers (NASEM, 2018; Buxton and Lee, 2023).

In the context of language, registers refers to “the variation in language choices that people make in engaging in a range of activities throughout the day” (NASEM, 2018. p. 12). The language register that one uses depends on the content being communicated, the people with whom one is communicating, and the modality of communication. We all use multiple registers each day. For example, the type of language and skills (the register) one uses to text a friend is very different than the register that the same person would use to write a letter to a politician or to interpret poetry. A typical student in a STEM course will negotiate multiple registers during a single class period as they interact with peers to discuss weekend plans, work in a group to solve a content-related question, listen to an instructor's lecture, and ask a question of a peer instructor.

Challenges for multilingual learners in STEM courses

Academic science and math registers tend to be very formal and to include certain linguistic features that may make comprehension more difficult for multilingual learners (Fang, 2006; Buono and Jang, 2021; Neri and Retelsdorf, 2022), including (1) the use of infrequent nontechnical vocabulary that would not be used in everyday talk (e.g., replacing “calculate” with “compute”), (2) the nominalization of verbs (e.g., changing “measure” to “measurement”), (3) the use of words that have different meanings in everyday and science talk, (4) the frequent use of passive voice, and (5) the use of longer sentence lengths. These linguistic features not only affect multilingual learners’ comprehension in STEM-related classes, but their performance as well (Buono and Jang, 2021).

Importantly, multilingual learners often encounter hurdles on high stakes STEM exams because of these linguistic features, which can negatively impact their ability to showcase their true understanding of the content (Buono and Jang, 2021). In other words, challenging linguistic features can become a systematic barrier when they cognitively interfere with students’ abilities to demonstrate content knowledge in a competitive, fast-paced, and mandatory science course. Even English-as-the-first-language students (EngL1) struggle with understanding words in the chemical context of technical terminology and discourse structures (Rees et al., 2019). For example, words such as volatile and state are well understood in everyday life, but poorly understood in the context of chemistry (Cassels and Johnstone, 1983). However, to decode these types of terms in the correct context, multilingual students must also navigate multiple language domains effectively and quickly. Experiencing these types of challenges can create substantial obstacles, especially during “high stakes” course assessments, which can have long-term consequences on grades and academic persistence.

Assessment of multilingual learners in STEM

Research on the assessment of multilingual learners in STEM subjects is limited and mostly contextualized in primary and secondary education (NASEM, 2018; Buxton and Lee, 2023; Pun et al., 2023); however, all of the research reports agree that, for multilingual learners, assessments do not always measure content knowledge. Instead, they often confound language competence with STEM competence (NASEM, 2018; Cardozo-Gaibisso et al., 2020; Buono and Jang, 2021; Grapin, 2023).

A large body of research has identified language as a source of “construct irrelevant variance—variation in test scores due to factors that do not have to do with the knowledge or skill being assessed. Many of these factors have to do with linguistic complexity; for example, complexity due to the use of unfamiliar and morphologically complex words, words with multiple meanings, idiomatic usages, and long or syntactically complex sentences in texts and accompanying test items and directions” (NASEM, 2018, pp. 213–214).

It is essential that STEM assessments be presented and carried out in as equitable a manner as possible because high-stakes assessments often determine who is allowed to learn, do, and be STEM (Fine and Furtak, 2020a). High-stakes STEM assessments carry not only educational consequences, but social consequences, such as determining who has access to STEM-related employment, the disproportionate economic value of STEM-related jobs, and the material goods and positive reputation that can result from employment in a STEM-related field (NASEM, 2018; Fine and Furtak, 2020a; Honey et al., 2020).

Efforts to make assessment of multilingual learners in STEM more equitable

So, what can or has been done to make assessment of multilingual learners more equitable? As previously mentioned, only a limited amount of research has been done about the assessment of multilingual learners in STEM classrooms, and most of this has been carried out in a K-12 context. Here, we provide a brief description of some of the efforts to provide equitable assessment for multilingual learners in STEM. In general, they fall into three categories: (1) providing resources to multilingual learners during assessments, (2) altering the type of assessment that multilingual learners engage in, and (3) modifying assessment items themselves.

Initial attempts at making assessments more equitable for multilingual learners included giving students access to certain accommodations during an assessment. For example, multilingual learners might have been provided with extra time to complete the assessment or a dictionary to translate words written in English into terms in the learners’ native language. Overall, these accommodations have not been shown to be specifically helpful to multilingual learners (NASEM, 2018; Cardozo-Gaibisso et al., 2020). In fact, some accommodations, such as extra test time, benefitted both multilingual and monolingual learners and, thus, did not reduce the performance gap between these learner groups.

More recent efforts at making assessments equitable suggest that multilingual learners’ STEM content knowledge is better demonstrated through alternate assessments that are multimodal, open-ended and dynamic, as these allow multilingual learners to demonstrate their understandings in a variety of ways (Lopez et al., 2017; NASEM, 2018; Buxton et al., 2019; Fine and Furtak, 2020a; MacDonald et al., 2020; Buxton and Lee, 2023; Grapin, 2023; Grapin et al., 2023). Other scholars recommend that multilingual learners be allowed to use trans-languaging practices during assessments (Lopez et al., 2017; MacDonald et al., 2020; Pierson and Grapin, 2021; Pun and Tai, 2021; Grapin et al., 2023). These scholars assert that multilingual learners do not work or think in one language and then switch to working and thinking in a different language. Instead, they have a set of integrated skills, knowledge, abilities, and resources that result from the synthesis of their multiple linguistic and cultural backgrounds. Trans-Languaging involves the multilingual learners’ use of any of their linguistic resources when communicating (Garcia et al., 2021). Assessments that encourage the use of trans-languaging by multilingual students might, for example, provide assessment items in multiple languages or include an explicit statement that learners are allowed to use drawings, words in different languages, etc., in their assessment responses. Trans-Languaging is a potentially transformative equity practice (Grapin et al., 2023), in that it changes what counts as evidence of STEM learning (i.e., it does not require that learners demonstrate their STEM content knowledge through written answers that use perfected discipline-specific academic language).

Other efforts at making STEM assessments more equitable for multilingual learners have focused on modifying assessment items to reduce their linguistic complexity (see, for example, Siegel, 2007; Siegel et al., 2008; Siegel et al., 2014; NASEM, 2018; Fine and Furtak, 2020a, b; Guzman-Orth et al., 2021). For example, the National Academies of Science, Engineering, and Mathematics (2018) and Guzman-Orth et al. (2021) recommend that instructors follow the universal design for assessment principles (Thompson et al., 2002). Among these are two elements that specifically address linguistic complexity: (1) ensuring that “the language in the items is accessible and promotes maximum readability and comprehensibility (i.e., minimize construct-irrelevant variance for nonreading items)” (Guzman-Orth et al., 2021, p. 7) and (2) ensuring that “presentation details such as font, style, spacing, contrast, and white space do not become sources of construct irrelevant variance” (Guzman-Orth et al., 2021, p. 7). As another example, Fine and Furtak (2020a, 2020b) developed the Science Assessment for Emergent Bilingual Learners (SAEBL) checklist, which includes five categories of features to consider when designing assessments for multilingual learners. One of these is the “integration of scaffolds.” Within this category, Fine and Furtak recommend using sentence starters, sentence frames that connect ideas, graphic organizers, visuals, shorter sentences, bulleted items, active voice, and definitions of key terms within sentences, among others when constructing assessment items to meet the needs of multilingual learners.

It is not surprising that reducing the linguistic complexity of assessment items and integrating scaffolds is included on each of these checklists for making assessments more equitable for multilingual learners. Linguistic complexity has been shown to contribute significantly to the performance gap between multilingual learners and monolingual students on assessments in STEM classes (Fang, 2006; NASEM, 2018; Buono and Jang, 2021; Neri and Retelsdorf, 2022). Interestingly, according to the NASEM report, “There is evidence that even changing one word or slightly rephrasing an expression in science items may make a difference on whether an item is or is not biased against linguistic minority students” (NASEM, 2018, p. 215).

Current chemistry assessment practices in higher education

A large-scale study that looked at 28 576 science faculty members’ grading practices revealed that chemistry faculty predominantly use multiple-choice items to assess students’ performance (Goubeaud, 2010). It is likely that faculty-perceived constraints, such as limited time, limited resources, and large undergraduate class sizes (Goubeaud, 2010), limit the use of alternative modes of assessment—such as multimodal, open, or dynamic exams—which have been shown to benefit multilingual learners. However, the strategies of reducing the linguistic complexity of and integrating linguistic scaffolds into assessment items hold promise for making chemistry assessments more equitable, while maintaining academic rigor and while working within the constraints identified by faculty members in the Goubeaud (2010) study.

While the need for more research on assessing multilingual learners has been established (Bergey et al., 2018; NASEM, 2018), particularly in the context of higher education, there's a crucial gap in understanding the specific challenges they encounter during these assessments. Current research often focuses on broader assessment practices for multilingual learners (Buxton et al., 2014). However, to effectively modify assessments for greater equity, we need a deeper understanding of the specific linguistic and cognitive challenges multilingual learners face on exams.

Conceptual framework

In this study, we adopted the Equitable Framework for Classroom Assessment (EFCA) (Siegel et al., 2008) as the conceptual framework. The EFCA has previously been applied to revise life sciences assessment items, demonstrating its effectiveness in narrowing the performance gap between multilingual learners and English monolingual (EngL1) middle school students (Siegel et al., 2014). However, a crucial question remains: Can the EFCA modifications be successfully applied to undergraduate chemistry assessments? While the framework has demonstrated promise, its effectiveness in this specific context, with its unique challenges and higher-level content, requires further investigation.

As discussed elsewhere (Lee and Orgill, 2021), the EFCA's primary goal is to modify assessments linguistically to more accurately measure multilingual learners’ content knowledge without compromising the inherent difficulty of the items. We operationalized the EFCA modifications outlined by Siegel et al. (2008) as described in Lee and Orgill (2021) (for a detailed explanation, see Gandhi-Lee, 2018).

However, current research does not establish which specific EFCA modifications are most useful for undergraduate chemistry assessments. To bridge this gap, we need to go beyond the existing framework and incorporate student voices directly. By asking multilingual learners themselves to identify the features they find challenging in typical assessments and those modified using the EFCA, we can gain crucial insights into which modifications are most effective in the particular context of tertiary chemistry courses.

Current study

The current study is part of a larger project that examines multilingual learners’ perceptions of typical general chemistry assessment items, as well as items that have been modified to reduce their linguistic complexity (Gandhi-Lee, 2018). In the current project, we asked multilingual learners to identify features of the original and modified assessment items that they found supportive, as well as features they found challenging. We have reported on the supportive features previously (Lee and Orgill, 2021). Here, we focus on the features that the multilingual learners found challenging in chemistry assessment items because knowledge of these challenges will help researchers and instructors determine which modifications will be most effective for generating chemistry assessments that are more equitable. Specifically, results related to the following research question will be discussed:

• What features do multilingual students think make it difficult for them to comprehend general chemistry assessment items?

Methods

Participants

The participants in this study met specific inclusion criteria: (1) they were born outside the United States or in countries where English is not the primary language, (2) their first language was not English, (3) they possessed basic reading, writing, and conversational English skills, and (4) they had resided in the United States for a maximum of eight years.

In total, we recruited 10 multilingual learner participants, all of whom were enrolled in a first-semester general chemistry course at a large university in the U.S. Southwest. The multilingual learner participants represented diverse backgrounds, including two Filipino students, two Thai students, two Hispanic students (from Colombia and Venezuela), one Asian–Indian student, one Chinese student, one Russian student, and one student from Guam. All participants identified as a Generation 1.5 student; however, systematic background information about their prior formal and/or informal exposure to the English language was not collected. To safeguard the anonymity of participants, we assigned pseudonyms to each participant.

Ethical considerations

This study adhered strictly to ethical guidelines for human subjects research, with participants receiving comprehensive information about the study's purpose, procedures, risks, and benefits. Participation was voluntary, with the option to withdraw at any time without consequence. Written informed consent was obtained, ensuring confidentiality and data anonymization. The Institutional Review Board at the University of Nevada, Las Vegas approved the study, confirming compliance with ethical standards. The research team upheld the highest standards of integrity and respect for participants' dignity and rights throughout the study.

Item selection

In this study, the assessment items were centered around the fundamental concepts of limiting reactant and percent yield, which are foundational topics in general chemistry. These topics delve into the quantitative relationships between compounds involved in chemical reactions. We specifically chose these topics because the word problems associated with them typically involve technical terminology and necessitate mathematical calculations, aspects that have proven to be challenging for students (Carter and Brickhouse, 1989).

The assessment items we selected for this study encompassed a combination of mathematical and chemical concepts, as well as phrases that demanded thoughtful reflection. In other words, these problems could not be solved by mere application of formulas. All the assessment items used in this study were sourced from general chemistry textbooks and the test banks of general chemistry instructors. In the process of item selection, we sought the expertise of experienced chemistry instructors and researchers in chemical education. Four items were selected and then modified according to the EFCA guidelines (Lee and Orgill, 2021), resulting in four original items and four revised items for use in the study. These items will be presented, along with their accompanying discussions, in the Results section.

Data collection

In this qualitative study, we employed a one-on-one semi-structured interview approach as our primary method of data collection. These interviews took place in person and were scheduled following the students' completion of the course exam on the topics of limiting reactants and percent yield.

The interview protocol comprised four sections. The first section involved obtaining informed consent and collecting demographic information through a series of questions. In the second section, participants were asked rapport-building questions to establish a comfortable and open dialogue. The third section presented both the original and modified versions of each assessment item in a randomized order. Participants were instructed to read, work through the problems on paper, and share their thoughts about the challenging aspects of the items that impeded them in the problem-solving process. Participants were also encouraged to jot down notes directly on the assessment items during the interview. The final part of the interview served to address any clarifying questions from participants and to implement member checking as a means to validate the findings (Creswell, 2007) and enhance the study's credibility.

Data analysis

We used thematic analysis (Braun and Clarke, 2006) to understand how participating students perceived and interpreted the assessment items presented to them. Our primary interest lay in identifying the specific types of features those multilingual learners found challenging within these assessment items.

Following the five phases of thematic analysis (Braun and Clarke, 2006), we transcribed (Phase 1) the audio-recorded interview data verbatim and embarked on an iterative coding process based on the transcript content. During this analysis, we closely examined the interview transcripts, searching for recurring themes related to students' perceptions of the challenging features of the assessment items. These features encompassed various elements such as visual representations and wording.

To generate initial codes (Phase 2), we coded the transcripts based on the challenging features that participants identified across each assessment item. These individual codes were then organized into different types, such as sentence structure and contextual cues, grouping similar categories together. Subsequently, we developed overarching themes (Phase 3) that encapsulated these categories, describing each group of challenging features. Additionally, the notes that participants made while working through each item were cross-referenced to validate the emergence of new categories and maintain the authenticity of the evolving themes. To ensure the reliability of our analysis, multiple coders were involved in the data analysis process (Phase 4). In instances where discrepancies arose in our interpretations, we engaged in discussions and reached a consensus on any necessary modifications to the category descriptions.

Based on our analysis, the following four themes emerged (Phase 5) that describe the challenging features of the assessment items as perceived by students:

1. Lack of metacognitive support

2. Use of complex vocabulary and syntax

3. Use of dense text layout

4. Inclusion of irrelevant information

Research positionality

As researchers and authors of this study, we recognize the importance of acknowledging our own perspectives and how they could have influenced our approach to this work. Understanding the potential impact of our individual positions on interpreting data and conducting research is vital.

Our team comprises two researchers with diverse professional and personal backgrounds. The first author, serving as an assistant professor and a general chemistry instructor, brings a unique perspective as a first-generation student and woman of color, who identifies as cisgender. Her background as a first-generation student, along with her proficiency in non-English languages, enables a level of connection with the study's participants. The second author is a professor and a general chemistry instructor who identifies as a cisgender woman. She is conversationally proficient in Spanish and spent a year and a half working with Spanish-speaking people. Her struggles to communicate during that time period give her some insight into the challenges her multilingual students face every day. With doctoral degrees (PhDs), both authors acknowledge the disparities within the academic system that may hinder equal opportunities for all individuals.

We emphasize that our distinct backgrounds, experiences, and expertise inevitably influence our interpretations of the data. Our commitment lies in maintaining transparency and reflexivity throughout our endeavors to ensure an impartial and meaningful exploration of the participants' experiences.

Results

In this study, multilingual learners were presented with both the original and EFCA-modified versions of four assessment items, with the order of presentation randomized. Their task was to identify and explain the features they found challenging in both versions.

An inductive analysis of interview transcripts revealed the following four primary challenges, ranked by frequency of mention:

1. Lack of metacognitive support

2. Use of complex vocabulary and syntax

3. Use of dense text layout

4. Inclusion of irrelevant information

Notably, some of these challenges identified by multilingual learners directly align with the modifications advocated by the EFCA. For instance, the lack of metacognitive support corresponds to the EFCA's emphasis on providing clear task instructions and scaffolding strategies. Similarly, the difficulties with complex vocabulary and dense text layouts mirror the EFCA's recommendations for using simple language and structuring the text in a user-friendly manner. This alignment between the identified themes and the EFCA modifications suggests that adopting the framework has the potential to enhance the assessment experience for multilingual learners. In the discussion that follows, items are referred to by a number followed by either the letter “O” or the letter “R” to represent the original and revised versions of the items, respectively.

Lack of metacognitive support

Multilingual learners most frequently reported a lack of metacognitive support in assessment items as a major challenge, which refers to the absence of wording or problem-solving guidance that helps students connect individual pieces of information into a coherent whole. In this case, students perceived the information within the items as a series of isolated ideas. This fragmented understanding hindered them from identifying key information, making correct inferences, and building conceptual connections. This absence encompassed two key elements.

1. Disconnected wording: Assessment items lacked internal wording that typically guides readers in understanding the overall flow of information and the key concepts being tested. These internal words function as signals that help readers navigate the text and grasp connections between ideas and include logical connectives (“because,” “however”), sequence words (“first,” “next”), emphasis words (“indeed,” “significantly”), and summarizing words (“overall,” “in conclusion”).

2. Insufficient problem-solving guidance: This element emphasizes the absence of specific and explicit support structures that guide students towards successful problem-solving. Support structures can include breaking down the question into smaller steps, providing prompts to guide students towards the correct approach, or offering sentence starters to initiate equation formulation. The absence of such guidance in assessment items left multilingual learners without crucial support, making it difficult for them to grasp the connections between different parts of the item and identify the core objective of the question.

Disconnected wording. The lack of clear connections between ideas in the items ultimately created a significant barrier for multilingual learners. Students reported having to read items multiple times in an attempt to glean meaning and formulate a plan for solving the problem. This difficulty was evident in items like 1-O (Fig. 1). The absence of internal signals left students uncertain about how the background information about methanol was related to the chemical reaction and calculations required. This lack of clear connections made it difficult to determine how to begin using the given information to set up the problem. Some students, like Lupe, expressed uncertainty:

Fig. 1 The original version of the assessment item 1 (Zumdahl and Zumdahl, 2012, p. 113).

Lupe stated, “Well, for me, you won't really know at first where to start. They just give you a bunch of numbers. It takes a while to figure out these numbers from this number and from this molecule, and then this one's from this. It's just hard to organize it at first. You actually have to work on it first before you really work on it or else you'll forget the details.”

Item 2-O (Fig. 2) presented a similar challenge. Although the initial sentences described the experimental set up of the containers with chlorine gas, the lack of linguistic signals made it unclear how that information was connected to Parts A through E. Students like Carlos reported spending significant time re-reading the question because of the difficulty in applying the information provided. Carlos noted, “…it took a couple of times to read because they asked for a lot of things so I just took a couple of times to read through it.”

Fig. 2 The original version of the assessment item 2 (Zumdahl and Zumdahl, 2012, p. 241).

Insufficient problem-solving guidance. The absence of support structures that guide students through problem-solving steps significantly impacted multilingual learners. When items included implicit steps without clear connections, students struggled to make the necessary connections and approach the problems effectively. Item 2-O (Fig. 2) exemplifies this challenge. The graph, presented as the central figure, drew students’ attention. However, the intentional omission of the y-axis values left them unsure whether it was a mistake or a prompt to calculate those values themselves. This ambiguity led them to seek clarification during the interviews. Providing explicit directions within the item, such as “Determine the missing y-axis values,” could have alleviated this confusion.

Naima stated that, “I think [we] would have a tough time trying to think about what the y-axis is, the numbers and values for the y-axis, and that would just throw us off [from] figuring out B, C, D parts, and eventually towards E.”

Students also highlighted the importance of clear connections within multi-part items. Item 2-O, with parts A through E, lacked clear transitions between these sections. This made it difficult for students to understand how each part built upon the previous one and ultimately solve the problem.

Rohan mentioned that “[item 2 original] is the hardest because it looks simple but the wording makes you think a lot, and people would get confused especially since you don't have any guidelines…confusing steps. There's no y-axis value and stuff.”

Use of complex vocabulary and syntax

Multilingual students found complex vocabulary and syntax present in assessment items to be challenging to understand. Based on literature, terms like “stoichiometry,” “molarity,” and “spectroscopy,” which are chemistry-specific words, can pose an additional layer of complexity for these students (Lee and Buxton, 2013). However, our findings reveal that students also encounter challenges with non-chemistry-specific elevated, or vague vocabulary/wording found in assessment items.

Elevated or unfamiliar vocabulary. Students found elevated and unfamiliar words within items to be challenging. In item 1-O (Fig. 1), for instance, terms such as “potential” and “manufactured” were considered overly complicated, adding to the cognitive effort required for interpretation. Rohan remarked on the taxing nature of such terminology during exams or quizzes, particularly when the information associated with these terms is extraneous to problem-solving.

Rohan stated, “I noticed that a lot of the word choices here are elevated and then in general seems like it has a longer sentence structure. ‘Potential’ and ‘manufactured’ are words that are sort of especially because some of this information is not super needed especially when it's talking about being simplest alcohol and like potential replacement for gasoline.”

Ambiguous or vague wording. In addition to complex vocabulary, vague wording also derailed students’ comprehension and problem-solving processes. Item 2-O (Fig. 2) stood out as particularly challenging, with unclear wording being a primary reason. Terms such as “and so on” and “explain” presented difficulties for students, necessitating not only translation and interpretation but also conjecture regarding the intended meaning. The word “explain” in part A was not precise enough to relay the type of information expected here. For example, an alternative could be “Explain the shape of the graph in terms of the amount of NaCl produced over time,” which may have better helped students understand which information to discuss.

In another case of ambiguous wording, item 4-O (Fig. 3) employed the term “roasting” to the process of making sodium chromate. This choice of language led to confusion among students, who were unfamiliar with the term and/or questioned its relevance to the item's purpose.

Fig. 3 The original version of item 4 (Instructor Test Bank).

Hector stated, “I don't understand how [roasting] would help me with the question. It's just a weird word…because when you think of roasting you think of like heat. Then, you had to think about if heat is lost or gained in the equation. So, it might confuse people, it's just not a great word to put in there.”

The inclusion of “roasting” in item 4-O introduced an unnecessary detail that did not contribute to the overall understanding of the problem. Instead, it served to distract and confuse students, detracting from their comprehension of the content.

Complex syntax. Chemistry text often contains long, complex sentences with multiple clauses with passive voice, which can be difficult for multilingual students to parse (Wellington and Osborne, 2001). Our findings indicate that these types of sentences present significant difficulties for multilingual students, particularly when laden with crucial information in assessment items.

Complex sentence structure. Students found the sentence structures to be complex and challenging to follow in the assessment items. Complex sentences combine independent clauses (full ideas) with dependent clauses (incomplete ideas) using subordinating conjunctions (because, although). This structure allows writers to express more intricate relationships between ideas in a single sentence; however, it can especially be difficult to interpret for students who are still in the process of developing their English language proficiency.

For example, item 1-O (Fig. 1) contained sentence structures that began with “Suppose…” and “If…” Conditional structures may not exist or be as commonly used in the multilingual student's native language(s). Navigating abstract concepts within hypothetical scenarios compounded the students’ challenges in this case.

Anastasia expressed her uneasiness when encountering these types of sentences during exams or quizzes, “Because people are really unsure and scared going through this and you hear, ‘Suppose,’ and couldn’t it just be stated like, ‘Yes, this is how much was put in.’ […] I’m already nervous and scared and your ‘suppose’ makes me nervous and scared for the rest of it. That's just my own personal feeling, [especially] if my test anxiety is already shooting.”

These types of sentences found in items 4-O (Fig. 3), 1-O (Fig. 1) and 4-R (Fig. 4) often utilized lengthy, compound sentences featuring coordinating conjunctions (“and,” “with”) to connect multiple clauses and phrases. These complex structures, compounded by the use of unfamiliar terminology (“source of chromium,” “roasting chromium”) and chemistry formulas of sodium carbonate and sodium chromate, can present significant challenges for multilingual learners. Simplifying the language and using shorter sentences with smaller chunks of information can make these assessment items much more accessible.

Fig. 4 The EFCA-revised version of item 4.

Sentences with multiple numerical values. The items’ data-packed sentences, often accompanied by complex sentence structure, overloaded multilingual learners and challenged their comprehension. This was evident in item 4-O (Fig. 3), where the first sentence included a lot of information about chromite. Students discussed that understanding this information took many re-reads. Additionally, in item 1-O (Fig. 1), students expressed feeling overwhelmed and perplexed by the copious information packed into the question, exemplified by phrases like “Suppose 68.5 kg CO (g) is reacted with 8.60 kg H₂ (g)…” This issue persisted in item 4-O (Fig. 3), where the final section stated, “1.2 kg of Na₂CrO₄ is produced from ore that contains 1.0 kg of FeCr₂O₄.”

This issue was also seen in item 4-R (Fig. 4) where students were confounded with processing the last statement containing dense information. The question part starting with “What is the percent yield if 1.2 kg of… 1.0 kg…?” was particularly challenging to follow for some students. Naima discussed how this type of sentence can lead to confusion,

“You put two values in the same question, but they are not supposed to be used in the same step, that gets confusing. Even if it was just 1.2 kilogram of sodium chromate was produced, period. What is the percent yield from that or contains – just separating those [values] even in the slightest bit. Because when you put in the same question, an average student wouldn’t probably know [how to start].”

Students articulated that grappling with multiple numerical values within a single convoluted sentence induced stress and confusion. Ina discussed, “I know most chemistry equations are written like this. It's just when there are two numbers back-to-back like this, I don’t know which one to use first on the conversion factor thing. Yes, it's just a bit confusing.”

Use of dense text layout

Multilingual students’ initial engagement with the assessment items can be negatively impacted by the overall visual and spatial presentation of the item. Our findings suggest that as soon as students look at the items (even before reading it), they can draw some conclusions about it based on the item's length and appearance. If the item appeared as lengthy or in big blocks of text, the students often perceived it to be “hard.”

This was most evident in how students responded to item 2-R (Fig. 5). Item 2-R (Fig. 5) was a modified version of item 2-O (Fig. 2), developed using the EFCA framework. These modifications aimed to improve the original item (Fig. 2) by adding visual representation (containers with Cl₂ and a list of the masses in Na in each container) and targeted contextual support for parts A through E. However, these modifications resulted in a noticeable increase in item length compared to the original version.

Fig. 5 The EFCA-revised version of item 2.

Students perceived the item 2-R (Fig. 5) to be too lengthy and reacted negatively to its appearance. They expressed concerns about the increased reading and translation time required during exams or quizzes. Interestingly, despite their initial reservations, students found item 2-R (Fig. 5) to be easier to follow and solve. Elara exemplified this sentiment, stating, “Okay, this one looks more overwhelming but [it's] easier to solve than the other one (2-O).”

Visual presentation also impacted students’ perceptions of difficulty. Item 1-O (Fig. 1) lacked spacing or separation between questions, appearing as a single dense block of text. Students, like Carlos, found this challenging: “Because with this one it looks like a paragraph of information. I feel that's how some of my chem questions are like, and this is when it starts to get hard for me because when I have so much information to sift through, I don't know what I'm looking for.”

These findings suggest that clear spatial organization can make assessments more accessible for multilingual students. On the other hand, while the length of an item can be a deterrent for multilingual students, they ultimately appreciate well-structured items with clear instructions that aid comprehension and problem solving.

Inclusion of irrelevant information

Another key challenge the students identified was the presence of extraneous information in items, which can act as a barrier to comprehension by overloading students with unnecessary details that they must then decipher and filter out in order to identify the core task being assessed. This type of information can be especially time-consuming and cognitively demanding for multilingual students, who may already be struggling to process complex language and unfamiliar concepts. As such, our findings suggest that irrelevant information can discourage students and hinder their ability to demonstrate their true understanding of the concept at hand.

Extraneous information. Although intended to provide context, students perceived background information to be irrelevant and misleading, obstructing their comprehension and hindering their ability to solve the problem. For example, students found the inclusion of background information about methanol to be particularly irrelevant and distracting in both original item 1-O (Fig. 1) and revised item 1-R (Fig. 6). The first two sentences of item 1-O (Fig. 1) describe methanol and its use in race cars. Students struggled with understanding how this information was related to solving the main questions, which was to find the theoretical and percent yield of methanol. They searched for a connection between the background information and the main questions, which often led to uncertainty and confusion.

Fig. 6 The EFCA-revised version of item 1.

Seojun mentioned, “So I don't know if it's really needed to include this mess [pointing at the first two lines] that it is also called a methyl alcohol just because throughout the rest of the problem it's referred to as methanol. So it really doesn't matter what else you call it. Then I guess the same thing with this one with simplest alcohol it's not very relevant.”

Even though item 1-O was revised to include simpler terminology (item 1-R, see Fig. 6), students still carried the same sentiment toward the background information. The overwhelming consensus was that this information was unnecessary and should be eliminated.

Sometimes, students were unable to determine if background information was simply provided for context or if it was needed to solve a problem. For example, in item 3-O (Fig. 7), information about “Haber process” and “high pressure” was perceived as relevant but led students astray. Ina expressed uncertainty about whether the Haber process information guided her in setting up the problem.

Fig. 7 The original version of item 3 (Brown et al., 2015, p. 238).

Ina said, “I found really interesting where it describes the Haber process. I've never heard of this before but if people know or if the students know what this process is then would [they] know how to set up this problem?”

The inclusion of this additional information in the item derailed students’ thinking as “high pressure” made some consider using the ideal gas law equation because in PV = nRT, P is related to pressure and CO and H₂ were said to be in gaseous state. Lupe discussed, “There's not really any areas that is helpful but the ‘high pressure’ in the first part, it throws you off […] For this one, you would think of whether you have to use the ideal gas law?”

Item 3-R (Fig. 8) was developed to improve readability of the original version 3-O (Fig. 7). While students found the first three sentences about making ammonia and the Haber process easier to read and comprehend, they still questioned the relevance of this information. Sheela expressed, “The Haber Process, I don't even know what is that…I didn't really need to know that.”

Fig. 8 The EFCA-revised version of item 3.

Similarly, in item 4-O (Fig. 3), students were inundated with an abundance of information, leading to prolonged reading and translating times. Despite initially seeming relevant (the background information mentioned chromite and the formation of sodium carbonate), students ultimately perceived the first sentence to be unnecessary for problem-solving. Many students invested time in deciphering and annotating this information in both their first language (L1) and English, only to later express frustration upon realizing its lack of utility. Anastasia discussed that this information was too much to process and questioned the use of it. “I think it's very wordy or dense, especially in the beginning. I read through and it said, ‘Use a source of chromium in chromium compounds,’ they really just put chromium and chromium back together, but how do I use this?” Even when the same information was presented using a shorter and simpler first sentence, as shown in Item 4-R (Fig. 4), students like Anastasia still felt this information was extraneous and should be eliminated.

Story-based contextualization. Interestingly, the inclusion of a storyline as contextual information within item 3-R (Fig. 8) was perceived to be counterproductive. While this approach had shown to be effective in enhancing comprehension of life science items for middle schoolers (Siegel et al., 2008), it was not well-received by the multilingual students in this study. They found the storyline, though interesting, time-consuming to read and irrelevant to solving the problem. As Carlos noted, “Because I'm taking the test and I'm just trying to know what the answer is. I do not really visualize myself working in a factory, but it's okay.” This finding highlights the importance of tailoring contextualization strategies to the specific student population and assessment objectives.

In alignment with literature, this finding also suggests that although rich contextual information can enhance student engagement during assessment, it can also hinder performance if overly complex or irrelevant. The Handbook of Accessible Achievement Tests for All Students (Abedi, 2011, p. 219) emphasizes the need to carefully balance contextual details with the core assessment needs for multilingual learners “…test items presented in rich language context would be more interesting to test takers. However, sometimes the language use for context in the assessment questions may be excessively complex and may cause misunderstanding and confusion. Therefore, it is of paramount importance to distinguish between the language that is a natural part of the assessment and essential to the assessment process and the language that is unrelated to the assessment process.”

Conclusions

In this study, we investigated the challenges perceived by multilingual students in comprehending and setting up general chemistry problems. Four key features emerged as challenging across four original and four EFCA-modified assessment items.

The most prevalent challenge was the lack of metacognitive support within items, which posed a massive entry barrier for multilingual students to linguistically access the item. Our previous study showed that multilingual students looked for extralinguistic features within an item to aid their comprehension (Lee and Orgill, 2021); these extralinguistic features are based in contextual cues and scaffolding. These results align with Jerome Bruner's theory in cognitive psychology, which emphasizes the importance of scaffolding for students in facilitating language acquisition, particularly when it coincides with learning new subject-specific language (Bruner, 1983). Scaffolding, as Bruner defines it (1983, p. 60) is a “a process of ‘setting up’ the situation to make the child's entry easy and successful…” Scaffolding, therefore, emerges as a powerful pedagogical tool for enriching the linguistic and extralinguistic contexts on assessment items. The absence of clear connections, emphasis markers, and step-by-step guidance forces multilingual learners to spend a disproportionate amount of time deciphering the meaning of the questions rather than focusing on the identifying the key information to solve the actual concepts being tested.

The findings also underscored that the use of complex vocabulary and syntax is perceived to be cumbersome and confusing. Elevated vocabulary words, not necessarily discipline-specific, may impede comprehension of the assessment items. Consequently, employing simpler alternatives (e.g., ‘made’ instead of ‘manufactured’) to convey the same meaning is advisable if the information is pertinent to setting up and solving the problem. It was also particularly challenging for students to read ambiguous wording in the main part of the question, leaving them more uncertain about how to respond. As instructors, we value our students’ ability to think critically and elicit the next steps to solve a problem; however, this finding shows that for multilingual learners, including ambiguous terms such as “explain” or “describe” seems to raise more confusion about what is being asked. Similarly, students were confused by terms that might have different meanings in everyday registers and chemistry registers. The use of the word “roasting” elicited interesting reactions from students as it made them think of heat or cooking; however, students did not know how to apply this term to the chemistry context. Overall, these findings align with research suggesting that complex language features can impede content comprehension for multilingual learners (Gibbons, 2006). Therefore, multilingual learners could benefit from clear and concise language, allowing them to focus on the core scientific concepts rather than deciphering complex sentence structures or unfamiliar/uncommon vocabulary.

The study also revealed a noteworthy finding among multilingual learners about item presentation: they often equated item length with difficulty. Students initially judged longer items (items 1-R and 2-R) as presenting greater difficulty in both comprehension and problem-solving. This was evident despite the improved visual and spatial organization within the modified item 1-R (Fig. 6). Similarly, the seemingly shorter original version of item 2 initially appeared easier than its modified counterpart, item 2-R (Fig. 5). However, upon delving into the content, students recognized the enhanced clarity and support offered by the revised versions. This finding suggests that for multilingual learners, the initial assessment of difficulty may not always reflect the actual cognitive load required of an item. Therefore, designing items that prioritize clear presentation with visual and spatial arrangement, even if it results in a slightly longer item, can ultimately benefit student comprehension.

Students’ perceptions about the inclusion of irrelevant information were clear if the information does not directly help with problem solving, it should be eliminated from the item. Item 2-R (Fig. 5) attempted contextualization through a narrative involving a student working in a factory—a strategy inspired by an earlier study using the EFCA in life science items among middle school students (Siegel, 2007). However, the students in this study found this approach unconventional and unnecessary for a timed exam solely aimed at finding answers. Some students expressed that they might have reacted differently if such contextualization had been introduced in homework items or if this style of questions were shown in their courses. Evidently, students hold different expectations regarding item formats encountered in homework assignments versus those in exams. This finding suggests that multilingual students prioritize identifying key information that leads to solving the problem. Any additional contextual details, intended to be helpful, can become a burden, especially in time-sensitive assessments.

Effective assessment design for multilingual learners in chemistry requires careful consideration of information inclusion. Extraneous details can impede comprehension, while contextualization strategies must be age-appropriate, relevant to the subject-matter, and aligned with student expectations for the assessment format.

By examining the challenges faced by multilingual students in specific assessment items, this research has identified key features that significantly impact their understanding of what is being asked. The findings underscore the importance of crafting assessments that transcend language barriers and highlight the value of accessible, well-structured assessments in fostering equitable demonstration of content knowledge. The insights gleaned here offer a valuable vantage point for educators to create assessment items that better reflect multilingual students’ knowledge and abilities while mitigating language-related barriers.

Limitations

While this study aimed to provide valuable insights into the challenging features of assessments faced by multilingual learners, it is important to acknowledge certain limitations inherent to our research design. This study concentrated on the topics of limiting reactant and percent yield in general chemistry. The specific focus on these topics might limit the applicability of our findings to other areas of chemistry. Additionally, while we modified the assessment items based on the EFCA (and vetted all items with input from chemistry instructors and researchers), the subjective nature of the modification process introduces a level of interpretation. The modifications may not perfectly align with the diverse linguistic and cultural backgrounds of all participants, and variations in interpretation could impact the study outcomes. Lastly, our participant pool consisted of 10 multilingual learners. While this sample size is appropriate for gaining a deeper understanding, it does not provide generalizable findings for a larger population. We also note that although all participants in this study were categorized as Generation 1.5 students, they exhibited varying levels of English language proficiency. These differences in language skills were not systematically measured for the purposes of this study.

Implications

This study provided interesting insights about the types of features that multilingual learners find challenging on assessment items. Instructors can use these insights in order to modify assessment items so that they are more inclusive for a diverse student body. A common challenge encountered when applying the EFCA to chemistry items is the management of conflicting linguistic modifications. For example, item 2 (Fig. 2 and 5) initially presented with concise steps but lacked explicit guidance. To enhance clarity, metacognitive support was introduced, necessitating longer, more detailed sentence structures. This trade-off prioritized metacognitive support over brevity due to the item's inherent difficulty. Conversely, item 4 (Fig. 3 and 4) prioritized syntactic simplicity over content depth as reading comprehension was considered a more significant barrier to problem-solving. To effectively navigate these challenges, consider the following strategies:

1. Identify the linguistic feature that poses the greatest barrier to understanding

2. Evaluate the potential consequences of each modification

3. Strive for a balance between simplifying language and maintaining the level of difficulty of the item

4. Test modifications with a small group. Refine and consider alternative approaches to address the accessibility issue.

5. Seek input from language specialists if possible and chemistry educators.

The following sections discuss some actionable strategies to design more inclusive, accessible and equitable items for students, particularly multilingual learners. While we offer practical strategies for linguistically simplifying assessment items based on the findings of the current and previous studies (Lee and Orgill, 2021), it's important to recognize that there is more than one way to linguistically simplify an item, which may vary based on the topic and/or subject area.

Pedagogical insights for inclusive assessments. Based on the findings of the current study, an important way to create more inclusive assessment is to use more inclusive language by avoiding uncommon language and phrases such as idioms, colloquialisms, and cultural references. For example, the sample original item (Fig. 9) below uses the phrase “burning the midnight oil” as well as the idiom “pulling an all-nighter.” These phrases may be unfamiliar to multilingual learners. They might spend time wondering if the question is asking about the effect of burning oil on the reaction or the actual amount of time the student spends working, which distracts from the core chemistry concepts being tested. They might focus on what an “all-nighter” is instead of focusing on key chemistry concepts in the problem. The original item also uses “Jacob” and the pronoun “he,” to represent the student, which can be exclusionary. It can also be beneficial to rephrase the scenario to avoid gender bias. Fig. 9 shows one way that the original item could be reworded and reformatted to be more inclusive and equitable (note the addition of bold text to highlight the goal of the assessment item).

Fig. 9 Original item (using non-inclusive language) on the left and revised item (using more inclusive language) on the right.

Language and extralinguistic considerations in assessment design. The findings also emphasized the significance of specific language considerations in the design of assessment items. Incorporating effective scaffolds and internal signals can be immensely helpful in guiding students through the text of the item and helping them identify key information.

For example, the sample original item in Fig. 10 does not provide any prompts or steps to guide students towards the concept of conservation of mass. It does not ask them to identify knowns and unknowns or set up an equation to solve for the missing mass. There is no mention of the chemical equation for the reactions, which might be crucial for understanding the formation of products and potential gaseous byproducts. The question lacks a clear connection between the law of conservation of mass and the observed discrepancy in mass. A more effective way to present this item would be to rearrange the background information to present smaller bits of information and scaffold the main questions so that students are first asked what the law of conservation of mass is before applying it to the chemical reaction.

Fig. 10 Original item (without metacognitive support) on the left and revised item (with additional metacognitive support) on the right.

Readability and vocabulary considerations in assessment design. The findings highlighted challenges related to readability and vocabulary, especially for multilingual learners. Recognizing the challenges multilingual learners face with complex sentence structures and advanced terms, instructors can benefit from exploring ways to simplify language without compromising the integrity of the content. This involves selecting clear and precise language, considering the nuances of vocabulary, and adapting assessments to cater to diverse linguistic backgrounds.

For example, the sample original item below (Fig. 11) has words like “constituent ions,” and “electrical neutrality,” which might be unfamiliar or pose difficulty for multilingual learners. The sentence structure is complex, particularly “separation of charges” which could be challenging to parse for students still developing their English language skills. In the revised version, simpler vocabulary has been used such as “breaks apart” instead of “dissociation” and “tiny, charged particles” instead of “constituent ions.” It also has simpler sentences. Please note that we are not suggesting the removal of all challenging chemistry terms from the assessment items. We believe that if these terms are essential to the evaluation and students have been adequately prepared to understand and use them, they should be included.

Fig. 11 Original item (with complex vocabulary and phrasing) on the left. Revised item (with simpler vocabulary and phrasing) on the right.

Overall, language barriers can significantly hinder students’ ability to demonstrate their knowledge. Therefore, designing equitable assessments requires careful consideration of language use in order to create a level playing field for students with diverse linguistic backgrounds. Our findings, along with a similar study involving EngL1 students (Gandhi-Lee, 2018) suggest that most of the modifications recommended by the EFCA framework can benefit all students by promoting clearer interpretation of assessment items. The only EFCA recommendation that students in the current study did not find useful was the addition of a storyline as contextualization, which the students found to require significant reading time—time that they did not have available on high-stakes, timed chemistry exams. Other EFCA recommended modifications, such as linguistic simplification of vocabulary and syntax and the use of bold type for emphasis, will help students overcome some of the challenges they identified in assessment items. In fact, based on the results of this study, the EFCA framework has the potential to narrow the performance gap between multilingual learners and EngL1 students in tertiary chemistry courses. However, future research should explore the broader impact of EFCA modifications and how they benefit all learners in more nuanced ways. This focus will be crucial in achieving equitable assessment practices in chemistry, ensuring that all students have a fair chance to demonstrate their understanding of the subject matter.

Author contributions

The first and second authors worked together to conceptualize and plan the study. The first author collected data and carried out the initial data analysis, meeting with the second author to discuss and verify the findings. Both authors contributed to the writing and revision of the manuscript.

Data availability

The data are not publicly available as approval for this study did not include permission for sharing data publicly.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

We acknowledge Anna Eunji Kim in helping us organize the data.

References

1. Abedi J., (2011), Language Issues in the design of accessible items, in Elliott S., Kettler R., Beddow P. A. and Kurz A. (ed.), Handbook of Accessible Achievement Tests for all students: Bridging the gaps between research, practice, and policy, Springer.
2. Bergey R., Movit M., Baird A. S. and Faria A., (2018), Serving English language learners in Higher Education: Unlocking the potential, American Institutes for Research.
3. Braun V. and Clarke V., (2006), Using thematic analysis in psychology, Qual. Res. Psychol., 3(2), 77–101.
4. Brown T. E., LeMay, Jr H. E. and Bursten B. E., (2015), Chemistry: The Central Science, 13th edn, Upper Saddle River, NJ: Pearson Education.
5. Bruner J., (1983), Child's talk: Learning to use language (M), Oxford: Oxford University Press, 1983.
6. Buono S. and Jang E. E., (2021), The effect of linguistic factors on assessment of English language learners’ mathematical ability: a differential item functioning analysis, Educ. Assess., 26, 125–144.
7. Buxton C., Allexsaht-Snider M., Aghasaleh R., Kayumova S., Kim S., Choi Y. and Cohen A., (2014), Potential benefits of bilingual constructed response science assessments for understanding bilingual learners’ emergent use of language of scientific investigation practices, Double Helix, 2, 1–21.
8. Buxton C., Harman R., Cardozo-Baibisso L., Jiang L., Bui K. and Allexsaht-Snider M., (2019), Understanding science and language connections: new approaches to assessment with bilingual learners, Res. Sci. Educ., 49, 977–988.
9. Buxton C. and Lee O., (2023), Multilingual learners in science education, in Handbook of research on science education, N. G. Lederman, D. Zeidler and J. Lederman, (ed.), (3rd edn), Routledge DOI:10.4324/9780367855758.
10. Cardozo-Gaibisso L., Kim S., Buxton C. and Cohen A., (2020), Thinking beyond the score: multidimensional analysis of student performance to inform the next generation of science assessments, J. Res. Sci. Teach., 57, 856–878.
11. Carter C. S. and Brickhouse N. W., (1989), What Makes Chemistry Difficult? Alternate Perceptions, J. Chem. Educ., 66 (3), 223.
12. Cassels J. R. T. and Johnstone A. H., (1983), The Meaning of Words and the Teaching of Chemistry, Educ. Chem., 20, 10–11.
13. Collier V. P., (1995), Acquiring a second language for school, Direct. Language Educ., National Clearinghouse for Bilingual Education, 1(4), 1–8.
14. Creswell J. W., (2007), Qualitative inquiry and research design: Choose among five approaches, 2nd edn, Thousand Oaks, CA: SAC Publications.
15. Cummins J., (2000), Language, power and pedagogy: Bilingual children in the crossfire, Clevedon, England: Multilingual Matters.
16. Deng J. M. and Flynn A. B., (2023), “I am working 24/7, but I can’t translate that to you”: the barriers, strategies, and needed supports reported by chemistry trainees from English-as-an-additional language backgrounds, J. Chem. Educ., 100, 1523–1536.
17. Fang Z., (2006), The language demands of science reading in middle school, Int. J. Sci. Educ., 28, 491–520.
18. Fine C. G. Mc. C. and Furtak E. M., (2020a), A framework for science classroom assessment task design for emergent bilingual learners, Sci. Educ., 104, 393–420.
19. Fine, C. G. Mc. C. and Furtak E. M., (2020b), The SAEBL Checklist: science classroom assessments that work for emergent bilingual learners, Sci. Teacher, 38–48.
20. Gandhi-Lee E. N., (2018), Breaking the language barrier: Equitable assessment in general chemistry, [Doctoral Dissertation, University of Nevada, Las Vegas]. ProQuest Dissertations & Theses Global.
21. Garcia O., Flores N., Seltzer K., Wei L., Otheguy R. and Rosa J., (2021), Rejecting abyssal thinking in the language and education of racialized bilinguals: a manifesto, Critical Inquiry Language Studies, 18, 203–2228.
22. Gibbons P., (2006), Bridging Discourses in the ESL Classroom: Students, Teachers, and Researchers, London: Continuum.
23. Goubeaud K., (2010), How is science learning assessment at the postsecondary level? Assessment and grading practices in college biology, chemistry and physics, J. Sci. Educ. Technol., 19, 237–245.
24. Grapin S. E., (2023), Assessment of English learners and their peers in the content areas: expanding what “counts” as evidence of content learning, Language Assessment Quarterly, 20, 215–234.
25. Grapin S. E., Pierson A., Gonzalez-Howard M., Ryu M., Fine C. and Vogel S., (2023), Science education with multilingual learners: equity as access and equity as transformation, Sci. Educ., 107, 999–1032.
26. Guzman-Orth D., Supalo C. A., Smith D. W., Lee O. and King T., (2021), Equitable STEM instruction and assessment: accessibility and fairness considerations for special populations, ETS Res. Rep. Ser, 2021(1), 1–16 DOI:10.1002/ets2.12324.
27. Honey M., Alberts B., Bass H., Castillo C., Lee O., Struches M. M., Vermillion L. and Rodriguez F., (2020), STEM education for the future: a visioning report, National Science Foundation.
28. International Students Community College, Leading Associate's Institutions, (2003/04/-2022/23), Open Doors Report on International Educational Exchange, Retrieved from. https://www.opendoorsdata.org.
29. Lee O. and Buxton C., (2013), Integrating Science and English Proficiency for English Language Learners, Theory Into Practice, 52(1), 36–42.
30. Lee O. and Fradd S. H., (1998), Science for all, including students from non-English-Language backgrounds, Educ. Res., 12–21.
31. Lee E. N. and Orgill M., (2021), Toward equitable assessment of English language learners in general chemistry: identifying supportive features in assessment items, J. Chem. Educ., 99, 35–48.
32. Lopez A. A., Turkan S. and Guzman-Orth D., (2017), Conceptualizing the use of trans-languaging in initial content assessments for newly arrived emergent bilingual students, ETS Res. Rep. Ser., 1, 1–12 DOI:10.1002/ets2.12140.
33. MacDonald R., Crowther D., Braaten M., Binder W., Chien J., Dassler T., Shelton T. and Wilfrid, J., (2020), Design principles of engaging multilingual learners in three-dimensional science, WCER Working Paper No. 2020-1. Wisconsin Center for Education Research. Available from: https://www.wcer.wisc.edu/publications/working-papers.
34. National Academies of Sciences, Engineering, and Medicine, (2018), English learners in STEM subjects: Transforming classrooms, schools, and lives, National Academies Press.
35. National Center for Education Statistics, (2022), The condition of education 2021 (NCES 2022-144), U.S. Department of Education. https://nces.ed.gov/pubs2021/2021144.pdf.
36. Neri N. C. and Retelsdorf J., (2022), The role of linguistic features in science and mat comprehension and performance: a systematic review and desiderata for future research, Educ. Res. Rev., 36(100460), 1–15.
37. Pierson A. E. and Grapin S. E., (2021), A disciplinary perspective on trans-languaging. Bilingual Res. J., 44, 318–334.
38. Pun J. K. H., Fu X. and Cheung K. K. C., (2023), Language challenges and coping strategies in English medium instruction (EMI) science classrooms: a critical review of the literature. Studies Sci. Educ., 60(1), 121–152.
39. Pun J. K. H. and Tai K. W. H., (2021), Doing science through trans-languaging: a study of trans-languaging practices in secondary English as a medium of instruction science laboratory sessions, Int. J. Sci. Educ., 43, 1112–1139.
40. Rees S., Kind V. and Newton D., (2019), Meeting the Challenge of Chemical Language Barriers in University Level Chemistry Education, Israel J. Chem. Educ., 59 (6–7), 470–477.
41. Siegel M. A., (2007), Striving for equitable classroom assessments for linguistic minorities: strategies for and effects of revising life science items, J. Res. Sci. Teach., 44(6), 864–881 DOI:10.1002/tea.20176.
42. Siegel M. A., Menon D., Sinha S., Promyod N., Wissehr C. and Halverson K. L., (2014), Equitable Written Assessments for English Language Learners: How Scaffolding Helps, J. Sci. Teacher Educ., 25(6), 681–708 DOI:10.1007/s10972-014-9392-1.
43. Siegel M. A., Wissehr C. and Halverson K., (2008), Sounds likes success: a framework for equitable assessment, The Science Teacher, 43–46.
44. Thompson S. J., Johnstone C. J. and Thurlow M. L., (2002), Universal design applied to large scale assessments, (NCEO Synthesis Report No. 44), University of Minnesota, National Center on Educational Outcomes.
45. Wellington J. and Osborne J., (2001), Language and Literacy in Science Education, Open University Press.
46. Wilson-Kennedy Z. S., Winfield L. L., Nielson J., Arriaga E. A., Kimble-Hill A. C. and Payton-Stewart F., (2022), Introducing the Journal of Chemical Education's Special Issue on Diversity, Equity, Inclusion, and Respect in Chemistry Education Research and Practice, J. Chem. Educ., 99 (1), 1–4 DOI:10.1021/acs.jchemed.1c01219.
47. Zumdahl S. S. and Zumdahl S. A., (2012), Chemistry: An Atoms First Approach, 2nd edn, Boston, MA: Cengage Learning.

---