Design, development, and evaluation of the organic chemistry representational competence assessment (ORCA)

Abstract

This paper describes the design and evaluation of the O̲rganic chemistry R̲epresentational C̲ompetence A̲ssessment (ORCA). Grounded in Kozma and Russell's representational competence framework, the ORCA measures the learner's ability to interpret, translate, and use six commonly used representations of molecular structure (condensed structures, Lewis structures, skeletal structures, wedge-dash diagrams, Newman projections, and chair conformations). Semi-structured interviews with 38 first-semester organic chemistry learners informed the development of the ORCA items. The ORCA was developed and refined through three pilot administrations involving a total of 3477 first-semester organic chemistry students from multiple institutions. The final version of the ORCA was completed by 1494 students across five institutions. Various analyses provided evidence for the validity and reliability of the data generated by the assessment. Both one-factor and three-factor correlated structures were explored via confirmatory factor analysis. The one-factor model better captured the underlying structure of the data, which suggests that representational competence is better evaluated as a unified construct rather than as distinct, separate skills. The ORCA data reveal that the representational competence skills are interconnected and should consistently be reinforced throughout the organic chemistry course.

---

Introduction and rationale

Learning and communicating with representations (e.g., chemical structures, graphs, or reaction equations) are essential goals of chemistry instruction (Gilbert, 2005; Kozma and Russell, 2005; Ainsworth, 2006). Correspondingly, representations are integral to chemistry assessment. A historical analysis of 18 organic chemistry American Chemical Society (ACS) exams revealed that more than 90% of exam items include at least one representation (Raker and Holme, 2013). The ACS Exams Institute includes visualizations as one of the ten anchoring concepts for undergraduate chemistry (Murphy et al., 2012). Additionally, the National Academies of Science emphasize the importance of reasoning with representations to engage in scientific and engineering practices (e.g., developing and using models, analysing and interpreting data, and obtaining, evaluating, and communicating information) (National Research Council, 2012a). It is, therefore, unsurprising that science educators and discipline-based education researchers have devoted considerable attention to learners’ understanding of scientific representations (National Research Council, 2012b).

Mastering the “language” of chemical representations is not an easy task. Novices must learn about representations (e.g., how to interpret representations) and learn with representations (e.g., use representations to make sense of chemical phenomena) (Talanquer, 2022). Research shows that learners rely on heuristics when making sense of representations (Talanquer, 2014), primarily focus on surface features of representations (Cooper et al., 2010; Popova and Bretz, 2018c), and struggle to connect the external features of representations to the conceptual information embedded in them (Popova and Bretz, 2018a, b; Ward et al., 2022; Rotich et al., 2024). Despite the important role that representational competence plays in learner success in chemistry (Seufert, 2003; Kozma and Russell, 2005; Ainsworth, 2006; Schönborn and Anderson, 2008; Stieff et al., 2016), and the considerable number of investigations into learners’ ability to reason about symbolic and submicroscopic representations (Bodner and Domin, 2000; Cooper et al., 2010; Padalkar and Hegarty, 2012; Padalkar and Hegarty, 2015; Stieff et al., 2016; Stull et al., 2016; Stull and Hegarty, 2016; Miller and Kim, 2017; Lansangan et al., 2018; Popova and Bretz, 2018a, b, c; Connor et al., 2019; Wright and Oliver-Hoyo, 2020; Farheen and Lewis, 2021; Ward et al., 2022; Farheen et al., 2024), a limited number of assessment instruments exist that measure learner competence with representations. Within this manuscript, we describe the development of an assessment to measure aspects of learner representational competence in the context of multiple commonly used representations of molecular structure.

Conceptual framework – representational competence

Representational competence (RC) is a “set of skills and practices that allow a person to reflectively use a variety of representations or visualizations, singly and together, to think about, communicate, and act on chemical phenomena in terms of underlying, a perceptual physical entities and processes” (Kozma and Russell, 2005, p. 131). Through an examination of expert and novice investigative chemistry practices (Kozma and Russell, 1997; Kozma et al., 2000), Kozma and Russell (2005) proposed a set of skills that comprise RC:

(a) use representations to describe observable chemical phenomena in terms of underlying molecular entities and processes (use),

(b) generate or select a representation and explain why it is appropriate for a particular purpose (generate and select),

(c) use words to interpret features of a particular representation (interpret),

(d) make connections across different related representations by mapping features of one representation onto those of another (translate),

(e) use representations in social situations to support claims, draw inferences, and make predictions about chemical phenomena (use),

(f) describe the limitations and affordances of different representations (identify affordances and limitations), and

(g) take the epistemological position that representations correspond to but are distinct from the phenomena that are observed (take epistemological position).

Most skills outlined above can be developed at a lower foundational level or a higher meta-level that allows for the reflective and purposeful use of representations (diSessa and Sherin, 2000; diSessa, 2004; Gilbert, 2005; Kozma and Russell, 2005; Ward et al., 2022). For the remainder of the article, we will use the succinct labels in parentheses to reference each skill.

RC is grounded in the cognitive theory of multimedia learning and situative theory. Together, these two theories highlight the interplay between cognitive processing and the social context of learning for understanding how students master chemical representations. The cognitive theory of multimedia learning assumes that learning occurs as a result of processing and synthesizing information across the audio and visual modes of instructional information (Mayer, 2002; Kozma and Russell, 2005). In particular, the ability to effectively process visual information is critical when it comes to learning chemical representations, as these often involve intricate visual details that convey essential information about sub-microscopic particles and processes. Situative theory suggests that the physical and social characteristics of a setting shape the processes of understanding and learning within that setting (Lave, 1991; Lave and Wenger, 1991; Kozma and Russell, 2005). As individuals become integrated into a community, they progressively develop in using its representational systems to construct new knowledge and communicate information. This integration and participation in communal practices enable learners to understand and manipulate the representations used by that community, facilitating deeper learning and expertise in the subject matter.

Several assumptions underlie the development of RC. First, a developmental trajectory is assumed, implying that learners' ability to use chemistry representations increases as they progress in their learning. With continued learning, representations become a valuable tool for constructing and communicating understanding, resulting in the sophistication of RC over time (Kozma and Russell, 2005). Second, levels of RC are not assumed to be consistent for different types of representations. For example, a learner may be proficient with most of the RC skills for a particular representation (such as Lewis structures) and only be able to interpret the features of another (such as reaction coordinate diagrams) (Kohl and Finkelstein, 2006; Stieff et al., 2011; Halverson and Friedrichsen, 2013). Finally, RC and conceptual understanding are separate but related components of learner knowledge (Hinze et al., 2013; Maroo and Johnson, 2018). Proficiency in understanding chemical concepts is achieved in tandem with proficiency in utilizing the visualizations that represent and explain those concepts (diSessa, 2004; Kozma and Russell, 2005).

Studies investigating learner RC in chemistry have primarily relied on the use of interviews. Interviews provide nuanced and rich data on how learners engage and reason with representations that result in transferable conclusions. Many of these studies suggested a relationship between RC and conceptual understanding (Lansangan et al., 2007; Höst et al., 2012; Hiong and Daniel, 2015; Pande et al., 2015; Lansangan et al., 2018; Pande and Chandrasekharan, 2022). However, interview data do not allow for drawing robust, generalizable conclusions for various groups of learners, for example, those experiencing different learning environments (Herrington and Daubenmire, 2014). Studies that use assessments that can be easily administered and scored on a large scale have the potential to improve our understanding of RC as a construct and how to effectively develop RC through instruction. This need was highlighted by Kozma and Russell, who noted that “new assessments must be designed and used that measure investigation practices and related skills, such as visualization skills or representational competence” (Kozma and Russell, 2005, p. 142).

RC (Sim and Daniel, 2014; Connor et al., 2021), or frameworks related to RC (e.g., Visualization Competence of Matter (Chang and Tzeng, 2017) and Visual Model Comprehension (Dickmann et al., 2019)) have been the foundation of some assessments developed to elicit secondary or postsecondary chemistry learners’ ability to make sense of representations. These assessments have corroborated findings from interviews stating that representational abilities are greater for learners who received more chemistry instruction (Chang and Tzeng, 2017; Vlacholia et al., 2017) or who have a higher conceptual understanding (Sim and Daniel, 2014; Dickmann et al., 2019). In addition, a few studies found that competence varies by skill or topic (Chang and Tzeng, 2017; Chang, 2018). However, these assessments are either not used to investigate learner RC explicitly (Chang and Tzeng, 2017; Vlacholia et al., 2017; Wang et al., 2017; Chang, 2018; Dickmann et al., 2019), are focused on a single aspect of RC (Connor et al., 2021), or a limited number of representations (Sim and Daniel, 2014; Connor et al., 2021). Some of these assessments have useful but limited evidence for the validity of resulting data, primarily in the form of content validity using expert panels (Chang and Tzeng, 2017; Chang, 2018). This contrasts with more robust evaluations of validity and reliability, including content validity as well as a broader array of psychometric evidence. To advance our understanding of learner RC, quality assessments that demonstrate evidence of valid and reliable data are needed that comprehensively capture multiple RC skills in the context of multiple representations (Kozma and Russell, 2005).

Purpose of this study

This work aims to develop and evaluate the O̲rganic chemistry R̲epresentational C̲ompetence A̲ssessment (ORCA) to characterize learners’ RC in the context of six commonly used representations of molecular structure in undergraduate organic chemistry instruction: condensed structures, chair conformations, wedge-dash diagrams, Lewis structures, skeletal structures, and Newman projections. Chemical representations have various dimensions (i.e., iconicity, granularity, dimensionality, and quantitativeness) that impact how learners reason about and with representations (Talanquer, 2022). This study, in particular, centers on symbolic representations with similar dimensions (Johnstone, 1993).

The ORCA is designed as a multiple-choice assessment, permitting testing with a large number of participants and allowing for quick grading. The assessment captures three RC skills: the ability to (a) use words to interpret features of a particular representation – the interpret skill; (b) make connections across different related representations by mapping features of one representation onto those of another – the translate skill; and (c) use representations to draw inferences – the use skill. These skills are among the most commonly taught by organic chemistry educators (Linenberger and Holme, 2015; Popova and Jones, 2021; Jones et al., 2022) and in organic chemistry textbooks (Gurung et al., 2022). The ORCA does not assess the learner's ability to generate representations, another skill typically taught in the target course, as it is difficult to evaluate this skill properly using multiple-choice questions. Other RC skills were not considered for the ORCA as current instruction offers little to no support for those skills (Linenberger and Holme, 2015; Popova and Jones, 2021; Gurung et al., 2022; Jones et al., 2022). Two research questions guide this study:

1. What evidence exists for the validity and reliability of the data generated from the Organic chemistry Representational Competence Assessment (ORCA)?

2. What do the data collected with ORCA reveal about learner representational competence?

Methods: data collection, analysis, and evidence of validity and reliability of the data

Criteria for developing and evaluating the ORCA were adopted from the Standards for Educational and Psychological Testing (American Educational Research Association, 2014) and guided by other primers for assessment development in chemistry education (Towns, 2008; Arjoon et al., 2013; Towns, 2014; Komperda et al., 2018). The development and evaluation processes are organized by the various stages and shown in Table 1. Institutional Review Board guidelines were met at every institution where data were collected and in all stages of the assessment development and evaluation. All participants were informed that their ORCA performance would not negatively impact their course grades. Additionally, the participants could consent or decline consent for their responses to be included in this research study.

Table 1 ORCA development and evaluation

Method	Outcome	Application
Stage I – assessment development
Semi-structured think-aloud interviews	Elicit learner ability to reason with representations	Generate items and distractors from participant responses to establish validity evidence based on test content
Expert evaluation	Obtain feedback from experts about the appropriateness of the items for the measurement of the target construct	Establish validity evidence based on test content and revise items accordingly
Stage II – pilot administrations
Item analysis	Understand item difficulty, item discrimination, and how well distractors function	Revise items accordingly to ensure item quality
Factor analysis	Investigate the number of factors, the variables that load onto each factor, and the level of correlation among factors	Establish internal structure validity and revise instrument accordingly
Cognitive interviews	Obtain insight into respondents’ thought processes and feedback about items	Establish response process validity and revise items accordingly
Expert evaluation	Obtain additional feedback from experts about the appropriateness of the items for the measurement of the target construct	Establish validity evidence based on test content and revise items accordingly
Stage III – final administration
Item analysis	Understand item difficulty, item discrimination, and how well distractors function	Ensure item quality
Factor analysis	Confirm the factor structure of the revised ORCA instrument	Establish internal structure validity
Reliability coefficient	Understand correlations between items	Establish internal consistency/reliability

---

We iteratively developed the ORCA by using a sequential, mixed-methods exploratory design (Towns, 2008) and a bottom-up approach that originated from analyses of learner reasoning about representations of molecular structure collected using semi-structured interviews (Bowen, 1994). At the time of their participation, all participants were in the first semester of a year-long organic chemistry course, having completed the two-semester general chemistry sequence. We wrote assessment items from the interview data collected at one institution (Stage I), and we refined these items through three pilot administrations with participants at six institutions (Stage II) using the Qualtrics platform. The final ORCA version was administered to learners at five institutions (Stage III) via Qualtrics. Only learners who completed the entire assessment received extra credit for their participation, which is why all participants completed ORCA in its entirety, and we did not have any missing data.

Stage I – assessment development

We conducted semi-structured interviews; data from a portion of these interviews were used to design the ORCA items. We recruited participants (N = 38) from a public university in the southeastern US during the Fall of 2019 and Spring of 2020. The participants were enrolled in five different Organic Chemistry I course sections taught by four instructors using the same curriculum and textbook (Bruice, 2016). We purposefully sampled learners, using a stratified method (Patton, 2002), to ensure a range of grades (from “A” to “C”) in General Chemistry II. Participants were compensated with a $20 gift card. Pseudonyms were assigned to all participants to protect their identities. Data were captured using an audio recorder, a video camera, and a Livescribe smartpen (https://www.livescribe.com).

The interviews were designed to elicit learners’ RC skills. To capture how learners interpret a representation, participants were asked to (a) describe what the representation communicates, (b) decode each diagrammatic feature of the representation (e.g., dashes, wedges, lines, symbols, etc.), (c) explain the purpose of the representation, and (d) identify the atomic composition for the given representation. To elicit how learners use a representation, participants were asked to examine a given representation and make inferences about (a) chemical and physical properties, (b) bonding, and (c) energy and stability. To capture how learners translate between various representations of molecular structure, the participants were given a specific representation and asked to choose the corresponding structure(s) from four options. They were informed that each task may have more than one corresponding structure. After selecting the corresponding structure(s), the participants were asked to explain why they believed the structure(s) corresponded and why the other options did not.

The data from the interviews were transcribed verbatim, inductively coded using ATLAS.ti software (version 9, https://atlasti.com/), and analyzed using constant comparative analysis (Glaser, 1965). The first and the second author coded the data and, depending on a code, obtained 84–100% agreement (Saldaña, 2013). Furthermore, they discussed each case of disagreement in their coding until a 100% negotiated agreement was reached. In addition, the first, second, and sixth authors met weekly to discuss the codes and analyze the data for patterns. Some of the qualitative results from these analyses have been published elsewhere (Ward et al., 2022; Rotich et al., 2024).

We then used these data to generate over 100 assessment items to capture how learners interpret, translate, and use six representations of molecular structure. Fig. 1 depicts how we used participant interview data (Fig. 1A) to write an item that asks participants to use Newman projections to make inferences about stability (Fig. 1B). Response choices, including distractors, were developed using participants’ ideas from the interview and reflected the most common patterns in participant thinking. We reviewed all the items and eliminated items if: (a) there were too many items about the same representation or skill, (b) items were complex multiple-choice questions (Albanese, 1993; Towns, 2014), or (c) items were not representative of the skills learners typically learn in the course (i.e., translating between a chair conformation and a condensed structure). This process resulted in condensing the assessment to 64 items.

Fig. 1 An example preliminary item (B) written based on participant interview data (A), as well as the refined item (D) based on expert feedback (C) to establish validity evidence based on test content.

An expert panel of six organic chemistry instructors and chemistry assessment developers evaluated all 64 items for content validity. We used the feedback from the expert panel members (Fig. 1C) to eliminate some items and refine other items (Fig. 1D). This process resulted in 48 items for which the expert panel established that the items adequately covered the relevant content to measure the target constructs (interpret, translate between, and use representations of molecular structure) and were appropriate for the target population (organic chemistry learners). In addition, they provided feedback about terminology and the scoring of items.

Stage II – pilot administrations

Following expert evaluation, we pilot-tested the 48 items. During each pilot test, participants completed the assessment at least three weeks after covering the six target representations of molecular structure in their Organic Chemistry I course. In consensus with the feedback received from the expert panel, and to minimize item priming, we administered the assessment items in a specific order, such that translate items appeared first, then use items, and lastly, interpret items. Within each skill, the items were randomized for each participant.

Pilot I. In Fall 2021, the 48 items were administered as a Pilot I to learners (N = 1120) taught by five instructors at two (medium and large) public universities in the southeastern US. To ensure a timely assessment, we split the items into two 24-item forms that could be completed in less than 30 minutes. Each form had comparable items that featured the same representations and targeted the same skills. For example, Form 1 and Form 2 each contained one item where participants were to use a Newman projection to make inferences about stability. Participants randomly received one of the two comparable forms. We used classical test theory considerations (i.e., item difficulty, discrimination, and response choice distributions) (Towns, 2014; Bandalos, 2018) to select the best-performing items across the two forms. This resulted in a 24-item assessment with seven interpret skill items, eleven translate skill items, and six use skill items. The six use items asked participants to infer stability from Newman projections, chair conformations, and Lewis structures (i.e., two items for each representation).

Pilot II. We administered the 24 items as a Pilot II in Spring 2022 to participants (N = 567) at the same two institutions as in Pilot I. This administration was used to provide preliminary evidence of the internal structure of the assessment through confirmatory factor analysis (CFA). Classical test theory considerations revealed the assessment demonstrated a range of item difficulties and discriminations for most items and a few items under the translate skill had poor answer choice distribution. We used data from cognitive interviews to improve those items. In addition, we wrote six new use items (generated using the interviews from Stage I) to target how learners infer physical properties from condensed structures, skeletal structures, and wedge-dash diagrams. We incorporated these new items into the response process interviews as well. We conducted cognitive interviews with learners (N = 5) two weeks after the Pilot II administration. We used stratified purposeful sampling to select high, middle, and low-scoring ORCA participants to obtain evidence for response process validity (Patton, 2002; Deng et al., 2021). For each item, the participants responded to the following questions:

• What does this question ask you to do?

• What information do you believe is relevant for responding to this question?

• Generate a representation that corresponds to this structure (translate items only).

• How did you decide to pick 〈response option selected〉?

• Did you experience any challenges when providing your response to this question?

During the interview, participants first addressed poor-performing items from the assessment and then the six newly written items. We used responses from the interviews to refine the item stems, modify distractors, and add or delete items. For some items, participants pointed out key features of the representations that cued them to eliminate a distractor or made the correct answer obvious. Based on participants’ feedback, we made adjustments to these items, deleted two items where participants described confusing terminology, and added four of the six newly generated use items to the assessment.

Pilot III. Finally, we tested the finalized 26-item ORCA again as a Pilot III in the middle of the Fall 2022 semester with participants (N = 1790) across six medium and large public universities in the southeastern, midwestern, and western US. The instrument contained seven interpret items, eleven translate items, and eight use items. Once again, we examined the quality of the items for distractor selection, item difficulty, and item discrimination, as well as conducted CFA to investigate the internal structure of the assessment. Only one item performed poorly on Pilot III. The terminology for this use item had been adjusted after the response process interviews in Pilot II and this led to poor discrimination and low difficulty. Recognizing the impact of this change, we reverted the terminology of this item to its original format from Pilot II for the final administration. Additionally, the expert panel evaluated the refined assessment instrument to confirm the content validity. Persons interested in obtaining a copy of the finalized ORCA should see the ESI.†

Stage III – final administration

The final ORCA was administered at the end of Fall 2022 to Organic Chemistry I learners from five institutions: one medium public university in the southeastern US and four large public universities in the southeastern, midwestern, and western US. Participants completed the assessment with a median time of 22 minutes, and 1508 participants consented for their ORCA responses to be used in this study. Many participants in Pilot III were also part of the final administration. However, the timeframe between the two administrations was over six weeks to minimize the likelihood of participants recalling specific items from Pilot III. Additionally, to further reduce the possibility of recall effects, we implemented a randomization strategy for both the items and their response choices during the final administration. This means that even if students participated in both phases, the presentation of the items and response options in the final administration differed from their arrangement in Pilot III.

Next, we identified any potential data outliers. Based on the recommendations of Aguinis and colleagues (2013), we used single (i.e., box plots and percentage analysis) and multiple construct techniques (i.e., scatterplots, Mahalanobis distance, and studentized deleted residuals). Fourteen outliers (i.e., complete response sets for a given participant) were removed through these analyses. Outlier data had interpret item scores of zero and often had extremely high (e.g., 340 minutes) or low (e.g., 94 seconds) completion times. This process resulted in a final sample of 1494 responses for analysis.

We conducted CFA using MPlus 8 (Muthén and Muthén, 2017) to investigate the internal structure of the assessment instrument. With the RC framework (specifically, the interpret, translate, and use skills) as a theoretical basis, we tested two hypothesized models to evaluate and confirm the number and nature of latent factors. The Means and Variance Adjusted Weighted Least Squares (WLSMV) method was best suited for the analysis (Finney and DiStefano, 2013) as all measured variables were categorical (binary). A good model fit for these categorical data is more stringent than the criteria suggested by Hu and Bentler (Hu and Bentler, 1999; Beauducel and Herzberg, 2006; Xia and Yang, 2019; McNeish and Wolf, 2023). Though the model fit cutoffs are not absolute and the conditions for their use are still undergoing research (McNeish et al., 2018; McNeish and Wolf, 2023), we use criteria adopted by Komperda and colleagues for good fit (i.e., Comparative Fit Index (CFI) > 0.95 and the Root Mean Square Error of Approximation (RMSEA) < 0.05). The proposed models were congeneric, which is why McDonald's omega (ω) was calculated as a reliability indicator to examine the internal consistency of the items within each latent factor (Hancock and An, 2023; Komperda et al., 2018).

Results and discussion

Gathering evidence for the internal structure validity and internal consistency of the ORCA data

Item difficulty and discrimination from the final ORCA administration with 1494 participants are reported in Fig. 2. All items sufficiently discriminated (ρ > 0.2) between the top 27% and bottom 27% of performers (Bandalos, 2018). There was a range of difficulties for the items, with most falling in the 0.3–0.8 range.

Fig. 2 Item difficulty and discrimination for the final ORCA administration. Interpret skill items are shown as purple squares, translate skill items as blue circles, and use skill items as orange triangles.

Confirmatory factor analysis. Based on Kozma and Russell's RC framework, we proposed two models to estimate the internal structure validity of the data. Model A is a three-factor model where each latent factor represents an RC skill (i.e., the ability to interpret, translate, and use representations). In this model, each item was set to load onto its corresponding factor with correlations between the three factors (Fig. 3A). Model B is a one-factor model in which all items load onto a single latent factor representing RC (Fig. 3B). Model fit statistics and the McDonald's omega coefficients are reported in Table 2.

Fig. 3 Standardized parameter estimates for the three-factor (A) and one-factor (B) models. Circles indicate the latent variables. Squares indicate the observed variable item scores. Double-headed arrows between circles represent correlations between factors and the arrows between circles and squares represent factor loadings. Items in model A were set to load on their assigned factors only. All factor loadings are significantly different from zero (p < 0.001).

Table 2 Fit statistics and reliability for the two proposed models

Model	Fit statistics					McDonald's omega (ω)
	χ ^2	df	p	RMSEA (≤0.05)	CFI (≥0.95)	Interpret	Translate	Use
A – 3 factor	694	296	<0.0001	0.030	0.955	0.807	0.843	0.554
B – 1 factor	741	299	<0.0001	0.031	0.950	0.897

---

The data meet the fit criteria for Model A. However, internal consistency for the factors, as measured through McDonalds’ omega reliability coefficient, varied by skill. The translate and interpret factors have a higher reliability, indicating that common constructs (i.e., the ability to translate between representations and the ability to interpret features of an individual representation, respectively) explain a larger amount of the observed variance to the total variance (Komperda et al., 2018). The interpret skill and translate skill items require learners to reason by attending to the diagrammatic features within single or multiple representations (e.g., interpreting dashes and wedges in wedge-dash diagrams, mapping dashes and wedges onto the appropriate axial and equatorial substituents in chair conformations). For this reason, the interpret skill and translate skill factors are highly correlated (Fig. 3A), as students reason about the features of the representations (i.e., composition, connectivity, or spatial information, if applicable) to interpret and translate between representations.

In contrast, the use skill factor has a lower McDonalds’ omega, so a common construct (i.e., the ability to use representations to make inferences) explains less of the observed variance. This is not surprising because the use skill items require participants to not only reason by attending to the features of representations but also by extracting the relevant domain-specific conceptual knowledge embedded in the representation. Moreover, this conceptual knowledge varies by representation (e.g., evaluating the arrangement of substituents in Newman projections to make an inference about stability; evaluating the functional groups in skeletal structures to make an inference about physical properties). In this case, not only do learners need to be able to make sense of the features of a representation, but they also need to have a conceptual understanding of stability or physical properties, and know which relevant features of the representation to attend to make inferences about these concepts. Therefore, a lower reliability for the use skill factor is hypothetically expected as it may not be a homogenous construct (Taber, 2018); however, this lower reliability also limits the inferences that can be made about the use skill from the items within this assessment. Moreover, the factor loadings (Fig. 3A) for the interpret and translate factors are higher than those in the use factor, further indicating that the interpret and translate skill factors explain more variance within the items (Bandalos, 2018).

Komperda and colleagues (2018) suggest that when “an instrument is known to be composed of multiple scales where scores will be reported separately, each scale should be evaluated to determine if it fits a single-factor model, and a reliability value should be provided for each scale of data.” Data for each factor of Model A were evaluated with individual latent factor models; these CFA results provide support that interpret and translate scales fit the data and have factor loadings >0.3 (see Fig. S1 and Table S1 in the Appendix, ESI†). However, while the RMSEA is appropriate, the CFI for the use skill is below the more stringent cut-off for categorical data. Given that the use scale has a weaker model fit, these analyses provide evidence that only the interpret and translate scales can function independently. In summary, although Model A has a solid theoretical basis, aligned with the stance that RC is a “set of skills and practices that allow a person to reflectively use a variety of representations” (Kozma and Russell, 2005, p. 131), and the data meet the thresholds for the fit statistics, the use scale has items that have lower factor loadings, weaker internal consistency, and, unlike the interpret and translate scales, cannot be used independently.

In comparison, Model B is a single-factor model where every item loads onto a single latent factor—RC. The data fit Model B (Fig. 3B), suggesting that this model effectively captures the underlying structure of the data. Model B accounts for the high latent correlation between the interpret and translate skill factors (evidenced as 0.934 in Fig. 3A). While it is possible that the ORCA may be limited in distinguishing these factors, it is more likely that these skills are very related and should be considered one factor, as reflected in Model B. Additionally, the internal consistency (as measured by McDonald's omega) is higher for the one-factor model B than the three-factor model A. While the increased internal consistency may be attributed to the larger number of items within a single factor (Malkewitz et al., 2023), the empirical data better align with the single-factor structure. This suggests that in the context of representations of molecular structure, these RC skills are better evaluated as a unified construct rather than as distinct, separate skills. Below, we delve deeper into the implications of these findings and share insights into measuring student RC.

Insights gained about learner representational competence

The section below outlines two main insights from examining the data collected with the ORCA about student RC with six key representations of molecular structure in undergraduate organic chemistry: condensed structures, chair conformations, wedge-dash diagrams, Lewis structures, skeletal structures, and Newman projections. These representations are essential to student success in organic chemistry, and our findings provide insights into learners’ proficiency with these symbolic representations.

Representational competence skills are interconnected. The high correlation between each skill in Model A led to the selection of Model B as the best data-fit model (Fig. 3). These correlations indicate that the items intended to measure separate skills may actually be assessing the same underlying construct. In practice, it is difficult to interpret, translate, or use representations of molecular structure independently, as these skills inherently rely on and reinforce one another. This is particularly the case with the interpret skill, which is requisite for the co-development of the rest of the skills. This interconnectedness suggests that RC might be better understood as an interconnected set of skills rather than distinct, separate abilities. Furthermore, many tasks and assessments in chemistry require learners to simultaneously apply multiple RC skills, reinforcing the idea that these skills are not isolated but are used in concert. This perspective highlights that the effective use of chemical representations often depends on the seamless integration of various interrelated skills.

Two examples from our item-level analysis of participant performance further illustrate the interconnectedness of RC skills. Specifically, we observed instances of students having the necessary conceptual understanding to use a representation to make an inference but not being able to connect this knowledge to the appropriate features of representations, which relates to the interpret skill.

Item U15, shown in Fig. 4A (the same item as in Fig. 1D), required participants to make inferences about the stability of Newman projections and justify their reasoning by relying on the concepts embedded in the diagrammatic features of the representation. With the answer choices in this item, participants could have relied on the concepts of substituent size or electronegativity. Fig. 4B demonstrates that the likelihood of selecting the correct answer C is lower for bottom ORCA performers (bottom 27%) and higher for top performers (top 27%). Almost a quarter of participants (23%, Fig. 4C) selected response choice A, in which they chose the wrong structure while reasoning about the appropriate concept of substituent size. These participants knew the productive conceptual information to make inferences about stability but could not connect that knowledge to the proper diagrammatic features of the Newman projection. Finally, almost a quarter of the participants relied on the wrong conceptual information (electronegativity) and selected responses D (13%) or B (11%). This quarter of participants comprise less than 9% of the top performers but over 45% of the bottom performers. For this item, the bottom performers struggled to identify the productive concept of interest.

Fig. 4 Item U15 (A) with the correct answer in bold and italics. The chart (B) shows the response choice distribution for the top 27% and bottom 27% of participants. The count and percentage of learners selecting each answer are shown in table (C).

Item U14 (Fig. 5A) demonstrated a similar pattern. The correct answer, B, was chosen by 41% of all participants. A slight majority of bottom performers (55%) relied on the productive concept reflected in choices B and D. These participants were able to identify the appropriate feature (charge) and connect it to the appropriate concept (octet). However, half of them could not connect that knowledge to the appropriate diagrammatic features to make accurate inferences about stability. Thus, they incorrectly selected the structure with an atom with an incomplete octet (D). Almost a third (30%) of all participants selected response choice D. Finally, nearly a third of the participants chose responses A (20%) or C (9%). These participants relied on features like the presence or absence of double bonds that were not as relevant to the concept of interest (stability) in this prompt.

Fig. 5 Item U14 (A) with the correct answer in bold and italics. The chart (B) shows the response choice distribution for the top 27% and bottom 27% of participants. The count and percentage of learners selecting each answer are shown in table (C).

These examples highlight the complex interplay of RC skills required to answer such questions. Successfully using representations to make inferences about a concept demands a combination of knowledge and ability to (1) interpret diagrammatic features of a representation, (2) conceptually understand the concept of interest, (3) distinguish the relevant from irrelevant features of the representation in relation to the concept of interest, and (4) connect the relevant features of the representation to one's conceptual understanding to make an inference about the concept of interest. Each of these steps is necessary when making inferences about representations.

Representational competence should not be assumed. Instructors are not always intentional about developing learner RC (Linenberger and Holme, 2015; Popova and Jones, 2021; Jones et al., 2022). Our data illustrate that instructors should not assume that RC has been acquired after traditional instruction. For example, instructors may assume that interpreting the wedge-dash diagram in Fig. 6 should be relatively simple and that learners have developed the fundamental ability to identify implicit hydrogens. However, we administered ORCA at the end of the first semester of organic chemistry, and 30% of participants selected an incorrect answer. This is problematic because the inability to identify all implicit hydrogen atoms may significantly impact learner success in the course. For example, this skill is necessary for identifying chiral centers, completing β-elimination reactions, and solving ¹H NMR spectroscopy problems. Our data show that providing learners with continuous practice interpreting implicit atoms in representations is important, even after initial instruction.

Fig. 6 Item I22 (A) with the correct answer in bold. The chart (B) shows the response choice distribution for the top 27% and bottom 27% of participants. The count and percentage of learners selecting each answer are shown in table (C).

Item T9 is another item that some may assume is easy but had relatively surprising responses. The item requires learners to translate between a cyclic skeletal structure and a condensed structure with two of the four answer choices in which the ring is open (Fig. 7A). Most participants selected the correct answer, but over 15% selected a non-cyclic condensed structure (Fig. 7C, options A and B). Participants who select these incorrect answer options for a translate task do not adequately understand the representations or the chemical phenomena represented, even after a whole semester of organic chemistry. As shown, developing RC should be intentional and repeatedly reinforced throughout the curricula.

Fig. 7 Item T9 (A) with the correct answer in bold. The chart (B) shows the response choice distribution for the top 27% and bottom 27% of participants. The count and percentage of learners selecting each answer are shown table (C).

Limitations

Several limitations should be acknowledged. First, the ORCA is limited as it includes items related to only three RC skills. However, these skills are among the most commonly taught by chemistry instructors (Linenberger and Holme, 2015; Popova and Jones, 2021; Jones et al., 2022) and reinforced in organic chemistry textbooks (Gurung et al., 2022). More research is needed to investigate and develop quality assessments that will incorporate the remaining RC skills.

Second, the ORCA assesses learner RC in the context of multiple representations of molecular structure but is not exhaustive (e.g., representations such as electrostatic potential maps and ball-and-stick models are not included even though they have unique affordances to students (Farheen et al., 2024)); we selected symbolic representations only to feature representations with similar dimensions and features (Talanquer, 2022). More assessments are needed to capture learner RC with other representations in chemistry.

Third, the use items capture learners’ ability to make inferences about stability and boiling point only, even though there are other properties that the six target representations could convey. This narrower focus connects to what is covered in instruction and ensures the assessment can be administered in a reasonable timeframe (about 20 minutes). There is potential to consider additional items that focus on other concepts (e.g., aromaticity, acidity).

Fourth, the participants who completed the final assessment also took the assessment during the pilot III administration. There were at least six weeks between assessment administrations (depending on the institution), but this means there is a possibility of practice effects (i.e., improved performance due to familiarity with test items) (American Educational Research Association et al., 2014). Currently, there is no benchmark for when practice effects diminish. Other studies have as low as one month between test administrations (Bretz and Linenberger, 2012).

Lastly, the assessment was administered only online; however, it can be easily adapted to the classroom setting for paper-and pencil administration. Administering the assessment online was a purposeful choice; this allowed for easy access to learners and a quick collection of data. Assessments given in different conditions (such as virtual versus in class), with differing constraints (e.g., not being able to sketch notes), and scored for correctness versus completion (including for bonus points) all likely have varying but overall consistent data (Harle and Towns, 2011).

Conclusions

Herein, we describe the development of the O̲rganic chemistry R̲epresentational C̲ompetence A̲ssessment (ORCA). Grounded in the RC framework (Kozma and Russell, 2005), the ORCA captures learner competence with six representations of molecular structure: condensed structures, chair conformations, wedge-dash diagrams, Lewis structures, skeletal structures, and Newman projections. ORCA items include three interconnected RC skills: the ability to interpret representations, translate between representations, and use representations to draw inferences. The assessment is designed to be multiple choice to permit testing (a) with a large number of learners, (b) in a short amount of time, and (c) to allow for quick and easy scoring. Evidence of the validity and reliability of the ORCA data were established through multiple administrations at several institutions in the US and captured via a variety of methods such as response-process interviews, expert panel feedback, CFA, and McDonald's omega (Table 1).

Two models were proposed to evaluate the internal structure validity of the data generated: a three-factor Model A and a one-factor Model B. While Model A has a solid theoretical basis, our data showed a better fit with Model B. This single-factor model consolidates the skills into a unified latent RC construct, accounting for the high correlation between the interpret and translate skill factors in Model A. This model can be readily used to assess organic chemistry learner RC in the context of representations of molecular structure. Our study is an important step toward advancing and reconceptualizing our understanding of RC as a network of interconnected skills.

Implications

The ORCA can be used as a formative assessment in instruction or as an assessment instrument in research. The ORCA takes about 20 minutes to complete and can be used by instructors to make the necessary adjustments to their instruction and provide feedback to their students about their competence with representations of molecular structure. A single score from the ORCA can provide a measure of learner RC, as reflected by Model B. Given that formative assessment is regarded as a high-impact instructional practice (National Research Council, 2012a,b; Offerdahl and Arneson, 2019), the use of ORCA to receive and implement formative feedback has the potential to improve learner RC and conceptual understanding.

For example, ORCA can be used as a diagnostic tool early in the course to identify areas where students may need additional support, such as with more complex skills like using representations. Based on students' performance, instructors can implement progressive scaffolding, offering explicit guidance early in the semester on how to use different representations. As students build confidence, the level of support can be gradually reduced to encourage independent use of representations.

At the same time, instructors should not assume that it is sufficient to discuss a given representation once, early in the semester. Our findings show that even by the end of the semester, many students struggled with basic tasks, such as identifying implicit hydrogen atoms in wedge-dash diagrams or distinguishing between cyclic and acyclic molecules. It is, therefore, imperative that instructors periodically incorporate explicit explanations of the features of representations that they use in their instruction without assuming that students mastered these representations at the beginning of the course.

Additionally, it's crucial for students to articulate their reasoning, especially when tackling more complex tasks that require using representations to make inferences. Our item analysis revealed that for students to effectively use molecular representations to make inferences, they must be able to do four essential steps: (1) interpret the diagrammatic features, (2) understand the underlying concept, (3) identify relevant versus irrelevant features, and (4) connect those relevant features to their conceptual understanding to draw inferences. Without eliciting students’ reasoning, an instructor or researcher will not be able to understand which of these steps present the challenge for students when using representations to make inferences. This complexity should be taken into consideration when designing assessment tasks or interview prompts.

The ORCA can also be used or adapted as an assessment instrument in studies that aim to advance our understanding of RC as a construct or how to best support learners in developing RC. Prior to using the ORCA, it is imperative that researchers evaluate their data for evidence of validity and reliability (American Educational Research Association et al., 2014; Lazenby et al., 2023). For example, the ORCA can provide evidence of how instructional practices impact learner RC and can be used in longitudinal studies to show how learner RC changes throughout instruction (Kozma et al., 2000). These investigations could help elucidate a learning progression for the development of RC or identify an optimal organic chemistry curriculum for developing RC with representations of molecular structure. While there are existing interventions that support individual RC skills in organic chemistry (Stieff et al., 2016; Stull et al., 2016; Stull and Hegarty, 2016), more research needs to be done to develop interventions that support RC more broadly. In this way, the ORCA can be used as a pre-post measure to provide empirical evidence of approaches that may be useful to support organic chemistry learners.

Future research could also use the ORCA data to conduct Structural Equation Modelling (SEM) to evaluate the relationship between learner RC and other relevant constructs such as their visuospatial ability, conceptual understanding, or success in the organic chemistry course (Nitz et al., 2012; Sim and Daniel, 2014; Stieff et al., 2014, 2018; Stieff and DeSutter, 2020).

While this work focuses on symbolic representations of molecular structure, organic chemistry and other chemistry disciplines also rely on a variety of other representations (e.g., ball-and-stick models, spectra, reaction coordinate diagrams, phase diagrams, and molecular orbital diagrams) that require RC. The process we used to develop the ORCA can be adapted to assess RC skills in these other contexts. Expanding this approach across different contexts will deepen our understanding of how learners develop RC. We encourage the community to build on this work to refine our collective understanding of RC and explore the most effective strategies for supporting learners in developing these important skills.

Ethical considerations

This study has been approved by the University of North Carolina's Institutional Review Board (IRB #20-0511). IRB guidelines were met at every institution where data were collected and in all stages of the assessment development and evaluation. All participants were informed that their ORCA performance would not negatively impact their course grades. Additionally, the participants could consent or decline consent for their responses to be included in this research study. In accordance with the approved IRB protocol, the findings are presented only in aggregate, and the original data sets cannot be shared due to confidentiality considerations.

Author contributions

All authors participated in data collection and/or analysis. Author M. P. conceptualized and led the project and obtained funding for the work. F. R. conducted the interviews and assisted in analyzing data. L. W. W. led the qualitative and psychometric analyses. J. R., S. N., and R. K. provided guidance on the psychometric analyses. L. W. W., M. P., and F. R., wrote the manuscript with input from the rest of the authors.

Data availability

In agreement with the confidentiality measures and procedures approved by the University of North Carolina's Institutional Review Board (IRB # 20-0511), the data cannot be made available.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

This work is supported by the National Science Foundation, DUE 2025216. We thank the instructors who administered ORCA, learners who completed ORCA, and experts who evaluated ORCA.

References

1. Aguinis H., Gottfredson R. K. and Joo H., (2013), Best-Practice Recommendations for Defining, Identifying, and Handling Outliers, Organ. Res. Methods, 16(2), 270–301.
2. Ainsworth S., (2006), DeFT: a conceptual framework for considering learning with multiple representations, Learn. Instruct., 16(3), 183–198.
3. Albanese M. A., (1993), Type K and Other Complex Multiple-Choice Items: An Analysis of Research and Item Properties, Educ. Measurement: Issues Practice, 12(1), 28–33.
4. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, (2014), Standards for Educational and Psychological Testing, Washington, DC: American Education Research Association, 1999.
5. Arjoon J. A., Xu X. and Lewis J. E., (2013), Understanding the state of the art for measurement in chemistry education research: examining the psychometric evidence, J. Chem. Educ., 90(5), 536–545.
6. Bandalos D. L., (2018), Measurement Theory and Applications for the Social Sciences, New York, USA: The Guilford Press.
7. Beauducel A. and Herzberg P. Y., (2006), On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA, Struct. Equ. Model., 13(2), 186–203.
8. Bodner G. M. and Domin D. S., (2000), Mental Models: The Role of Representations in Problem Solving in Chemistry, University Chem. Educ., 4(1), 24–30.
9. Bowen C. W., (1994), Think-aloud methods in chemistry education: understanding student thinking, J. Chem. Educ., 71(3), 184–190.
10. Bretz S. L. and Linenberger K. J., (2012), Development of the enzyme–substrate interactions concept inventory, Biochem. Mol. Biol. Educ., 40(4), 229–233.
11. Bruice P. Y., (2016), Organic Chemistry, 8th edn, Pearson.
12. Chang H. Y., (2018), Students’ representational competence with drawing technology across two domains of science, Sci. Educ., 102(5), 1129–1149.
13. Chang H. Y. and Tzeng S. F., (2017), Investigating Taiwanese Students’ Visualization Competence of Matter at the Particulate Level, Int. J. Sci. Math. Educ., 16(7), 1207–1226.
14. Connor M. C., Finkenstaedt-Quinn S. A. and Shultz G. V., (2019), Constraints on organic chemistry students’ reasoning during IR and ¹H NMR spectral interpretation, Chem. Educ. Res. Pract., 20(3), 522–541.
15. Connor M. C., Glass B. H. and Shultz G. V., (2021), Development of the NMR Lexical Representational Competence (NMR-LRC) Instrument As a Formative Assessment of Lexical Ability in ¹H NMR Spectroscopy, J. Chem. Educ., 98(9), 2786–2798.
16. Cooper M. M., Grove N., Underwood S. M. and Klymkowsky M. W., (2010), Lost in Lewis structures: an investigation of student difficulties in developing representational competence, J. Chem. Educ., 87(8), 869–874.
17. Deng J. M., Streja N. and Flynn A. B., (2021), Response Process Validity Evidence in Chemistry Education Research, J. Chem. Educ., 98, 3656–3666.
18. Dickmann T., Opfermann M., Dammann E., Lang M. and Rumann S., (2019), What you see is what you learn? The role of visual model comprehension for academic success in chemistry, Chem. Educ. Res. Pract., 20(4), 804–820.
19. diSessa A. A., (2004), Metarepresentation: native competence and targets for instruction, Cogn. Instruct., 22(3), 293–331.
20. diSessa A. A. and Sherin B. L., (2000), Meta-representation: an introduction, J. Math. Behav., 19, 385–398.
21. Farheen A. and Lewis S. E., (2021), The impact of representations of chemical bonding on students’ predictions of chemical properties, Chem. Educ. Res. Pract., 22(4), 1035–1053.
22. Farheen A., Martin N. and Lewis S. E., (2024), Student perceptions of partial charges and nucleophilicity/electrophilicity when provided with either a bond-line, ball-and-stick, or electrostatic potential map for molecular representation, Chem. Educ. Res. Pract., 25(1), 343–359.
23. Finney S. J. and DiStefano C., (2013), Nonnormal and categorical data in structural equation modeling, in Hancock G. R. and Mueller R. O. (ed.), Structural equation modeling: A second course, IAP Information Agre Publishing, pp. 439–492.
24. Gilbert J. K., (2005), Visualization: A Metacognitive Skill in Science and Science Education, Visualization in Science Education, Springer, pp. 9–27.
25. Glaser B. G., (1965), The Constant Comparative Method of Qualitative Analysis, Social Problems, 12(4), 436–445.
26. Gurung E., Jacob R., Bunch Z., Thompson B. and Popova M., (2022), Evaluating the Effectiveness of Organic Chemistry Textbooks for Promoting Representational Competence, J. Chem. Educ., 99, 2044–2054.
27. Halverson K. L. and Friedrichsen P., (2013), Learning tree thinking: developing a new framework of representational competence, Multiple Representations in Biological Education, vol. 7, pp. 185–201.
28. Hancock G. R. and An J., Scale Reliability in Structural Equation Modeling, https://ncme.elevate.commpartners.com/products/digital-module-2-scale-reliability-in-structural-equation-modeling (accessed July 2023) (accessed July 2023).
29. Harle M. and Towns M., (2011), A review of spatial ability literature, its connection to chemistry, and implications for instruction, J. Chem. Educ., 88(3), 351–360.
30. Herrington D. G. and Daubenmire P. L., (2014), Using interviews in CER projects: options, considerations, and limitations, in Bunce D. M. and Cole R. S. (ed.), Tools of Chemistry Education Research (ACS Symposium Series), vol. 1166, pp. 31–59.
31. Hinze S. R., Rapp D. N., Williamson V. M., Shultz M. J., Deslongchamps G. and Williamson K. C., (2013), Beyond ball-and-stick: students’ processing of novel STEM visualizations, Learn. Instruct., 26, 12–21.
32. Hiong S. J. and Daniel E. G. S., (2015), Same Substances, but They “See” Different Things: Student-Generated Representations as a Research Tool, J. Sci. Math. Educ. Southeast Asia, 38(1), 1–31.
33. Höst G. E., Schönborn K. J. and Palmerius K. E. L., (2012), Students use of three different visual representations to interpret whether molecules are polar or nonpolar, J. Chem. Educ., 89(12), 1499–1505.
34. Hu L. T. and Bentler P. M., (1999), Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives, Struct. Equat. Model., 6(1), 1–55.
35. Johnstone A. H., (1993), The development of chemistry teaching: a changing response to changing demand, J. Chem. Educ., 70(9), 701–705.
36. Jones T., Romanov A., Pratt J. M. and Popova, M., (2022), Multi-framework case study characterizing organic chemistry instructors’ approaches toward teaching about representations, Chem. Educ. Res. Pract., 23(4), 930–947.
37. Kohl P. B. and Finkelstein N. D., (2006), Effects of representation on students solving physics problems: a fine-grained characterization, Phys. Rev. Spec. Top.—Phys. Educ. Res., 2(1), 1–12.
38. Komperda R., Pentecost T. C. and Barbera J., (2018), Moving beyond Alpha: A Primer on Alternative Sources of Single-Administration Reliability Evidence for Quantitative Chemistry Education Research, J. Chem. Educ., 95(9), 1477–1491.
39. Kozma R., Chin E., Russell J. and Marx N., (2000), The Roles of Representations and Tools in the Chemistry Laboratory and Their Implications for Chemistry Learning, J. Learn. Sci., 9(2), 105–143.
40. Kozma R. B. and Russell J., (1997), Multimedia and Understanding: Expert and Novice Responses to Different Representations of Chemical Phenomena, J. Res. Sci. Teach., 34(9), 949–968.
41. Kozma R. and Russell J., (2005), Students Becoming Chemists: Developing Representational Competence, in Gilbert J. K. (ed.), Visualization in Science Education, Springer, pp. 121–145.
42. Lansangan R., Orleans A. and Camacho V. M., (2007), Representational competence of secondary chemistry students in understanding selected chemical principles, Int. J. Sci. Educ., 24(1), 7930–7934.
43. Lansangan R., Orleans A. and Camacho V. M., (2018), Assessing Conceptual Understanding in Chemistry Using Representation, Adv. Sci. Lett., 24(11), 7930–7934.
44. Lave J., (1991), Situated learning in community of practice, in Resnick L. B., Levine J. M. and Teasley S. D. (ed.), Perspectives on socially shared cognition, American Psychological Association, pp. 63–82.
45. Lave J. and Wenger E., (1991), Situated learning: Legitimate peripheral participation, Cambridge University Press.
46. Lazenby K., Tenney K., Marcroft T. A. and Komperda R., (2023), Practices in instrument use and development in chemistry education research and practice 2010–2021, Chem. Educ. Res. Pract., 24(3), 882–895.
47. Linenberger K. J. and Holme T. A., (2015), Biochemistry instructors’ views toward developing and assessing visual literacy in their courses, J. Chem. Educ., 92(1), 23–31.
48. Malkewitz C. P., Schwall P., Meesters C. and Hardt J., (2023), Estimating reliability: a comparison of Cronbach's α, McDonald's ω and the greatest lower bound, Soc. Sci. Human. Open, 7(1), 100368.
49. Maroo J. D. and Johnson S. L., (2018), The Use of a Representational Triplet Model as the Basis for the Evaluation of Students’ Representational Competence, in Daniel K. (ed.), Towards a Framework for Representational Competence in Science Education, Springer, pp. 247–262.
50. Mayer R. E., (2002), Cognitive theory and the design of multimedia instruction: an example of the two-way street between cognition and instruction, New Direct. Teach. Learn., 89, 55–71.
51. McNeish D., An J. and Hancock G. R., (2018), The Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models, J. Personality Assess., 100(1), 43–52.
52. McNeish D. and Wolf M. G., (2023), Dynamic fit index cutoffs for one-factor models, Behav. Res. Methods, 55(3), 1157–1174.
53. Miller K. and Kim T., (2017), Examining student heuristic usage in a hydrogen bonding assessment, Biochem. Mol. Biol. Educ., 45(5), 411–416.
54. Murphy K., Holme T., Zenisky A., Caruthers H. and Knaus K., (2012), Building the ACS exams anchoring concept content map for undergraduate chemistry, J. Chem. Educ., 89(6), 715–720.
55. Muthén L. K. and Muthén B. O., (2017), Mplus User's Guide, 8th edn, Muthén & Muthén.
56. National Research Council, (2012a), A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas, Committee on a Conceptual Framework for New K-12 Science Education Standards. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
57. National Research Council, (2012b), Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering, in Singer S. R., Nielse N. R. and Schweingruber H. A. (ed.), Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Board on Science Education, Division of Behavioral and Social Sciences and Education, Washington, DC: The National Academies Press.
58. Nitz S., Nerdel C. and Prechtl H., (2012), Modeling the Relationship between Representational Competence and Domain Knowledge, EARLI Conference SIGc, pp. 160–162.
59. Offerdahl E. G. and Arneson J. B., (2019), Formative Assessment to Improve Student Learning in Biochemistry, ACS Symposium Series: Biochemistry Education: From Theory to Practice, American Chemical Society, vol. 1337, pp. 197–218.
60. Padalkar S. and Hegarty M., (2012), Improving Representational Competence in Chemistry with Model-Based Feedback, Proc. Annu. Meet. Cogn. Sci. Soc., 34(34), 2162–2167.
61. Padalkar S. and Hegarty M., (2015), Models as feedback: developing representational competence in chemistry, J. Educ. Psychol., 107(2), 451–466.
62. Pande P. and Chandrasekharan S., (2022), Expertise as Sensorimotor Tuning: Perceptual Navigation Patterns Mark Representational Competence in Science, Res. Sci. Educ., 52(2), 725–747.
63. Pande P., Shah P. and Chandrasekharan S., (2015), How do experts and novices navigate chemistry representations-an eye-tracking investigation, Proc. EPISTEME-6, 102–109.
64. Patton M. Q., (2002), Qualitative Research and Evaluation Methods, 3rd edn, Sage Publications.
65. Popova M. and Bretz S. L., (2018a), “It's Only the Major Product That We Care About in Organic Chemistry”: An Analysis of Students’ Annotations of Reaction Coordinate Diagrams, J. Chem. Educ., 95(7), 1086–1093.
66. Popova M. and Bretz S. L., (2018b), Organic chemistry students’ challenges with coherence formation between reactions and reaction coordinate diagrams, Chem. Educ. Res. Pract., 19(3), 732–745.
67. Popova M. and Bretz S. L., (2018c), Organic chemistry students’ interpretations of the surface features of reaction coordinate diagrams, Chem. Educ. Res. Pract., 19(3), 919–931.
68. Popova M. and Jones T., (2021), Chemistry instructors’ intentions toward developing, teaching, and assessing student representational competence skills, Chem. Educ. Res. Pract., 2(3), 733–748.
69. Raker J. R. and Holme T. A., (2013), A historical analysis of the curriculum of organic chemistry using ACS exams as artifacts, J. Chem. Educ., 90(11), 1437–1442.
70. Rotich F., Ward L., Beck C. and Popova M., (2024), Attention is currency: how surface features of Lewis structures influence organic chemistry student reasoning about stability, Chem. Educ. Res. Practice, 25, 1071–1089.
71. Saldaña J., (2013), The coding manual for qualitative researchers, 2nd edn, Los Angeles: SAGE Publications.
72. Schönborn K. J. and Anderson T. R., (2008), A Model of Factors Determining Students’ Ability to Interpret External Representations in Biochemistry, Int. J. Sci. Educ., 31(2), 193–232.
73. Seufert T., (2003), Supporting coherence formation in learning from multiple representations, Learn. Instruct., 13(2), 227–237.
74. Sim J. H. and Daniel E. G. S., (2014), Representational competence in chemistry: a comparison between students with different levels of understanding of basic chemical concepts and chemical representations, Cogent Educ., 1(1), 1–17.
75. Stieff M. and DeSutter D., (2020), Sketching, not representational competence, predicts improved science learning, J. Res. Sci. Teach., 1–29.
76. Stieff M., Hegarty M. and Deslongchamps G., (2011), Identifying representational competence with multi-representational displays. Cogn. Instruct., 29(1), 123–145.
77. Stieff M., Lira M. and Desutter D., (2014), Representational Competence and Spatial Thinking in STEM Spatial Thinking in STEM, Proc. Int. Conf. Learn. Sci., ICLS, 2, 987–991.
78. Stieff M., Origenes A., DeSutter D., Lira M., Banevicius L., Tabang D. and Cabel G., (2018), Operational constraints on the mental rotation of STEM representations, J. Educ. Psychol., 110(8), 1160–1174.
79. Stieff M., Scopelitis S., Lira M. E. and Desutter D., (2016), Improving Representational Competence with Concrete Models, Sci. Educ., 100(2), 344–363.
80. Stull A. T., Gainer M., Padalkar S. and Hegarty M., (2016), Promoting Representational Competence with Molecular Models in Organic Chemistry, J. Chem. Educ., 93(6), 994–1001.
81. Stull A. T. and Hegarty M., (2016), Model manipulation and learning: fostering representational competence with virtual and concrete models, J. Educ. Psychol., 108(4), 509–527.
82. Taber K. S., (2018), The Use of Cronbach's Alpha When Developing and Reporting Research Instruments in Science Education, Res. Sci. Educ., 48(6), 1273–1296.
83. Talanquer V., (2014), Chemistry Education: ten heuristics to tame, J. Chem. Educ., 91(8), 1091–1097.
84. Talanquer V., (2022), The Complexity of Reasoning about and with Chemical Representations, JACS Au, 2(12), 2658–2669.
85. Towns M. H., (2008), Mixed Methods Designs in Chemical Education Research, in Bunce D. and Cole R. (ed.), Nuts and Bolts of Chemistry Education Research, American Chemical Society, vol. 15, pp. 135–148.
86. Towns M. H., (2014), Guide to developing high-quality, reliable, and valid multiple-choice assessments, J. Chem. Educ., 91(9), 1426–1431.
87. Vlacholia M., Vosniadou S., Roussos P., Salta K., Kazi S., Sigalas M. and Tzougraki C., (2017), Changes in visual/spatial and analytic strategy use in organic chemistry with the development of expertise, Chem. Educ. Res. Pract., 18(4), 763–773.
88. Wang Z., Chi S., Luo M., Yang Y. and Huang M., (2017), Development of an instrument to evaluate high school students’ chemical symbol representation abilities, Chem. Educ. Res. Pract., 18(4), 875–892.
89. Ward L. W., Rotich F., Hoang J. and Popova M., (2022), Representational Competence Under the Magnifying Glass—The Interplay Between Student Reasoning Skills, Conceptual Understanding, and the Nature of Representations, in Graulich N. and Shultz G. (ed.), Student Reasoning in Organic Chemistry: Research Advances and Evidence-based Instructional Practices, Royal Society of Chemistry, pp. 36–55.
90. Wright L. and Oliver-Hoyo M. T., (2020), Student assumptions and mental models encountered in IR spectroscopy instruction, Chem. Educ. Res. Pract., 21(1), 426–437.
91. Xia Y. and Yang Y., (2019), RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: the story they tell depends on the estimation methods, Behav. Res. Methods, 51(1), 409–428.

---

Footnote

† Electronic supplementary information (ESI) available. See DOI: https://doi.org/10.1039/d3rp00188a

---