Designing performance-based measures to assess the scientific thinking skills of chemistry undergraduate researchers

Abstract

Undergraduate research (UR) is a vetted educational tool that is commonly perceived to prepare students for entering graduate school and careers in STEM fields; however, scholarly attention to date has largely relied on self-report data, which may limit inferences about the causal effects on student outcomes. In light of this, recent calls have been made for innovative and rigorous assessment strategies to better understand the efficacy and impact of UR on key disciplinary skills, both in classroom and internship UR models, that can help inform decisions about educational refinement. To more accurately measure the effect of UR on students, well-designed performance-based assessments can be used to provide direct evidence to the development of targeted skills during their research experience. Given the limited availability of tested, adaptable (and freely available) performance measures for assessing undergraduate chemistry students' scientific thinking skills, this article outlines a five-step process drawn from the literature about how reliable tasks and rubrics can be developed by faculty interested in assessing the effect of research training in the lab and classroom. For this purpose, as an applied example, the manuscript describes the development, testing, and validation of the Performance assessment of Undergraduate Research Experiences (PURE) instrument, which was designed to directly characterize the effects of research experiences on chemistry students' analytical and data-driven decision-making through open-response tasks situated in real-world scientific problems. Initial results reveal that the PURE instrument has high face validity and good reliability in measuring the scientific thinking skills of chemistry student researchers, and documents differences in UR students' answer quality over time supporting the effect of UR on research skill growth and the viability of performance data to assess these changes.