Pedagogies of engagement use in postsecondary chemistry education in the United States: results from a national survey

Abstract

Pedagogies of engagement (i.e., Peer-Led Teaming Learning, Problem-Based Learning, and Process-Oriented Guided Inquiry Learning) are active learning approaches used in postsecondary chemistry courses. In this study, we use data from a national survey of postsecondary chemistry instructors in the United States to estimate use of three pedagogies in the course for which the instructor feels they have the most control. We found that 16.6% of these faculties report they are implementing Peer-Led Team Learning (PLTL), 10.6% report implementing Problem-Based Learning (PBL), and 10.7% report implementing Process-Oriented Guided-Inquiry Learning (POGIL). We compare use of select teaching practices and other active learning strategies with implementation of PLTL, PBL, and POGIL. Additionally, we use items from the survey to understand course, institution, and instructor characteristics associated with use. Key findings include that lower-level courses and courses with large enrollments are the most likely places for PLTL to be implemented and that instructors who are not on the tenure track are more likely to implement PLTL and POGIL than tenured/tenure-track instructors. Instructors who report implementing PLTL and PBL have more student-centered beliefs about teaching and learning, while instructors who report implementing POGIL have more teacher-centered beliefs about learning, albeit all with small effect sizes. Implications are offered for how instructors, researchers, developers and disseminators of these pedagogies can use our results to inform their practices and efforts.

---

Introduction

The phrases “pedagogies of engagement” and “pedagogies of engaged learning” are attributed to R. Edgerton in a white paper on education presented to the Pew Charitable Trusts’ Board of Directors in 1997. Edgerton (2001) argued that transformative instructional strategies that purposely, and without exception, engage students in learning are necessary to produce postsecondary graduates with the knowledge and skills to be “resourceful, engaged workers and citizens.” In 2008, Eberlein et al. (2008) operationalized Edgerton's instructional strategies as three named pedagogies in chemistry education: Problem-Based Learning (PBL; Savery, 2006), Process-Oriented Guided-Inquiry Learning (POGIL; Moog, 2014; Moog et al., 2006; Moog and Spencer, 2008), and Peer-Led Team Learning (PLTL; Gosser et al., 1996, 2010). Eberlein et al. (2008) argued that these “pedagogies of engagement” embrace the purpose (i.e., to promote higher order thinking skills, conceptual understanding, teamwork, and collaborative problem-solving skills) and theoretical underpinnings (i.e., constructivism from the perspectives of Piaget, Dewey, and Vygotsky) of Edgerton's call for educational transformation. Despite similarities between the pedagogies, each pedagogy takes a different approach to the locus of knowledge, processes for knowledge construction and skill development, and ideal contexts for implementation (Eberlein et al., 2008). Developers and disseminators of the pedagogies routinely investigate the effectiveness of the pedagogies for achieving learning goals (e.g., de Gale and Boisselle, 2015; Dochy et al., 2003; Gehrke and Kezar, 2016; Lewis and Lewis, 2005, 2008; Şen et al., 2015; Walker and Leary, 2009; Walker and Warfa, 2017). Missing from the literature, however, are estimates of prevalence of use. Estimates of frequency of implementation have been made by disseminators based on participation in workshops and conferences, instructional material sales, and self-identification through project websites or social media groups (e.g., The POGIL Project, 2019); these estimates, though, are likely understated given the data collection methods.

Through a national survey of postsecondary chemistry education faculty members in the United States, we collected data on the use of three pedagogies of engagement (i.e., PBL, POGIL, and PLTL). We chose these three pedagogies due to established literature on their theoretical and practical similarities (Eberlein et al., 2008). Our method for population definition, stratified sampling strategy, survey item set, and weighted-data analyses have positioned us to make well-reasoned estimates of pedagogies of engagement use not previously reported in the literature. These estimates, when considered in light of course-level (i.e., upper or lower), student enrollment in the course, instructional practices (e.g., lecturing, small group work, discussion of the primary literature), and instructor beliefs about teaching and learning, form a robust foundation from which disseminators can gauge the success of their pedagogies and identify opportunities for further dissemination efforts. This work is intended to provide a snapshot of implementation of pedagogies of engagement and serve as a baseline for future work in which the growth of pedagogical use can be characterized.

Research questions

This study is guided by two overarching research questions:

1. What percentage of instructors use PBL, POGIL, or PLTL in the undergraduate chemistry course for which they perceive they have the most control?

2. What is the association between use of the pedagogies of engagement and course, institution, and instructor characteristics?

Pedagogies of engagement

Pedagogies of engagement, as operationalized for our study, encompass three named pedagogies: Problem-Based Learning (PBL; Savery, 2006), Process-Oriented Guide-Inquiry Learning (POGIL; Moog, 2014; Moog et al., 2006; Moog and Spencer, 2008), and Peer-Led Team Learning (PLTL; Gosser et al., 1996, 2010). These student-centered learning approaches have shared and contrasting roles of the instructor in learning and learning activities enacted in the classroom.

Peer-led team learning

PLTL incorporates peer leaders (i.e., students who have recently passed the course) into lecture periods, weekly recitation/discussion sessions or supplemental instruction to guide small groups of students (typically six to ten students per group) in the course through completing practice problems (e.g., back-of-a-chapter problems or instructor-developed worksheets; Gosser et al., 2010; Wilson and Varma-Nelson, 2016). This pedagogy is rooted in Vygotsky's (1978) zone of proximal development, wherein the peer leaders are able to effectively assist students in navigating the gap between where the students currently are in their learning and the goals of the learning experience. PLTL has reportedly been used in a variety of course contexts, including courses enrolling few (n < 30) to large (n > 180) numbers of students (Wilson and Varma-Nelson, 2016), and implemented through in-person and cyber peer leading sessions (Smith et al., 2014; Wilson and Varma-Nelson, 2019). PLTL is not generally used as a replacement for lecture time; instead, PLTL supplements lecture courses by adding group work sessions (Eberlein et al., 2008). Research suggests that PLTL has a positive impact on student learning and development of affect, reasoning and critical thinking skills (Wilson and Varma-Nelson, 2016). PLTL has shown promise when used in combination with flipped classroom pedagogies (Robert et al., 2016).

Problem-based learning

The central tenet of PBL is that the pedagogy positions “learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem” (Savery, 2006). Thus, purposefully selected, ill-defined problems serve as a key driver for learning (Walker and Leary, 2009; Wood, 2003). PBL has origins in medical education, wherein the ‘problem’ that medical students are faced with is diagnosing a patient (Barrows, 1986; Savery, 2006). Meta-analyses suggest more positive outcomes using the problem-based learning pedagogy compared to traditional instructional methods in a variety of contexts and disciplines (Dochy et al., 2003; Walker and Leary, 2009). Problem-based learning activities situated in chemistry courses have been reported (Costantino and Barlocco, 2019; Nagarajan and Overton, 2019; Shultz and Zemke, 2019; Williams, 2017). Lecture is minimal when implementing the PBL pedagogy (Hmelo-Silver, 2004; Savery and Duffy, 1995). It is hypothesized that “basic science concepts will be understood and remembered longer when they are learned, discussed, and applied in a practical, real-world context” (Eberlein et al., 2008). Instructors (in some contexts including peer leaders and learning assistants) serve as facilitators and guides, helping small groups (which may be as large as 10 students) navigate the often complex and interdisciplinary problems (Wood, 2003); problem resolution (i.e., solving the problem) typically extends beyond a single class period and involves students gathering knowledge and data outside of the class period in pursuit of a problem solution (Amador et al., 2006). PBL has been reported as most frequently used in classes with 30 or fewer students (Amador et al., 2006; Eberlein et al., 2008). PBL activities have been characterized as analogous to scientific research and may include students engaging with primary literature (Hmelo-Silver, 2004).

Process-oriented guided-inquiry learning

The POGIL (Moog, 2014; Moog et al., 2006; Moog and Spencer, 2008) pedagogy is situated in groups of three to five students engaging with learning materials that are specifically designed to guide students through the learning cycle (i.e., exploration, concept invention, and application; Atkin and Karplus, 1962; Karplus, 1980). POGIL began in postsecondary chemistry, but its use has expanded to include a broad array of science and mathematics disciplines at secondary and postsecondary levels. In addition to content and skill development, POGIL activities are designed to promote process skills including problem solving, deductive reasoning, communication, and self-assessment (Eberlein et al., 2008; Moog, 2014; Moog et al., 2006; Moog and Spencer, 2008). Positive impacts on student learning in chemistry courses have been reported (e.g., de Gale and Boisselle, 2015; Gehrke and Kezar, 2016; Lewis and Lewis, 2008, 2005; Şen et al., 2015; Walker and Warfa, 2017). Instructors serve as facilitators of student-managed groups where students have roles such as manager, recorder, reflector, technician, or presenter; instructors give students minimal assistance, particularly when students should be challenged to consider their prior knowledge and skills in the context of the POGIL activity (The POGIL Project, 2011). Instructors can purchase POGIL activities developed and sold by the POGIL project or develop their own materials based on POGIL guiding principles. Occasionally, mini lectures lasting between fifteen and twenty minutes are offered. These lecture times are set within whole class discussions and small groups reporting out answers to key worksheet items (The POGIL Project, 2011). The POGIL pedagogy is typically used in classes enrolling 30 or fewer students (Eberlein et al., 2008); however, instructors have found means to utilize the pedagogy in classes with larger enrollments (Lewis and Lewis, 2005, 2008).

Factors associated with active learning pedagogy use

The survey used to gather data for this study is framed by the Teacher-Centered Systemic Reform (TCSR) model (Gess-Newsome et al., 2003; Woodbury and Gess-Newsome, 2002). The TCSR model has been used to evaluate the impact of a new faculty workshop, to understand the use of clickers and flipped classroom strategies, and to understand the enacted instructional practices in postsecondary chemistry education (Gibbons et al., 2017, 2018a, 2018b; Lund and Stains, 2015; Popova et al., 2020; Srinivasan et al., 2018; Stains et al., 2015). The TCSR model hypothesizes that three interrelated components influence enacted instructional practices: personal context, teacher thinking, and cultural context (Gess-Newsome et al., 2003; Woodbury and Gess-Newsome, 2002). Personal context includes participation in professional development activities, years teaching, and formal pedagogical training. Teacher thinking includes, for example, self-efficacy in enacting instructional practices and beliefs about how learning occurs. Cultural context includes, for example, number of students enrolled in the course and perceived institutional barriers to pedagogical reform. We have identified, and evaluated in this study, six factors that have been linked to choices to enact instructional practices in postsecondary chemistry education (Gibbons et al., 2017, 2018a, 2018b; Srinivasan et al., 2018): course level (i.e., lower or upper), student enrollment, classroom teaching practices (e.g., small group work, instructor moving through the classroom), active learning strategies (e.g., think-pair-share), beliefs about teaching and learning (i.e., student- or instructor-centered), and faculty rank. Hypothesized relationships between use of pedagogies of engagement and these factors are reported in Table 1; these relationships are derived from the research literature on the development and dissemination of the three pedagogies (Amador et al., 2006; Barrows, 1996; Savery, 2006) with particular attention paid to Table 1 from Eberlein et al. (2008) in which the key developers of these pedagogies outline ideal implementation contexts. These factors relate to the cultural context (i.e., course level, student enrollment), enacted practices (i.e., classroom teaching practices, active learning strategies), teacher thinking (i.e., beliefs about teaching and learning), and personal context (i.e., faculty rank).

Table 1 Hypothesized relationships between factors and pedagogies of engagement use

Factor	PLTL	PBL	POGIL
Course level	Used in lower-level courses	Used in upper-level courses	Used in lower-level courses
Student enrollment	Used in larger courses	Used in smaller courses	Used in smaller courses
Classroom teaching practices	(No associations are hypothesized)	Associated with small group work, less “lecture” time, questions and discussion, moving around the room, use of primary literature	Associated with small group work, less “lecture” time, questions and discussion, moving around the room
Active learning strategies	(No associations are hypothesized)	Used with teaching case studies	Used with just-in-time teaching
Beliefs about teaching and learning	High student-centered learning beliefs	High student-centered learning beliefs	High student-centered learning beliefs
	High self-efficacy beliefs	High self-efficacy beliefs	High self-efficacy beliefs
Faculty rank	Used by non-tenure-track faculty	Used by tenured or tenure-track faculty	Used by non-tenure-track faculty

---

Methods

A national survey of postsecondary chemistry faculty members in the United States was conducted in 2017; the study reported herein utilizes a portion of data collected from that survey. The population was defined using data from a national repository of degree completions in the United States. A stratified random sample was selected based on highest level of chemistry degree awarded at the institution and public or private control of the institution. Data were collected using online survey research software (i.e., Qualtrics) in accordance with application #Pro00025183 approved by the University of South Florida's Institutional Review Board on January 29, 2016. Nonresponse bias analyses were conducted to increase the trustworthiness of the results. Survey weights were assigned to each respondent, from which weighted-data analyses are conducted.

Population

The survey targeted chemistry faculty members at four-year postsecondary institutions in the United States that had awarded one or more bachelor's degrees in chemistry in the five years prior to the survey. Target institutions (n = 1137) met this criterion as reported in the Integrated Postsecondary Education Data System (National Center for Education Statistics, Department of Education, U.S. Government, 2016). Publicly available faculty member lists from these institutions were used to construct the target population list. Faculty lists were unavailable from 23 of the identified institutions. A total of 14 988 chemistry faculty members were identified at 1114 institutions (see Table 2). Faculty members were stratified based on the highest chemistry degree awarded at their institution (i.e., bachelor's or graduate) and institutional control (i.e., public or private); these strata were chosen based on prior studies of postsecondary chemistry faculty members. The strata represent semi-homogeneous groups from which proposed analyses will be conducted, for which expected response rates are known and thus better sampling strategies can be implemented (Gibbons et al., 2017, 2018a, 2018b; Srinivasan et al., 2018).

Table 2 Strata, sample definition, respondents, and response rates

Stratum	Inst. control	Highest chem. deg. awarded	n(institutions)	n(faculty)	n(sample)	W i	n(resp.)	URR (%)	W f
Note: Inst. = institution(s), Chem. = chemistry, Deg. = degree, Wi = initial weight, Resp. = respondents, URR = unit response rate, Wf = final weight.
1	Public	Bachelor's	241	2828	1231	2.30	203	16.5	13.9
2	Public	Graduate	223	6059	2263	2.68	260	11.5	23.3
3	Private	Bachelor's	569	4221	1084	3.89	216	19.9	19.5
4	Private	Graduate	81	1880	1810	1.04	150	8.30	12.5
Totals			1114	14 988	6388		829	14.4

---

Sampling strategy and response rate

A stratified sampling strategy was used to select 6388 faculty from the four strata (see Table 2). The number of faculty sampled for each stratum was based on a desired 95% confidence level and a 5% confidence interval. Projected response rates were based on a 2016 iteration of a similar survey (Gibbons et al., 2017). Initial weights (W_i) for each stratum were determined as the inverse of the probability of selection. A total of 829 faculty members responded to the survey, resulting in an overall unit response rate of 14.4% (National Center for Education Statistics, 2012).

Data collection

Data were collected using Qualtrics in Spring 2017. An invitation to participate was sent to potential respondents followed by two follow-up emails at one-week intervals sent to those who had not responded; the survey was open to collect responses for three weeks.

Nonresponse bias analysis

Based on the low overall unit response rate and guidelines from the National Center for Education Statistics (NCES, 2012), a nonresponse bias analysis was conducted. Non-response bias is error that can result when less than 100% of the population are sampled or do not respond to a survey; it represents the systematic and meaningful difference between respondents and non-respondents on a substantive variable measured by the survey (Halbesleben and Whitman, 2013; Taris and Schreurs, 2007).

First, response rates were compared across frame characteristics (i.e., institutional control and highest chemistry degree awarded by institution; NCES, 2012). Observed differences in response rates by frame characteristics can suggest non-response bias (Groves, 2006). A Chi-square test of independence was performed to examine the relationship between responses and institutional control (i.e., public or private); the relationship between these variables is not significant, χ² (1, N = 6388) = 0.51, p > 0.05. A Chi-square test of independence was performed to examine the relationship between response and highest chemistry degree awarded at the institution (i.e., bachelors versus graduate); the relationship between these variables is significant, χ² (1, N = 6388) = 84.34, p < 0.00001. These results suggest that response rates differed by highest chemistry degree awarded but not by institutional control.

Second, response patterns by each of the pedagogies of engagement were evaluated by early and late responders (i.e., those that responded after a reminder email). Differences in response patterns by early and late responders can also suggest non-response bias (Curtin et al., 2000; Olson, 2006). Chi-square tests of independence were performed to examine the relationship between early and late responders and use of PLTL, PBL, and POGIL. The relationship between early and late responders and PLTL is not significant, χ² (1, N = 828) = 0.06, p > 0.05, the relationship between early and late responders and PBL is not significant, χ² (1, N = 829) = 0.49, p > 0.05, and the relationship between early and late responders and POGIL is not significant, χ² (1, N = 826) = 2.64, p > 0.05. These results suggest minimal potential bias due to response patterns.

These studies of non-response bias suggest minimal to negligible bias when looking at institutional control, highest chemistry degree awarded by institution, and response patterns. Weighted data analyses were conducted to further minimize and negate any potential bias.

Weighted data analyses

Final weights (W_f), stratification, and a finite population correction were used in Stata 14 to conduct statistical analyses. Upper- and lower-bound (ub, lb) 95% confidence intervals (CIs) are reported for measures where appropriate. Weighted survey data analyses are used, including two-way cross-tabulations with tests of independence followed by weighted logistic regressions (and associated odds ratios) as unstandardized effect size measures (Chen et al., 2010), and weighted means analyses of variance (ANOVAs) followed by η² (eta-squared) effect size measures (Cohen, 1988).

Results and discussion

Descriptive statistics are presented for the overall use of each pedagogy followed by descriptive, inferential statistics, and corresponding effect sizes for the hypothesized factors associated with pedagogies of engagement use. Participants were asked to respond to the survey items in reference to a single undergraduate, non-laboratory course taught in the last three years for which they felt they had the most control; thus, results reflect the most ideal context in which the respondents are enacting instructional practices.

Research question 1: percentage of pedagogies of engagement use

Respondents were asked to indicate how often they used each of the three pedagogies of engagement (i.e., PLTL, PBL, and POGIL) in the course for which they perceived to have the most control over teaching methods, textbook, etc.; answer options were “every class period”, “weekly”, “several times per semester”, “rarely”, or “never”. Respondents were presented with descriptors of each of the pedagogies (see Table 3). For analyses, responses are condensed into two categories: “use” and “non-use”. “Use” is defined as a respondent selecting “every class period” or “weekly”. Developers and disseminators of pedagogies of engagement suggest that to “use” the pedagogy, a faculty member must be using the pedagogy on a regularly basis (Eberlein et al., 2008). We have interpreted this to mean at minimum weekly.

Table 3 Percent use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy – description presented to surveyed faculty	Use (%)	95% C.I.
		lb	ub
Peer-led team learning – in groups, students complete a worksheet with problems designed to build conceptual understanding and problem-solving skills. Small groups are led by peer leaders (i.e., trained undergraduate students that have taken the course).	16.6	14.2	19.4
Problem-based learning – acting primarily as a facilitator and placing students in self-directed teams to solve open-ended problems that require significant learning of new course material.	10.6	8.6	13.0
Process-oriented guided-inquiry learning – in groups, students complete a worksheet designed around the learning cycle.	10.8	8.8	13.2

---

Weighted estimates suggest that approximately 17% of chemistry instructors report using PLTL in the course for which they perceive they have the most control (see Table 3). Approximately 11% of chemistry instructors report using PBL and 11% report using POGIL in the course for which they perceive they have the most control. These are the first estimates based on survey data of the implementation of each of these pedagogies in postsecondary chemistry courses. These estimates could be lower than the actual number of courses using the pedagogies because instructors were specifically asked to respond about the course for which they felt they have the most control.

When use of multiple pedagogies of engagement are estimated simultaneously, approximately 5% of respondents report using two of the pedagogies simultaneously in their courses with PLTL most commonly paired with either PBL or POGIL. This is not unexpected, as the use of peer leaders and learning assistants, for example, has emerged as a method to incorporate active learning pedagogies such as PBL or POGIL into large enrollment courses (e.g., Lewis and Lewis, 2005, 2008).

Research question 2: association between pedagogies of engagement use and course, institution, and instructor characteristics

Course, institution, and instructor characteristics are hypothesized to be associated with use of pedagogies of engagement in postsecondary chemistry courses (see Table 1, above). Hypothetical associations were derived from theoretical and empirical associations reported in the literature of associations with enacted instructional practices (Gess-Newsome et al., 2003; Gibbons et al., 2017, 2018b; Lund and Stains, 2015; Srinivasan et al., 2018; Stains et al., 2015; Woodbury and Gess-Newsome, 2002).

Course level

Use of PLTL, PBL, and POGIL were estimated by course level (see Table 4). Course level was defined as “lower” if the respondent reported that the course was taught primarily for first- and second-year postsecondary students; course level was defined as “upper” if the respondent reported that the course was taught primarily for third- and fourth-year postsecondary students. Course level was cross-checked with course titles reported by respondents: general chemistry, organic chemistry, and general, organic, and biological chemistry (GOB) courses were classified as lower level. Analytical chemistry, biochemistry, and inorganic chemistry were classified as both lower- and upper-level courses; these courses do not have common placement across U.S. postsecondary chemistry contexts. Special topic and physical chemistry courses were classified as upper-level courses.

Table 4 Percent use of pedagogies of engagement in postsecondary chemistry courses by course level

Pedagogy	Use (%) [lb, ub]		F(df,df)	Odds ratio
	Lower-level courses	Upper-level courses
*p < 0.05, **p < 0.01, ***p < 0.001.
PLTL	21.1 [17.7, 24.9]	9.8 [7.0, 13.7]	17.19*** (1823)	2.44
PBL	11.4 [8.8, 14.6]	9.4 [6.6, 13.2]	0.80 (1824)	—
POGIL	12.3 [9.6, 15.5]	8.6 [6.0, 12.2]	2.72 (1821)	—

---

There is one significant difference in reported use by course level: the odds of using PLTL was 2.44 times higher in lower-level courses compared to upper-level courses. This confirms our hypothesis about the prevalence of PLTL use in lower-level courses (Eberlein et al., 2008). Our hypotheses about PBL (higher reported use in upper-level courses) and POGIL (higher reported use in lower-level course) were not confirmed; this may suggest, in part, that appropriate curricular materials have been developed to catalyze use of these pedagogies across the chemistry curriculum and not only in the lower level courses (Amador et al., 2006; Barrows, 1986; Savery, 2006; The POGIL Project, 2011).

Student enrollment

The mean number of total students enrolled in the course (see Table 5) and mean number of students enrolled in the respondent's sections of the course (see Table 6) were estimated by use and non-use of each of the pedagogies. This analysis was conducted to evaluate whether course size is associated with use of each of the pedagogies.

Table 5 Mean number of total students enrolled by use/non-use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy	Mean # of total students [lb, ub]		F(df,df)	η ^2
	Use	Non-use
*p < 0.05, **p < 0.01, ***p < 0.001.
PLTL	305 [234, 376]	174 [153, 196]	11.35*** (1822)	0.014
PBL	216 [150, 281]	194 [172, 217]	0.34 (1823)	—
POGIL	231 [160, 302]	189 [167, 211]	1.23 (1820)	—

---

Table 6 Mean number of students enrolled in the respondent's sections by use/non-use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy	Mean # of instructors’ students [lb, ub]		F(1822)	η ^2
	Use	Non-use
*p < 0.05, **p < 0.01, ***p < 0.001.
PLTL	128 [106, 151]	86 [77, 96]	9.96**	0.012
PBL	106 [73, 139]	92 [82, 101]	0.63	—
POGIL	85 [60, 111]	94 [85, 104]	0.40	—

---

Small effect sizes were observed for the mean difference in reported use and non-use of PLTL, with courses using PLTL having more total students enrolled, and students enrolled in the respondent's sections of the course. Coupled with the relationship been reported use and course-level, these results suggest that the proposed utility of PLTL as a means to ‘reduce class size’ by increasing the number of ‘instructors’ in the classroom in the form of peer leaders is being realized in practice (Robert et al., 2016). As lower-level courses tend to have higher enrollments, the prevalence of PLTL in both lower-level courses and courses with high enrollment is unsurprising.

Differences in course enrollment and reported use of PBL and POGIL were not significant. These results may suggest that instructors have found suitable strategies to overcome the scalability concern of implementing PBL and POGIL; indeed, there is a growing body of literature that suggests the pedagogies of engagement can and are being used in courses with a wide range of student enrollments (Amador et al., 2006; Moog and Spencer, 2008; The POGIL Project, 2011).

Teaching practices

Use of common teaching practices (n = 13) was considered by use of each of the pedagogies; these teaching practices mirror the Classroom Observation Protocol for Undergraduate STEM (COPUS; Lund et al., 2015; Smith et al., 2013; Stains et al., 2018) and this method has been shown to produce valid and reliable data as survey items (Gibbons et al., 2018b).

Percent reported use and non-use of common teaching practices for those respondents who use each pedagogy are reported (see Table 7).

Table 7 Percent use of common teaching practices by use/non-use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy	Use (%)		F(df,df)	Odds ratio
	Use	Non-use
*p < 0.05, **p < 0.01, ***p < 0.001.a Reverse odds ratio reported.
Lecturing
PLTL	97.5	97.7	0.03 (1822)	—
PBL	97.2	97.7	0.07 (1823)	—
POGIL	88.3	98.8	25.15*** (1820)	10.7^a
Writing on the board, using the projector, or document camera
PLTL	96.5	97.2	0.17 (1822)	—
PBL	95.8	97.2	0.49 (1823)	—
POGIL	94.8	97.3	1.47 (1820)	—
Posing questions for which you expect a student response
PLTL	99.5	98.2	1.52 (1822)	—
PBL	97.6	98.5	0.31 (1823)	—
POGIL	98.6	98.4	0.01 (1820)	—
Answering questions from students
PLTL	98.6	99.2	0.54 (1823)	—
PBL	100.0	99.0	—	—
POGIL	100.0	99.0	—	—
Asking clicker questions
PLTL	34.2	25.0	4.68* (1813)	1.55
PBL	32.3	25.8	1.53 (1814)	—
POGIL	34.8	25.5	3.31 (1811)	—
Assigning students to work in groups
PLTL	80.8	63.5	13.95*** (1822)	2.43
PBL	82.3	64.4	10.03** (1823)	2.55
POGIL	89.5	63.5	18.45*** (1820)	4.90
Moving through the class, guiding ongoing student work
PLTL	81.5	66.4	11.26*** (1824)	2.23
PBL	88.2	66.5	14.23*** (1825)	3.76
POGIL	96.1	65.6	23.10*** (1822)	13.08
Extended discussion with small groups or individuals
PLTL	73.6	49.0	25.26*** (1824)	2.90
PBL	77.2	50.2	19.29*** (1825)	3.36
POGIL	77.3	50.4	20.07*** (1822)	3.35
Showing or conducting a demonstration, experiment, simulation, video, or animation
PLTL	60.7	62.0	0.08 (1822)	—
PBL	67.2	61.2	1.15 (1823)	—
POGIL	65.7	61.4	0.58 (1819)	—
Asking students to make a prediction about the outcome of a demonstration or experiment before it is performed
PLTL	67.1	61.5	1.47 (1822)	—
PBL	72.7	61.1	4.03* (1823)	1.69
POGIL	59.2	63.0	0.45 (1821)	—
Referencing and discussing the primary literature
PLTL	56.3	52.0	0.85 (1824)	—
PBL	68.0	50.8	8.26** (1825)	2.05
POGIL	47.9	53.5	0.95 (1822)	—
Discussing the process by which a model, theory, or concept was developed
PLTL	77.9	83.2	2.14 (1824)	—
PBL	90.1	81.3	3.57 (1825)	—
POGIL	78.9	82.7	0.74 (1822)	—
Initiating a whole class discussion, including explanation, opinion, or judgement provided by students
PLTL	57.8	50.0	2.97 (1823)	—
PBL	70.0	48.7	12.17*** (1824)	2.41
POGIL	70.3	48.8	13.39*** (1821)	2.49

---

For the three pedagogies, greater than 88% of respondents use lecturing at minimum “weekly” in the course for which they have the most control; while pedagogies of engagement are designed to minimize lecturing, these results suggest that lecturing still occurs even when pedagogies of engagement are used. Note that a limitation of survey measures of instructional practices is accurate measures of the percent of time per class period that the instructional practice is used (Gibbons et al., 2018b; Stains et al., 2018); complimentary classroom observation studies suggest that while lecturing is still used in more engaged instruction, the percent of classroom time that lecturing is used is greatly reduced (Lund et al., 2015; Stains et al., 2018). Of the pedagogies, there was only one observed difference: Instructors that report not using POGIL were more likely to report “lecture” than instructors who report using POGIL (OR = 10.7). Minimization of lecture is a key tenet of POGIL; this result supports that lecture is minimized in instructional practice when POGIL is implemented.

For “writing on the board”, “posing questions”, and “answering questions”, no significant differences were observed between those who reported using and not using each of the pedagogies.

Instructors who reported using PLTL were more likely to report using “clickers” in their courses (OR = 1.55). No significant differences in use of clicker questions were observed between PBL and POGIL users and non-users. This result mirrors PLTL prevalence in large, lower-level courses in which increased clicker use has been reported (Gibbons et al., 2017).

The next three teaching practices (i.e., “work in groups”, “guiding ongoing student work”, and “extended discussions with small groups or individuals”) are considered essential features of pedagogies of engagement (Eberlein et al., 2008). As expected, users of each pedagogy were more likely to report engaging in these teaching practices than those who did not report using the pedagogies (ORs between 2.23 and 13.08).

In relation to teaching practices associated with engaging in science practice skills (i.e., “showing or conducting demonstrations”, “asking students to make a prediction”, “referencing and discussing the primary literature”, and “discussing the process by which a model was developed”), there were only observed significant differences between reported users and non-users of PBL. PBL users were more likely to report “asking students to make predictions” (OR = 1.69) and more likely to report “referencing and discussing the primary literature” (OR = 2.05) compared to PBL non-users; this reflects the PBL pedagogy being situated in the practice of science in which predictions and primary literature are integral (Hmelo-Silver, 2004; Savery, 2006; Savery and Duffy, 1995).

Finally, the odds of PBL and POGIL users reporting “initiating a whole class discussion” were higher than non-users (OR = 2.41 and 2.49, respectively). This teaching practice would be considered an essential feature of these pedagogies (Hmelo-Silver, 2004; Moog and Spencer, 2008; Savery, 2006; Savery and Duffy, 1995; The POGIL Project, 2011).

These results roughly support our hypotheses in Table 1 with notable differences in the observed relationship between clicker and PLTL use, and lack of relationship between lecture and PBL use. Additionally, the reported use of teaching practices that coincide with the pedagogies of engagement provides further trustworthiness to the survey method; the reporting of these strategies essential to the pedagogies in combination with reporting use of the pedagogies indicates the instructors are implementing the pedagogies in a somewhat similar way to what was intended by the distributors.

Active learning strategies

Our prior work suggests that instructors utilize an array of active learning strategies simultaneously in their courses (Gibbons et al., 2018b, 2017; Srinivasan et al., 2018). To further evaluate this claim, we consider the use of other active learning strategies in combination with each pedagogy of engagement (see Table 8).

Table 8 Percent use of other research-based instructional practices by use/non-use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy	Use (%)		F(df,df)	Odds ratio
	Use	Non-use
*p < 0.05, **p < 0.01, ***p < 0.001.
Think-pair-share
PLTL	52.3	36.4	11.49*** (1821)	1.92
PBL	61.8	36.5	18.43*** (1822)	2.81
POGIL	51.7	37.6	6.16* (1819)	1.78
Just-in-time teaching
PLTL	28.3	15.6	11.81*** (1821)	2.14
PBL	37.1	15.4	21.46*** (1822)	3.23
POGIL	23.9	17.1	2.36 (1819)	—
Teaching with case studies
PLTL	32.3	20.6	8.46** (1822)	1.84
PBL	38.0	20.7	12.12*** (1823)	2.35
POGIL	26.9	22.1	0.99 (1820)	—
Flipped classroom
PLTL	35.7	23.2	8.94** (1822)	1.84
PBL	47.6	22.6	22.58*** (1823)	3.11
POGIL	47.2	22.5	23.14*** (1820)	3.08

---

These results suggest that the pedagogies of engagement are implemented with other active learning strategies (Eberlein et al., 2008). The relationship between the pedagogies and reported think-pair-share use is in alignment with the student–student interaction-based strategies central to each of the pedagogies of engagement (ORs between 1.78 and 2.81; McTighe and Lyman, 1988; Rowe, 1986).

We hypothesized a relationship between just-in-time teaching (Formica et al., 2010; Marrs and Novak, 2004; Simkins and Maier, 2009) and reported POGIL use; however, significant differences were only found between reported PBL and PLTL use and just-in-time teaching (OR = 2.14 and 3.23, respectively). This may stem from the more structured curricular worksheets used in POGIL and the importance of student self-discovery versus “telling” that may be inferred to occur with just-in-time teaching.

A significant relationship between reported PBL use and teaching with case studies (Coleman et al., 1997; Herreid, 1994) was found (OR = 2.35). This supports the “problem-based” component of PBL that is situated in real-life problems. Unexpectedly, a significant relationship was observed between reported PLTL use and teaching with case studies (OR = 1.84); however, studies are emerging about the synergistic combination of PLTL and case studies (e.g., Hewlett, 2004).

Finally, we found significant relationships between reported flipped classroom use and the PLTL, PBL, and POGIL (ORs between 1.83 and 3.11). Flipped classroom pedagogies are not active learning strategies per se (Hamdan et al., 2013), but when coupled with other pedagogies, flipped classroom use is a means to freeing class period time for more active and engaging pedagogies (Srinivasan et al., 2018).

Beliefs about teaching and learning

Beliefs about teaching and learning have been shown to be associated with use of more active and engaging pedagogies and practices in the classroom (Gess-Newsome et al., 2003; Woodbury and Gess-Newsome, 2002). Mean student-centered and teacher-centered belief scores and pedagogy self-efficacy and content self-efficacy scores were evaluated for PLTL, PBL, and POGIL (see Table 9) using the teaching beliefs and self-efficacy instrument reported by Gibbons et al. (2018b). PLTL users were more student-centered than non-users with a small effect size. There was a significant difference on the student-centered scale between PBL users and non-users; however, the effect size was negligible. Users of POGIL held less teacher-centered beliefs than non-users of POGIL with a small effect size. There was a significant difference on the pedagogy self-efficacy scale between POGIL users and non-users; however, the effect size was negligible. These results, however, mirror small and negligible effect sizes when used to evaluate the differences between enacted instructional practices and beliefs about learning using the same instrument (Gibbons et al., 2018b).

Table 9 Mean beliefs about teaching and learning scores by use/non-use of pedagogies of engagement in postsecondary chemistry courses

Pedagogy	Mean score [lb, ub]		F(df,df)	η ^2
	Non-use	Use
*p < 0.05, **p < 0.01, ***p < 0.001.
Peer-led team learning
Student-centered	1.91 [1.84, 1.98]	2.06 [2.03, 2.09]	13.62*** (1824)	0.016
Teacher-centered	2.14 [2.05, 2.24]	2.17 [2.13, 2.22]	0.38 (1824)	—
Pedagogy self-efficacy	1.85 [1.73, 1.96]	1.93 [1.88, 1.97]	1.86 (1823)	—
Content self-efficacy	1.68 [1.58, 1.77]	1.74 [1.70, 1.79]	1.62 (1823)	—
Problem-based learning
Student-centered	1.95 [1.85, 2.04]	2.04 [2.01, 2.07]	3.90* (1825)	0.005
Teacher-centered	2.27 [2.14, 2.39]	2.16 [2.12, 2.20]	2.84 (1825)	—
Pedagogy self-efficacy	1.84 [1.70, 1.98]	1.92 [1.88, 1.96]	1.34 (1824)	—
Content self-efficacy	1.66 [1.52, 1.79]	1.74 [1.70, 1.78]	1.43 (1824)	—
Process-oriented guided-inquiry learning
Student-centered	1.97 [1.89, 2.05]	2.04 [2.01, 2.07]	2.84 (1825)	—
Teacher-centered	2.37 [2.23, 2.51]	2.15 [2.11, 2.19]	9.37** (1822)	0.011
Pedagogy self-efficacy	1.80 [1.68, 1.92]	1.92 [1.88, 1.97]	3.87* (1821)	0.005
Content self-efficacy	1.79 [1.66, 1.91]	1.73 [1.68, 1.77]	0.87 (1821)	—

---

Faculty rank

Tenure status can be used as a proxy for academic title and the percentage of time the faculty member spends on teaching. Landrum et al. (2017) and Shadle et al. (2017), for example, have found associations between tenure status and adoption of evidence-based instructional practices; likewise, Lund and Stains (2015) found that tenure and promotion policies influenced enacted instructional practices in postsecondary chemistry courses. Percent report use of each pedagogy by tenure status is reported in Table 10.

Table 10 Percent use of pedagogies of engagement in postsecondary chemistry courses by tenure status of the instructor

Pedagogy	Use (%) [lb, ub]				F(df,df)
	Tenured	On tenure-track	Not on tenure track	No tenure system
*p < 0.05, **p < 0.01, ***p < 0.001.
PLTL	14.4 [11.5, 17.8]	16.8 [11.3, 24.2]	25.8 [19.4, 33.4]	6.0 [1.5, 21.4]	4.71** (3822)
PBL	9.9 [7.5, 12.9]	12.6 [7.8, 19.6]	12.2 [7.8, 18.5]	4.5 [0.6, 25.6]	0.68 (3823)
POGIL	8.6 [6.4, 11.4]	12.6 [7.9, 19.5]	16.0 [10.8, 23.0]	16.8 [7.0, 35.3]	2.58* (3820)

---

Significant differences were observed for PLTL use with respondents “not on a tenure track” being more likely to report using PLTL than “tenured” respondents (OR = 2.07) and respondents at an institution with “no tenure system” (OR = 5.41). No differences were observed for PBL use. Significant differences were observed for POGIL use with respondents “not on a tenure track” more likely to report using POGIL than “tenured” respondents (OR = 2.03). These results support the idea the faculty members who have teaching as a larger part of their responsibilities are more likely to implement active learning strategies; however, over interpretation of these results is cautioned, as differences were not observed between all tenure status groups suggesting a more complex association between the measures. The results could also reflect that non-tenure-track faculty are more likely to be teaching the lower-level courses in which we found PLTL to be more prevalent in.

Limitations

Members of the author team have previously discussed two key limitations of our survey research studies on enacted instructional practices in the context of considering flipped-classroom use in postsecondary chemistry courses (Srinivasan et al., 2018): namely, our work is focused on “what” is occurring rather than “how” or “why.” Any speculation in our discussions as to why a respondent is implementing a particular pedagogy of engagement is inferred from the association/correlation studies and not from direct observation or probing for further explanation from the respondent. Secondly, our work is focused on a population of chemistry faculty members in the United States. Pedagogy of engagement use should be cautiously generalized to non-U.S. contexts. As previously stated by members of the author team (Srinivasan et al., 2018), studies in non-U.S. contexts corroborate associations between instructional practices and the personal context, cultural context, and teacher thinking constructs addressed herein. Therefore, we do argue that those results have a broader applicability to non-U.S. contexts.

We offer an additional limitation to our work: the balance between broad and specific measures of enacted instructional practices. Our estimates are based on self-reported data. While it can be claimed that such data are invalid and unreliable, our prior work has shown that highly similar findings come from such self-reported data and classroom observations suggesting that survey study measures can be trusted (Gibbons et al., 2018b). The relative costs, in both time and resources, are vastly different for broad survey-based studies versus specific observational studies with the latter often coupled with time-intensive semi-structured interviews. Cost restraints become prohibitive when conceptualizing an observation study with a random sample of postsecondary chemistry faculty members. We recognize that our work is limited in characterizing how much of class time is devoted to each of the pedagogies and to what extent fidelity of implementation is held. Similarly, observational studies are limited in making generalizable claims as to the frequency of observed enacted instructional practices in the broader population of chemistry courses.

Implications

The results of this survey have implications for instructors, researchers, developers, and disseminators of pedagogies of engagement. While the specifics of this work are targeted at use of PBL, PLTL, and POGIL, the work has broader impacts when considering development, implementation, and dissemination of other active learning and innovative pedagogies.

For instructors

Having a reference for how prevalently a pedagogy is used and in what contexts can be meaningful data when considering adopting or continuing to implement the pedagogy. We estimate that one in ten postsecondary chemistry faculty are implementing PBL and one in ten postsecondary faculty are implementing POGIL in the course for which they have the most control. We estimate that one in seven are implementing PLTL. While implementation of any one of these pedagogies may be rare on a given campus, there are many faculty members across the United States implementing the pedagogies in postsecondary chemistry courses.

Similarly, having knowledge of the contexts in which pedagogies are being implemented can be meaningful data for consideration. The pedagogies are reported being used at all course levels with PLTL being more prevalent in lower-level courses, a finding that is confounded with PLTL being used in larger enrollment courses. Thus, PLTL may be a means for considering which of these three pedagogies to implement in general chemistry or organic chemistry courses (i.e., lower level courses) that have large student enrollment. Adoption of any of the three pedagogies is associated with increased small group work and the instructor moving through the classroom space facilitating learning of individual students or small groups of students. From a utility perspective, it is known that flipped classrooms provide a means to “present” content before class which allows class time to include more active learning; an instructor can note that others are using flipped classroom strategies in conjunction with PLTL, PBL, or POGIL as a potential means for ensuring that “content presentation” has occurred.

For researchers

The results we have presented are the first instance of rigorous survey research methods being applied to measuring the use of PLTL, PBL, and POGIL in postsecondary chemistry courses. Furthermore, outside of work published by authors of this manuscript (Gibbons et al., 2017, 2018a, 2018b; Srinivasan et al., 2018), we are unable to point to similar studies that use a well-defined population and weighted estimates of such measures for the population. The resulting confidence intervals on our estimates provide a clearer understanding of our measures for the population, and thus we do not have to limit our estimates to the sample, while guessing at how our estimates extrapolate to the population. The use of rigorous survey research methodologies provides means for making more definitive declarations in our results.

While we acknowledge in the Limitations section that self-reported data can be flawed, internal measures of validity provide assurances that the measures are trustworthy. Results on how teaching practices (akin to the COPUS tool) and use of other active learning strategies (i.e., think-pair-share, just-in-time teaching, teaching with case studies, flipped classroom) matched our hypotheses based on the critical components of each pedagogies. Therefore, those data are evidence to support the validity of our measures of pedagogy use and enacted teaching practices. Despite our confidence in these measures, specifics on implementation, particularly fidelity of implementation, remain elusive in our pursuit to characterize enacted instructional practices in postsecondary chemistry courses. More work is needed to develop suitable survey items to capture valid and reliable data to measure fidelity of implementation.

Estimates of PLTL, PBL, and POGIL use are restricted to the course in which the respondent had the most control over instructional practices, textbooks, assessments, etc.; thus, we cannot extrapolate our findings to all postsecondary chemistry courses. While such an estimate may be of interest, population definition (i.e., counting how many total postsecondary chemistry courses are offered in a given academic term) is a methodological hurdle that may not be worth the necessary resources. Our estimates are foundational in considering the choices faculty members make when teaching their courses and are made in a context for which the respondent has the most choice to enact teaching practices in their most ideal way. Thus, we have results that leave open the question: why do faculty members do what they do in the classroom?

For PLTL in particular, we were able to answer such a question by weaving together data and results about how the pedagogy has utility in large, lower-level courses taught by faculty members with a teaching mission and is associated with high use of classroom response systems (also, a teaching practice associated with large, lower-level courses). We have argued previously that some teaching practices find utility in a niche context (Gibbons et al., 2018b); our results on PLTL use may be pointing towards a niche context akin to classroom response systems. A similar argument was not discernable for PBL and POGIL. Despite beginning to explain why instructors use PLTL, we have many questions about “why” the pedagogies are adopted that will require further exploration. Our results serve as a foundation for developing targeted investigations into the specific processes and rationales utilized by instructors when deciding how to teach their courses.

Lastly, our work herein and previously reported has focused on the United States due to our familiarity with the postsecondary chemistry education system and access to necessary degree completion data, institutional characteristics, and faculty member contact lists from which to implement our survey research methodologies. There is an opportunity, thus, for researchers to conduct similar studies within other national, continental, or international postsecondary chemistry education systems to make similar or different measures about enacted instruction.

For pedagogy developers and disseminators

We expect that developers and disseminators of the three named pedagogies will want to utilize the results presented in communicating to their stakeholders and users the importance of their work and in recruiting new faculty members to adopt their pedagogies. Knowing that one in seven or one in ten of faculty members in the United States report using their pedagogy in the course for which they have the most control suggests that dissemination efforts have been successful. Additionally, knowing about course-level and enrollment levels for courses in which the pedagogies have been implemented further provides data to use when convincing others of the utility of the pedagogies. Lastly, respondents who reported not being in a tenure-track position reported higher levels of PLTL and POGIL use than those that were tenured. For PLTL, this is confounded with PLTL use in lower-level courses typically taught by more non-tenure track faculty members. This suggests an opportunity for disseminators to target faculty who have teaching as their primary responsibilities (i.e., those typically in non-tenure track positions).

Finally, we expect that developers and disseminators will ask for additional measures of use at future time points from which to argue for growth or decline in adoption of their pedagogy. Given the lack of measures of teaching practice in postsecondary chemistry courses, there are no baseline data from which to make arguments about how efforts to meaningfully impact science education are occurring.

Conclusion

PLTL, PBL, and POGIL are being implemented respectively by 16.6%, 10.6%, and 10.7% of chemistry faculty members in the United States in the course for which they have the most control over; this is the first reported measure of adoption of use of the pedagogies of engagement using stratified sampling methods and weighted data analysis. Associations of reported use with course level, student enrollment, teaching practices, active learning strategies, beliefs about teaching and learning, and faculty rank provided a more nuanced understanding of the contexts in which the pedagogies of engagement are enacted. These results have implications for instructors using or deciding to use the pedagogies, researchers interested in further understanding of how to make similar measures or understanding faculty choices to enact particular pedagogies, and, finally, developers and disseminators of the three pedagogies. The results presented create a baseline from which further dissemination efforts can be measured.

Conflicts of interest

This material is based upon support from the ACS Examinations Institute and the University of South Florida. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the ACS Examinations Institute or the University of South Florida. Jeffrey R. Raker is the Associate Director of the ACS Examinations Institute; the University of South Florida's Institutional Review Board has reviewed this study and deemed that Dr Raker's role with the ACS Examinations Institute is not likely to affect the safety of study participants or the scientific quality of the study. Kristen L. Murphy is the Director of the ACS Examinations Institute.

Acknowledgements

We would like to thank the 829 chemistry faculty members who graciously gave of their time to complete the survey. We would also like to thank Marilyne Stains (University of Virginia) for assisting us with the development of the survey items used in this study.

References

1. Amador J. A., Miles L., Peters C. B., (2006), The practice of problem based learning, Wiley.
2. Atkin J. M., Karplus R., (1962), Discovery or invention? Sci. Teach., 29, 45–51.
3. Barrows H. S., (1986), A taxonomy of problem-based learning methods, Med. Educ., 20, 481–486.
4. Barrows H. S., (1996), Problem-based learning in medicine and beyond: a brief overview. New Dir. Teach. Learn., 68, 3–12.
5. Chen H., Cohen P., Chen S., (2010), How big is a big odds ratio? Interpreting the magnitudes of odds ratios in epidemiological studies, Commun. Stat. Simul. Comput., 39, 860–864.
6. Cohen J., (1988), Statistical Power Analysis for the Behavioral Sciences, Lawrence Earlbaum, Hillsdale, N.J.
7. Coleman E. B., Brown A. L., Rivkin I. D., (1997), The effect of instructional explanations on learning from scientific texts, J. Learn. Sci., 6, 347–365.
8. Costantino L., Barlocco D., (2019), Teaching an undergraduate organic chemistry laboratory course with a tailored problem-based learning approach, J. Chem. Educ., 96, 888–894.
9. Curtin R., Presser S., Singer E., (2000), The effects of response rate changes on the index of consumer sentiment, Public Opin. Q., 64, 413–428.
10. de Gale S., Boisselle L. N., (2015), The effect of POGIL on academic performance and academic confidence, Sci. Educ. Int., 26, 56–61.
11. Dochy F., Segers M., Van den Bossche P., Gijbels D., (2003), Effects of problem-based learning: a meta-analysis, Learn. Instruct., 13, 533–568.
12. Eberlein T., Kampmeier J., Minderhout V., Moog R. S., Platt T., Varma-Nelson P., White H. B., (2008), Pedagogies of engagement in science: a comparison of PBL, POGIL, and PLTL, Biochem. Mol. Biol. Educ., 36, 262–273.
13. Edgerton R., (2001), Education White Paper.
14. Formica S. P., Easley J. L., Spraker M. C., (2010), Transforming common-sense beliefs into Newtonian thinking through Just-In-Time Teaching, Phys. Rev. Spec. Top. – Phys. Educ., 6, 020106.
15. Gehrke S., Kezar A., (2016), STEM reform outcomes through communities of transformation, Change, 48, 30–38.
16. Gess-Newsome J., Southerland S. A., Johnston A., Woodbury S., (2003), Educational reform, personal practical theories, and dissatisfaction: The anatomy of change in college science teaching, Am. Educ. Res. J., 40, 731–767.
17. Gibbons R. E., Laga E. E., Leon J., Villafañe S. M., Stains M., Murphy K., Raker J. R., (2017), Chasm crossed? Clicker use in postsecondary chemistry education, J. Chem. Educ., 94, 549–557.
18. Gibbons R. E., Reed J. J., Srinivasan S., Villafañe S. M., Laga E., Vega J., Murphy K. L., Raker J. R., (2018a), Assessment in postsecondary chemistry education: a comparison of course types, Assessment Update, 30, 1–16.
19. Gibbons R. E., Villafañe S. M., Stains M., Murphy K. L., Raker J. R., (2018b), Beliefs about learning and enacted instructional practices: An investigation in postsecondary chemistry education, J. Res. Sci. Teach., 55, 1111–1133.
20. Gosser D., Roth V., Gafney L., Kampmeier J., Strozak V., Varma-Nelson P., Radel S., Weiner M., (1996), Workshop chemistry: Overcoming the barriers to student success, Chem. Educ., 1, 1–17.
21. Gosser D. K., Kampmeier J. A., Varma-Nelson P., (2010), Peer-Led Team Learning: 2008 James Flack Norris Award Address, J. Chem. Educ., 87, 374–380.
22. Groves R. M., (2006), Nonresponse rates and nonresponse bias in household surveys, Public Opin. Q., 70, 646–675.
23. Halbesleben J. R. B., Whitman M. V., (2013), Evaluating survey quality in health services research: A decision framework for assessing nonresponse bias, Health Serv. Res., 48, 913–930.
24. Hamdan N., McKnight P., McKnight K., Arfstrom K. M., (2013), The flipped learning model: A white paper based on the literature review titled a review of flipped learning, Flipped Learning Network/Pearson/George Mason University.
25. Herreid C. F., (1994), Case studies in science – A novel method of science education. J. Coll. Sci. Teach., 23, 221–229.
26. Hewlett J. A., (2004), In search of synergy: Combining peer-led team learning with the case study method, J. Coll. Sci. Teach., 33, 28–31.
27. Hmelo-Silver C. E., (2004), Problem-based learning: What and how do students learn? Educ. Psychol. Rev., 16, 235–266.
28. Karplus R., (1980), Teaching for the development of proportional reasoning, Res. Sci. Educ., 10, 1–10.
29. Landrum R. E., Viskupic K., Shadle S. E., Bullock D., (2017), Assessing the STEM landscape: The current instructional climate survey and the evidence-based instructional practices adoption scale, Int. J. STEM Educ., 4, 25.
30. Lewis S. E., Lewis J. E., (2005), Departing from lectures: An evaluation of a Peer-Led Guided Inquiry alternative. J. Chem. Educ., 82, 135–139.
31. Lewis S. E., Lewis J. E., (2008), Seeking effectiveness and equity in a large college chemistry course: an HLM investigation of Peer-Led Guided Inquiry, J. Res. Sci. Teach., 45, 794–811.
32. Lund T. J., Pilarz M., Velasco J. B., Chakraverty D., Rosploch K., Undersander M., Stains M., (2015), The best of both worlds: Building on the COPUS and RTOP observation protocols to easily and reliably measure various levels of reformed instructional practice, CBE – Life Sci. Educ., 14:ar18, 1–12.
33. Lund T. J., Stains M., (2015), The importance of context: an exploration of factors influencing the adoption of student-centered teaching among chemistry, biology, and physics faculty, Int. J. STEM Educ., 2, 13.
34. Marrs K. A., Novak G., (2004), Just-in-time teaching in biology: Creating an active learner classroom using the internet, Cell Biol. Educ., 3, 49–61.
35. McTighe J., Lyman F. T., (1988), Cueing thinking in the classroom: The promise of theory-embedded tools, Educational Leadership, 18–24.
36. Moog R. S., (2014), Process Oriented Guided Inquiry Learning, In: McDaniel M. A., Frey R. F., Fitzpatrick S. M., Roediger H. L., (ed.), Integrating Cognitive Science with Innovative Teaching in STEM Disciplines, pp. 147–166.
37. Moog R. S., Creegan F. J., Hanson D. M., Spencer J. N., Straumanis A. R., (2006), Process-Oriented Guided Inquiry learning: POGIL and the POGIL Project. https://journals.iupui.edu/index.php/muj/article/download/20287/19880/0.
38. Moog R. S., Spencer J. N., (2008), Process-Oriented Guided Inquiry Learning: ACS Symposium Series 994, American Chemical Society.
39. Nagarajan S., Overton T., (2019), Promoting systems thinking using project- and problem-based learning, J. Chem. Educ., 96, 2901–2909.
40. National Center for Education Statistics, (2012), 2012 Revision of NCES Statistical Standards: Final, Institute of Education Sciences, Department of Education, U.S. Government.
41. National Center for Education Statistics, Department of Education, U.S. Government, (2016), Integrated Postsecondary Education Data System, https://nces.ed.gov/ipeds/.
42. Olson K., (2006), Survey participation, nonresponse bias, measurement error bias, and total bias, Public Opin. Q., 70, 737–758.
43. Popova M., Shi L., Harshman J., Kraft A., Stains M., (2020), Untangling a complex relationship: teaching beliefs and instructional practices of assistant chemistry faculty at research-intensive institutions, Chem. Educ. Res. Pract., 21, 513–527.
44. Robert J., Lewis S. E., Oueini R., Mapugay A., (2016), Coordinated implementation and evaluation of flipped classes and Peer-Led Team Learning in general chemistry, J. Chem. Educ., 93, 1993–1998.
45. Rowe M. B., (1986), Wait time: Slowing down may be a way of speeding up! J. Teach. Educ., 37, 43–50.
46. Savery J. R., (2006), Overview of problem-based learning: Definitions and distinctions, Interdisciplinary Journal of Problem-Based Learning, 1, 9–20.
47. Savery J. R., Duffy T. M., (1995), Problem based learning: An instructional model and its constructivist framework, Educ. Technol., 35, 31–38.
48. Şen Ş., Yılmaz A., Geban Ö., (2015), The effects of process oriented guided inquiry learning environment on students’ self-regulated learning skills, Problems of Education in the 21st Century, vol. 66, pp. 54–66.
49. Shadle S. E., Marker A., Earl B., (2017), Faculty drivers and barriers: laying the groundwork for undergraduate STEM education reform in academic departments. Int. J. STEM Educ., 4, 8.
50. Shultz G. V., Zemke J. M., (2019), “I wanna just Google it and find the answer”: Student information searching in a problem-based inorganic chemistry laboratory experiment. J. Chem. Educ., 96, 618–628.
51. Simkins S., Maier M., (ed.), (2009), Just-in-time teaching across the disciplines and across the academy, Stylus, Sterling, VA.
52. Smith J., Wilson S. B., Banks J., Zhu L., Varma-Nelson P., (2014), Replicating Peer-Led Team Learning in cyberspace: Research, opportunities, and challenges, J. Res. Sci. Teach., 51, 714–740.
53. Smith M. K., Jones F. H. M., Gilbert S. L., Wieman C. E., (2013), The Classroom Observation Protocol for Undergraduate STEM (COPUS): A new instrument to characterize university STEM classroom practices, CBE – Life Sci. Educ., 12, 618–627.
54. Srinivasan S., Gibbons R. E., Murphy K. L., Raker J. R., (2018), Flipped classroom use in chemistry education: results from a survey of postsecondary faculty members. Chem. Educ. Res. Pract., 19, 1307–1318.
55. Stains M., Harshman J., Barker M. K., Chasteen S. V., Cole R., DeChenne-Peters S. E., Eagan M. K., Esson J. M., Knight J. K., Laski F. A., Levis-Fitzgerald M., Lee C. J., Lo S. M., McDonnell L. M., McKay T. A., Michelotti N., Musgrove A., Palmer M. S., Plank K. M., Rodela T. M., Sanders E. R., Shimpf N. G., Schulte P. M., Smith M. K., Stetzer M., Van Valkenburgh B., Vinson E., Weir L. K., Wendel P. J., Wheeler L. B., Young A. M., (2018), Anatomy of STEM teaching in North American universities, Science, 359, 1468–1470.
56. Stains M., Pilarz M., Chakraverty D., (2015), Short and long-term impacts of the Cottrell Scholars Collaborative New Faculty Workshop. J. Chem. Educ., 92, 1466–1476.
57. Taris T. W., Schreurs P. J. G., (2007), How may nonresponse affect findings in organizational surveys? The tendency-to-the-positive effect. Int. J. Stress Manage., 14, 249–259.
58. The POGIL Project, (2019), 2019 Community Report. POGIL. https://pogil.org/community-report.
59. The POGIL Project, (2011), POGIL: Implementation Guide. POGIL. https://pogil.org/uploads/attachments/cjay281cc08qzw0x4ha9nt7wd-implementationguide.pdf.
60. Vygotsky L. S., (1978), Mind in society: The development of higher psychological processes, Harvard University Press, Cambridge, Massachusetts.
61. Walker A., Leary H., (2009), A problem based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3, 12–43.
62. Walker L., Warfa A.-R. M., (2017), Process Oriented Guided Inquiry Learning (POGIL®) marginally effects student achievement measures but substantially increases the odds of passing a course. PLoS One, 12, e0186203.
63. Williams D. P., (2017), Learn on the move: A problem-based induction activity for new university chemistry students. J. Chem. Educ., 94, 1925–1928.
64. Wilson S. B., Varma-Nelson P., (2019), Characterization of first-semester organic chemistry Peer-Led Team Learning and Cyber Peer-Led Team Learning students’ use and explanation of electron-Pushing Formalism. J. Chem. Educ., 96, 25–34.
65. Wilson S. B., Varma-Nelson P., (2016), Small groups, significant impact: A review of Peer-Led Team Learning research with implications for STEM education researchers and faculty. J. Chem. Educ., 93, 1686–1702.
66. Wood D. F., (2003), ABC of learning and teaching in medicine: Problem based learning. BMJ, 326, 328–330.
67. Woodbury S., Gess-Newsome J., (2002), Overcoming the paradox of change without difference: A model of change in the arena of fundamental school reform. Educ. Policy, 16, 763–782.

---