View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/C5RP00190K (Paper) Chem. Educ. Res. Pract., 2016, 17, 700-710

The development of a tool for measuring graduate students' topic specific pedagogical content knowledge of thin layer chromatography

L. V. A. Hale , J. C. Lutter and G. V. Shultz *
University of Michigan, Department of Chemistry, 930 N. University, Ann Arbor, MI 48109, USA. E-mail: gshultz@umich.edu

Received 23rd October 2015 , Accepted 19th April 2016

First published on 25th April 2016

Abstract

Graduate students play a critical role in undergraduate education at doctorate granting institutions; but generally have minimal opportunity to develop teaching expertise. Furthermore, little is known about how graduate students develop teaching expertise in this context. We investigated the development of topic-specific pedagogical content knowledge among chemistry graduate student teaching assistants (GTAs). Thin layer chromatography was selected as the topic of investigation because undergraduate students encounter it throughout organic chemistry lab and it is connected to several foundational chemistry topics. An instrument was developed to measure both content knowledge (CK) and pedagogical content knowledge (PCK) of thin layer chromatography for GTAs with a range of teaching experience. Data from the test instrument were transformed using the Rasch model and statistically analysed. Our analysis showed that graduate students at all levels of experience performed well on content knowledge questions, but even experienced GTAs demonstrated low levels of pedagogical content knowledge. Importantly, experienced GTAs demonstrated a greater proficiency than novice GTAs, which suggests that pedagogical content knowledge is developed over time.

Introduction

Many pioneering studies have focused on the characterization of undergraduate learning (Anderson et al., 2011; Kober, 2014; Talanquer, 2014) with the goal of improving STEM education. However, less attention has been paid to the study of STEM instruction at the postsecondary level or its impact on student learning (Bond-Robinson, 2005; Alvarado et al., 2015; Mack and Towns, 2015; Rollnick and Davidowitz, 2015). In particular, the research-intensive culture at doctorate granting institutions means that instructional development is often underemphasized (Anderson et al., 2011). Therefore, research is needed to better understand the nature of effective STEM instruction and its impact on student learning.

One approach to improving STEM instruction is to focus on the development of graduate students’ teaching expertise, because they play a critical role in undergraduate instruction at doctorate granting institutions. GTAs have more contact hours with undergraduates in introductory courses relative to faculty and as a consequence GTA instruction may have an outsized impact on undergraduate learning (Luft et al., 2004). Although many graduate students go on to pursue faculty positions, they typically receive minimal formal pedagogical training during graduate school. At most large institutions, GTA training is short, overgeneralized, and does not reflect what is known about teaching and learning science (Anderson and Mitchner, 1994). Furthermore, graduate students often do not teach beyond their first year and thus have little opportunity to improve teaching through practice. In the absence of professional development aimed at teaching, graduate teaching assistants (GTAs) often teach the way they were taught and in a manner that does not reflect new knowledge about how people learn science (Grossman, 1989; Singer et al., 2012). Very little data exists on the efficacy of GTA training and existing reports in the literature are almost all based on student and TA perceptions (Nyquist et al., 1989; Robinson, 2000; French and Russell, 2002; McManus, 2002). Therefore, cognitive measures of teaching knowledge are needed to fully understand the impact of training on teaching so that it can be aligned with assessments of student learning gains.

A key component of teaching expertise is pedagogical content knowledge (PCK), which is a teacher-specific type of knowledge. PCK was conceptualized by Lee Shulman (Shulman, 1986, 1987), who described it as:

“that special amalgam of content and pedagogy that is uniquely the province of teachers, their own special form of professional understanding.”

As a fusion of pedagogical knowledge and content knowledge, PCK is subject matter dependent; meaning that the character of an English teacher's PCK is distinct from that of a Physics teacher. Research is needed to elucidate the unique nature of graduate students' PCK because the majority of studies on this teacher-specific knowledge have focused on K-12 pre-service teachers (Kind, 2009; Mavhunga and Rollnick, 2013; Alvarado et al., 2015; Rollnick and Davidowitz, 2015) and post-secondary faculty (Padilla and Van Driel, 2011; Mack and Towns, 2015). The character of GTA teaching knowledge and the way in which it is developed is distinct from both faculty and K-12 teachers. Many graduate students teach during their first year of graduate school and thus have unformed subject matter knowledge relative to faculty (Bhattacharyya and Bodner, 2014). Furthermore, most graduate students enter graduate school with little to no pedagogical training as compared to K-12 teachers (Grossman, 1989) and are unlikely to receive formal training during graduate school.

To date the only reports of research examining chemistry graduate student specific PCK are studies by Bond-Robinson (Bond-Robinson, 2005), who coined the term “PChK” or pedagogical ‘chemistry’ knowledge. She described this discipline-specific PCK based on observations of graduate students teaching and undergraduate perceptions of their teaching. The study found that in the lab setting GTAs more readily develop lab management skills rather than teaching expertise. Management skills, while important, are not necessarily correlated with a GTA's ability to teach chemical concepts. Furthermore, Bond-Robinson found that those GTAs who performed well as lab managers tended to rate highly on teaching evaluations, even if their teaching expertise was not well-developed (Bond-Robinson, 2005). Therefore, self-reports by students and GTAs alone may be an inadequate method for evaluating the impact of GTA training.

Pedagogical content knowledge is a challenging target for study because it is partly an internal construct that exists in the mind of the instructor and thus may not be evident through observation alone (Baxter and Lederman, 2001). In order to uncover GTAs' tacit knowledge of teaching chemistry, a complementary approach is needed that can be used in combination with observational techniques and self-reports. However, there is a paucity of inferential test instruments for measuring PCK in general, and more specifically for PCK among science instructors (Baxter and Lederman, 2001). This study builds on Bond-Robinson's work by developing a cognitive measure of PCK that is specific to graduate students. The goals of this study were to (1) develop an instrument to measure GTA PCK in the context in which they typically develop teaching experience and (2) to determine if and how PCK is developed by GTAs over time.

Thin Layer Chromatography was selected as the specific focus of this study because graduate students will repeatedly encounter this topic when teaching organic chemistry lab (Martin et al., 2011). The graduate student will teach various aspects of the technique including (1) selecting an eluent; (2) preparing a sample with correct concentration; (3) preparing the plate; (4) spotting the compound on the plate; (4) developing the plate and (5) visualizing the plate (Pavia et al., 2013). They must also help students to connect each of these steps to their microscale conceptions of the phenomenon including how it relates to solvent selection and how to predict how each compound will travel on the plate. In this way, TLC is related to several fundamental topics in chemistry including polarity, solubility, concentration and intermolecular interactions.

Theoretical framework

Pedagogical content knowledge

Shulman described PCK as “the capacity of a teacher to transform the content knowledge he or she possesses into forms that are pedagogically powerful.” (Shulman, 1987). This notion of teacher knowledge has formed the basis of an evolving theoretical model used to explain the nature of teachers' knowledge of teaching specific content (Park and Oliver, 2008; Juttner et al., 2013; Rollnick and Davidowitz, 2015). The model describes teachers with pedagogical content knowledge as those having developed an ability to transform subject matter based on their teaching knowledge foundation, which includes (1) students' prior knowledge; (2) curricular saliency; (3) what makes a topic easy or difficult to understand; (4) representations or analogies; and (5) conceptual teaching strategies (Rollnick and Davidowitz, 2015).

Most recently, a consensus PCK model was proposed (BSCS, 2016), which describes PCK as an instructors’ knowledge for teaching a particular topic (“reflection on action”) as well as the particular way they act on that knowledge (“reflection in action”) (BSCS, 2016). An important component of this recent model is the distinction between an instructor's “personal” PCK and their “canonical” PCK, which describes that that an instructor may acquire it either through reflection on their own practice (personal) or that it can be gained through professional development and other social means (canonical) (Smith and Banilower, 2012; Alvarado et al., 2015; Garritz, 2015). According to Smith and Banilower “All teachers have personal PCK, whether tacit or explicit. Not all teachers possess canonical PCK” (Smith and Banilower, 2012). The latter is likely to apply to GTAs because they have fewer professional development opportunities than pre-service K-12 teachers.

PCK can be examined at general, subject specific, and topic specific levels. Where general PCK is distinct from pedagogical knowledge in that it has a specific character that is unique to each discipline. Subject specific PCK is distinct within a particular area of science. For example a chemistry graduate student will develop a “way of thinking” about chemical phenomena and how to teach it that would be distinct from that of a physics graduate student (Veal and MaKinster, 2001). The study reported here is framed in a ‘topic-specific’ PCK (or TS-PCK) model (Hill, 2008; Mavhunga, 2012; Rollnick and Davidowitz, 2015), which identifies that the character of an instructor's PCK is unique to each discrete topic within a discipline. For example, an instructors' PCK on the topic of stoichiometry will be distinct as compared to acid–base chemistry, because both depend on their content knowledge as well as their personal teaching experiences. The TS-PCK model identifies that each specific topic is transformed to a ‘pedagogically powerful form” by the teacher (Hill, 2008; Mavhunga, 2012; Rollnick and Davidowitz, 2015). This transformation can occur as a teacher develops their own awareness of teaching through reflection on their own teaching experience (personal PCK) or through professional development (canonical PCK) (Alvarado et al., 2015; BSCS, 2016).

TS-PCK is an appropriate model for this study for several important reasons. First, it recognizes that student learning of discrete topics has an individual character that differs from topic to topic (Mavhunga, 2012). It is therefore aligned with constructivist theories of learning that describe knowledge as being constructed in the mind of the learner through their unique experiences and reflection (Cochran, 1993). Second, it makes explicit the mechanism of knowledge transformation and in so doing clarifies how novice instructors may acquire PCK (Kind, 2009).

Content knowledge

Content knowledge is an important component of a teacher's knowledge base, which enables them to develop PCK at the topic level (Mavhunga, 2012; Alvarado et al., 2015). Thus it is important to consider both the subject matter knowledge of teachers as well as the knowledge of the students they will teach. Content knowledge for this study is framed through the lens of Novak's theory of meaningful learning and human constructivism (Ausubel et al., 1978; Lowery-Bretz, 2001). Novak identifies that in order for meaningful learning to occur the learner must access relevant prior knowledge, the new knowledge must be presented to the learning in a meaningful way, and the learner must choose to engage in the learning process (Ausubel et al., 1978; Lowery-Bretz, 2001). This theory places much of the responsibility for learning on the student because the teacher can only control the way in which new knowledge is presented to the student (Lowery-Bretz, 2001).

In describing this theory Novak contrasts meaningful learning with rote learning (i.e. memorization) (Novak, 1985). Rote learning is described as occurring when new information is not integrated with students' existing knowledge structure whereas meaningful learning occurs when new information is specifically integrated (Novak, 1985). Students tend to adapt their learning strategies to achieve success, which from a students perspective, can mean earning a higher grade This perspective may lead toward rote learning strategies (Edmondson and Novak, 1993). The tendency of GTAs to use rote learning strategies in their own studies, coupled with the absence of professional instructional training, may mean that GTAs will teach the way they were taught using techniques that promote rote learning (Roehrig et al., 2003) and thereby perpetuate them.

Methodology

A mixed methods approach was used to develop and validate an inferential measure of graduate student's development of TS-PCK of thin layer chromatography (TLC). The test instrument itself was a quantitative measure of both GTA PCK and content knowledge. Additionally, survey questions were developed to characterize participant's relative teaching experience, interest and background information. Cognitive interviews were performed to validate the test instrument and to uncover the experiences that contributed to graduate students' development of PCK.

Test design

As part of a larger study, a test instrument was designed to measure the TS-PCK of chemistry graduate students with various levels of teaching experience. Graduate students in chemistry most frequently teach laboratory sections; therefore the topics for the full test were selected from among the topics most commonly encountered in introductory organic chemistry lab courses. Thin layer chromatography is the focus of this paper and was selected because it is used repeatedly throughout Organic I & II labs and thus GTAs will frequently encounter opportunities to develop TS-PCK around it. Application of TLC requires both technical knowledge (spotting, sample preparation, analysis) as well as understanding of foundational chemistry concepts. The underlying concepts that are needed in order for students to understand TLC, such as intermolecular forces (Williams et al., 2015), solubility and polarity (Teichert et al., 2008), are conceptually difficult for students to learn. Furthermore, reconciling what they do in lab (technique) to what they learn in lecture (concepts) requires that students translate between symbolic, microscopic, and macroscopic representations of phenomena. This is typically easier for GTAs, who are relative experts (Taber, 2001) and, because it is likely a tacit form of knowledge, they may not recognize the need to teach it explicitly.

PCK is both an internal and external construct (Baxter and Lederman, 2001). It consists of what an instructor knows, what they do with that knowledge, and the reasons behind their actions. An inferential technique is designed to uncover an instructor's level of PCK, whereas observational techniques may reveal how a teacher acts on that knowledge (Baxter and Lederman, 2001). In this case an inferential technique was used as the approach to uncover graduate students knowledge of teaching TLC. The PCK questions were designed to be open-ended short-answer questions, because multiple-choice questions would limit the range of possible responses. In writing these items we took direction from Hill, who considered what subject matter experts who lack teaching experience would know (Hill, 2008). In this case, we considered chemistry graduate students, who should be adept at using TLC but may not have taught it previously.

A test blue print (Table 1) was created to inform the design of each question and the target component of CK and PCK. Using a standard method of test development the questions were drafted, judged by experts and piloted. The final version of the test included four CK questions and four PCK questions (see Table 1). Questions were designed to measure both PCK as well as CK, because PCK is dependent on content knowledge (Mavhunga, 2012). The CK questions were based on textbook questions and the authors' teaching experience. The CK questions were designed to probe graduate students declarative knowledge, procedural knowledge, and conditional knowledge specific to the topic of TLC based on Bloom's taxonomy (Anderson et al., 2001). PCK questions were written based on the literature of TS-PCK (Hill, 2008; Mavhunga, 2012) and our own classroom experiences. Two different components of PCK were probed including teaching strategies and what makes topics difficult or easy (Table 1). For example, PCKQ4 asks the GTA to identify the aspects of TLC that a student will find most challenging. Categories such as knowledge of students' prior knowledge and knowledge of representations and analogies were not targeted specifically in the questions. However, some responses to the open-ended questions revealed knowledge of these categories on the part of some GTAs.

Table 1 Test blueprint for PCK and CK questions
Type of CK CK question
Declarative knowledge CKQ1. Describe a TLC plate where a solvent of too high polarity is used. A solvent of too low polarity.
Procedural knowledge CKQ2. Calculate Rf
CKQ3. Predict the order of elution
Conditional knowledge CKQ4. Explain appearance of TLC plate (why are the spots on the left larger than those on the right).
Aspect of PCK PCK question
Teaching strategies PCKQ1. As if to a student, explain how a compound may travel differently in solvents of relatively low or high polarity.
PCKQ2. Identify an issue in an example TLC plate and explain how they might improve on technique so that their plates develop with spots on the right lane rather than those on the left.
PCKQ3. How would you help students to understand how to identify solvents of high or low polarity?
What makes a topic easy or difficult PCKQ4. What aspects of TLC do you think are most challenging for students to master?

Additional data for this study includes audiotaped cognitive interviews as well as the interviewer's field notes taken during the interviews. Survey questions were also incorporated to obtain GTA attitudes on teaching, prior teaching experience and other background information. Cognitive interviews were performed to ascertain the validity of the results by determining whether what was conceptualized as PCK was actually measured by the test items. For example, whether a participant's response was coded for PCK but arose from a process of reasoning rather than from their knowledge of students (Willis et al., 1999). During the interviews, participants were asked to report how they answered questions related to PCK using a protocol based on a “verbal probing technique” to provide a sense of the cognitive processes that participants engaged in when responding (Willis et al., 1999). Interview questions included general probing questions such as “How did you arrive at that answer?” and more specific questions such as “Do you recall when you first realized that students would find that challenging?”

Participants

Participants consisted of sixty-seven GTAs enrolled in a chemistry doctorate program at a public university. Of those participants, thirty-six were 1st-year graduate students who participated in this study during orientation. The remaining participants were GTAs with a range of experience. Twenty-seven GTAs were teaching the Organic II lab at the time of the study and took the test at the beginning and end of the term. The remaining GTAs had previously taught Organic I or II labs two or more times, but were not currently teaching. Five of the participants participated in cognitive interviews. All participants consented voluntarily in this study and IRB approval was obtained (Table 2).
Table 2 GTA characteristics
Characteristic Number of GSIs
Gender
Male 41
Female 27
Year in program
1 53
2 3
3 2
4 3
5 or more 7
Division
Organic 42
Other 26
Terms teaching experience
0 36
1 19
2 2
3 2
4 2
5 or more 7

Data analysis

The PCK responses were scored using the rubric described in Table 3, which is similar to that reported by Rollnick (Mavhunga and Rollnick, 2013) and Park (Park and Oliver, 2008). Each of the PCK elements tested was organized on a 5-point scale, from incorrect (0) to transforming (4) (see Table 3 and Appendix 1 for additional examples). The operational definitions for the rubric were the result of discussion between three coders. The definitions and application of the rubric were discussed until greater than 95% rater agreement was reached and Chronbach's Alpha was 0.61. The raw scores for both PCK and CK were subjected to Rasch analysis (Neumann et al., 2011) using Winsteps software. Raw scores were converted into LOGIT units, which are probability measures that are on an equal scale. The Rasch model also provided reliability estimates and additional quantitative measures of validity. The converted CK and PCK scores were then statistically analysed.
Table 3 Rubric designed to measure GTA responses to PCK questions
PCK rating Description
(0) Incorrect Answers the questions incorrectly
(1) Limited Answers the questions correctly but with no explanation
(2) Basic Provides standardized knowledge as explanation
(3) Developing Provides standardized knowledge as explanation and elaborates correctly
(4) Transforming Uses dynamic and conceptual teaching strategies, includes representations and analogies in response

Cognitive interviews were transcribed verbatim and coded by one of the authors for the sources of knowledge invoked to respond based on a coding scheme adapted from Hill (Hill, 2008). The Hill coding scheme was modified for the question type (open questions only), the topic specific PCK model used to conceptualize the test questions, and the sources of knowledge that are unique to graduate students (i.e. research as well as teaching) (Table 4). Hills' original code for test-taking skills was not used because the PCK portion of the instrument did not include multiple choice or forced response type questions on which respondents might be likely to apply test-taking skills. The code for “mathematical reasoning” was adapted to “chemical reasoning” and divided between reasoning that originated from knowledge gained through coursework and knowledge gained through research, which is an important distinction between graduate students and pre-service teachers. This type of GTA specific reasoning was observed in Michael's responses:

Table 4 Rubric of GTA sources of knowledge on PCK questions
PCK rating Description
Topic-specific knowledge of students GTA conveys familiarity with the challenges students have, knowledge of common errors, or experience explaining a concept that is specific at the topic level
General knowledge of students GTA conveys familiarity with the challenges students have, knowledge of common errors, or experience explaining a concept that is not specific to the topic, but is drawn from another teaching experience.
Chemical reasoning based on experience as a student GTA uses deduction, inference, or other reasoning to support their answer and connects it to an experience they had as a student.
Chemical reasoning based on research experience GTA uses deduction, inference, or other reasoning to support their answer and connects it to an experience they had doing research.

“I was just thinking about my peers in class and kind of seeing where everyone was and what we all learned as freshman from my institution. That was my reference point.”

“I would say this was from my own experience, especially over-spotting.”

In the first statement Michael's reasoning is based on knowledge gained through coursework, while in the second statement his reasoning is based on knowledge gained through research experience. Finally, the code for knowledge of students was modified to reflect the topic-specific nature of PCK (Mavhunga, 2012) and divided into knowledge of students that arose from an experience teaching that specific topic and knowledge that arose from general teaching experience.

A subset of the PCK and CK test responses were rated and discussed by two other researchers and operational definitions used for each question were revised until greater than 90% agreement was reached. Likewise, another researcher coded a subset of the interview data and the results and discrepancies in data coding were resolved through discussion. Due to the relatively small sample sizes for interviews (n = 5), statistical analysis was not conducted.

Results

Test development and validity

The Rasch analysis showed that the instrument for measuring both CK and PCK was valid with high reliability indices. Measures of person reliability, which indicates whether the test discriminates sufficiently across the ability range of the participants in the sample, and item reliability, which indicates whether the test items represent a range of difficulty, were used (Neumann et al., 2011; Bond and Fox, 2013). Questions designed to measure content knowledge provided a person reliability of 0.32 and item reliability of 0.98. PCK questions provided a person reliability of 0.73 and item-reliability of 0.94, which are comparable to previously reported values for similar measures of PCK (Table 5) (Mavhunga and Rollnick, 2011; Juttner et al., 2013). The low person reliability for CK indicates that the test did not thoroughly discriminate between the test respondents (Bond and Fox, 2013). This may be attributed to the relatively high knowledge of the graduate students on the test items, which were designed for undergraduate students who were newly learning thin layer chromatography, as discussed in the test design.
Table 5 GTA Rasch reliability values
Indices Content knowledge Pedagogical content knowledge
Person reliability 0.32 0.73
Item reliability 0.98 0.94
Fit statistics (t = −2 and +2) All scores All scores

Item–person maps were generated to provide a visual representation of alignment and spread of a GTA's ability along with the difficulty of the individual test items (Fig. 1). The specific location of a person on an item–person map indicates the probability that this person will correctly answer items of matching difficulty (Bond and Fox, 2013). The distribution in Fig. 1 shows the spread of GTA performance with respect to CK questions. A mean person performance of 2.56 (std dev. 1.41) on content knowledge questions indicates that the overall GTA content knowledge is high. The most difficult question, CKQ4, assessed the conditional knowledge of a GTA (interpretation of a TLC plate), however, more than half of the GTAs had a greater than 50% probability of giving the correct answer. Approximately 75% of the GTAs had a high probability of correctly answering CKQ1, which assessed declarative knowledge (describe a TLC plate that was developed with too high or low polarity eluent). GTAs performed the best on the questions that assessed their procedural knowledge, where about 95% had a high probability to correctly answer CKQ3 (predicting the order of elution) and all GTAs had a high probability of correctly answering CKQ2 (calculating Rf).


image file: c5rp00190k-f1.tif
Fig. 1 Item–person map of GTA content knowledge of thin layer chromatography.

The distribution in Fig. 2 shows the spread of GTA proficiency with respect to PCK. The mean person performance was −0.62 (std dev. 1.95), which indicates that GTAs had a relatively low proficiency with PCK. Only 30% of GTAs had a high probability of sufficiently answering questions related to PCK. Predicting what would be challenging for students (PCKQ4) and employing teaching strategies related to improving spotting technique (PCKQ2) were the most challenging questions. Knowledge of teaching strategies (PCKQ1) and (PCKQ3) were somewhat less challenging.


image file: c5rp00190k-f2.tif
Fig. 2 Item–person map of GTA pedagogical content knowledge of thin layer chromatography.

We found that while the majority of GTAs have high levels of content knowledge pertaining to the TLC, the reasoning behind their PCK answers was limited. For example, when asked what aspects of TLC are most challenging for a student to master (PCKQ4), most GTAs identified a challenge (i.e. choosing the right solvent) but few elaborated on why it is challenging. Many GTAs noted that students had difficulty selecting a solvent as an eluent, but not why, as in the following case:

“Finding the most suitable eluting system.”

Only a few GTA responses demonstrated that they understood why students have difficulty with TLC. For example:

“I have found that students had a difficult time grasping the idea that the compounds actually interacted with the silica – the reasoning behind polar compounds sticking and not moving well up the plate without polar solvent.”

Additionally when asked to explain a concept, as in PCKQ1, incorrect notions on the part of the GTA were sometimes uncovered and identified as an incorrect answer with a score of zero on the rubric. For example:

“If a compound has a similar polarity to the solvent it will move with it higher up the plate. Where as if it is not similar in polarity, it will stick to the silica gel and not move.”

This GTA misapplied the general notion of “like-dissolves-like” to the solvent only and failed to address the interaction of the compound with the silica plate. The “like-dissolves-like” heuristic is commonly used to introduce solubility and students who learned solubility by rote memorization may misapply it. In contrast, a GTA who demonstrated PCK for PCKQ1 used non-chemistry terms to personify the dynamic interaction between compound and silica, which may serve to connect to students existing knowledge structure:

“The TLC plate itself is very polar. Very polar compounds tend to be attracted to each other and thus a very polar compound will “stick” to the TLC plate. Non-polar compounds, on the other hand, will not be as attracted to plate and is thus “less sticky.”

Relationship between CK and PCK

A scatterplot relating CK and PCK is shown in Fig. 3. A positive and statistically significant correlation was found between GTA CK and PCK. Three quadrants are populated in the plot resulting in a lower correlation coefficient. This indicates that, while GTA CK is correlated with PCK, there are also GTAs who are subject matter experts (high CK), but have not necessarily developed teaching expertise on this topic (low PCK). Importantly, no GTA was found to show high PCK and low CK, which supports the content validity of the test and is consistent with prior studies (Mavhunga and Rollnick, 2011; Juttner et al., 2013).
image file: c5rp00190k-f3.tif
Fig. 3 Scatterplot relating content knowledge to pedagogical content knowledge of thin layer chromatography.

Cognitive interviews

Cognitive interviews were performed to ascertain how the test questions were interpreted by the GTAs and to uncover the source of their teaching knowledge. One particular item, PCKQ1, was revealed to be problematic. In this case two of the interviewees indicated that they felt restricted by having to explain something in writing, but had they had a student in front of them they would have used strategies such as questioning or drawing on a chalkboard. This limitation is consistent with the internal and external nature of PCK, which indicates that PCK is both what a teacher knows (“reflection on action”) and what they do (“reflection in action”) (BSCS, 2016). This suggests that some GTA's responses to this question were limited in their capacity to capture what GTAs know versus what GTAs would do in practice. Two approaches to addressing this limitation in future iterations of the instrument include (1) refining the question prompt to include drawing as a response or to include authentic student dialog from which a GTA may respond or (2) by redrafting the question based on the responses given by other GTAs in this study.

During cognitive interviews GTAs were questioned about the source of their knowledge in answering each survey question. Using the coding scheme described above (Table 4) each answer was identified as being generated from (1) topic-specific teaching experience; (2) general teaching experience; (3) reasoning from content knowledge gained through coursework or (4) reasoning from content knowledge gained from research. Each question could receive more than one code, if the graduate student indicated that both sources of knowledge were used in answering the question. For example, a graduate student may describe a specific teaching instance, but may also recall the first time they experienced the same issue as a student. Table 6 shows the percentage of each response by category. Sometimes the origin of their response was given spontaneously (i.e. “the students always do that”). In all other instances probing questions were used to elicit responses.

Table 6 Percentages indicating origin of reasoning for GTA survey responses
GTA Dwight Pam Jim Michael Kelly
Topic-specific teaching experience 78% 0 22% 0 0
General teaching experience 0 29% 21% 29% 21%
Reasoning from coursework knowledge 17% 25% 17% 8% 33%
Reasoning from research knowledge 0 50% 50% 0 0
Number of terms taught 4 0 0 0 0
PCK test score 1.52 −0.51 −3.42 1.08 1.08
CK test score 3.05 1.41 1.41 3.05 3.05

Each graduate student described a range of experiences that contributed to their responses. Dwight, who was the only experienced GSI to be interviewed had the highest PCK and largely indicated that his response came from specific experiences teaching thin layer chromatography. All graduate students indicated that they reasoned from their own content knowledge. Two of the graduate students related their responses to experiences in which they began learning how to use TLC in their own research and reasoned from these experiences to recognize common errors that students would make. Four of the five graduate students used prior teaching experience related to teaching other chemistry topics to inform their responses.

Growth in GTA PCK with experience

The bubble plot shown in Fig. 4 relates the number of terms teaching experience to the mean PCK score for both experienced and inexperienced GTAs. The relative numbers of students in each bin are conveyed by the size of the data point in the plot. Note that the number of GTAs with multiple terms of teaching experience is small as compared to GTAs with no teaching experience. Those GTAs with no experience displayed the lowest mean PCK score of −1.5. The mean PCK score increases with number of terms teaching experience, supporting the claim that PCK is developed over time and with practice. Thus, a greater PCK would be expected for more experienced graduate students.
image file: c5rp00190k-f4.tif
Fig. 4 Plot relating the number of terms teaching experience to mean pedagogical content knowledge.

A positive and significant correlation was observed between PCK and number of terms overall teaching experience and organic chemistry teaching experience (Table 7). These values also demonstrate the convergent validity of the instrument. The correlation between PCK and other factors such as interest in teaching, confidence in teaching ability, gender, students' field of study, or year in graduate school were not significant.

Table 7 Pearson correlations between CK, PCK and GSI characteristics
Indices Content knowledge Pedagogical content knowledge
a p(two-tailed) < 0.001. b p(two-tailed) < 0.01.
Content knowledge 1 0.451a
Pedagogical content knowledge 0.451a 1
Number of terms taught 0.24 0.338b
Interest in teaching 0.113 −0.615
Confident in teaching −0.181 −0.19
Gender −0.133 0.324
Year in program 0.175 −0.148

Limitations

PCK is a complex type of knowledge, because it is both an internal and external construct, and is therefore not easy to assess (Carlson, 1990). Inferential techniques, such as the short answer questions used in this test instrument, are limited in that they may not capture how a teacher will act on their PCK. Multiple choice answers have further limitations, because they may overlook particular aspects of PCK that were not anticipated in test design and may show poor criterion-related validity (Carlson, 1990). In contrast, short answer questions, because they do not force participants toward a predefined set of answers, should provide a greater opportunity to capture PCK as an internal construct. Cognitive interviews were performed to validate the survey and to better understand the source of GTA knowledge. However, only five GTAs were interviewed and only one of these was an experienced GTA.

Discussion

A test instrument was designed to measure graduate students' topic-specific pedagogical content knowledge of thin layer chromatography. The construct validity of the PCK measure was established by multiple methods including cognitive interviews, reliability values determined using the Rasch model, and through statistical correlations between GTA PCK, CK, and teaching experience, which demonstrated the convergent validity of the instrument.

Several key themes arose from our findings that contributed to a more refined understanding of the nature of graduate students PCK. First, graduate students demonstrated a strong content knowledge of TLC, but had relatively low pedagogical content knowledge on this topic. Many GTAs relied on heuristics, such as like-dissolves-like, when explaining the intricacies of TLC to students. Some graduate students demonstrated an incomplete or incorrect understanding of aspects of TLC that may have come about through learning TLC by rote and misapplying such heuristics. This suggests that GTAs propagate rote learning strategies in their own teaching, most likely from reliance on past academic experiences.

Second, our findings are consistent with previous models of PCK in that content knowledge plays a significant role on the development of PCK, as first described by Shulman (Shulman, 1986, 1987). Mavhunga further extended this concept by developing a model to investigate the influence of CK on PCK (Mavhunga, 2012). Consistent with Mavhunga's transformation theory, the small but significant correlation between CK and PCK corroborates the accepted belief that CK is a pre-requisite for developing PCK. Notably, no GTA was found to have high PCK without CK, further suggesting that CK is required as a foundation for PCK among GTAs.

A third important observation is that GTAs with more teaching experience demonstrated greater PCK, even in the absence of targeted training. This finding is consistent with previous literature, which showed that PCK appears to be limited in novice teachers such as GTAs, but does develop through practice and reflection (Lederman and Gess-Newsome, 1999; Davis and Krajcik, 2005). Further, Bond-Robinson observed that when guided, GTAs can develop certain levels of PCK, mainly those defined as lab-management skills (Bond-Robinson, 2005). It is important to note that, although we observed an increase in PCK with level of teaching experience, the PCK of experienced GTAs is still low.

Finally, based on observations from cognitive interviews, it is clear that GTAs draw on a variety of experiences and knowledge to inform their responses. It was found that the source of PCK was primarily due to reflection on personal teaching experiences. No graduate student indicated that GTA training was a source for responding to PCK questions. The experiences reported by GTAs are distinct from K-12 pre-service teachers on which the majority of PCK research has been conducted and for which training methods have been applied. Taken together, these findings underscore the need for a GTA specific model of PCK that can inform the design of GTA specific training.

Conclusions

The test instrument described here is the first inferential measure of pedagogical content knowledge that is specific to chemistry graduate students. While previous studies have examined graduate student pedagogical content knowledge through observational studies, which characterized what a graduate student does while teaching, this study provided a method that can be used to better understand how graduate students think about teaching. We found a modest difference between the pedagogical content knowledge of experienced graduate students as compared to novice a graduate student, which suggests that PCK is developed through experience. We also found that the source of graduate students' PCK is distinct from that of pre-service teachers and thus, this study serves as a jumping off point for developing a graduate student specific model of pedagogical content knowledge.

Very little is understood about the specific nature of graduate students' teaching expertise and the ways in which it is developed. Multiple studies have examined student learning of fundamental chemistry topics (bonding, intermolecular forces, acid–base chemistry) (Taber, 2001; Teichert et al., 2008; Cooper et al., 2010), which are repeatedly touched on in organic chemistry lab courses. Given that graduate students teach primarily at the introductory level, additional work is needed to understand how graduate students teaching expertise is related to student learning of these specific topics.

Appendix 1

Tables 8–11.
Table 8 PCKQ1: As if to a student, explain how a compound may travel differently in solvents of relatively low or high polarity
Score Student example
(1) Limited: correctly identifies the dependence of compound distance travelled and solvent polarity “High polarity usually travels faster, so the Rf is usually larger than the low polarity one”
(2) Basic: correctly identifies dependence and explains using standard knowledge “With less polar solvent, the intermolecular interaction between the solvent and the compound is weaker, hence the compound might not travel as far as it does in solvent with relatively higher polarity”
(3) Developing: expands on the explanation “A compound's is affected by two forces. It is both attracted to the plate's polar groups and it can be pushed up the plate by the polarity of the solvent. Low polarity solvent will push less hard then high polarity solvents. So in low polarity solvents, the compounds will stick more to the plate and move less in comparison to high polarity solvents”
(4) Transforming CK to PCK: employs an analogy “If you had two people tugging on you from either arm, with the strength of their pull being the strength of the “polarity”, where do you think you would end up if you had one person significantly stronger in “polarity” than the other? If the two people were equal in “polarity”, then where would you end up?”

Table 9 PCKQ2: Identify an issue in a sample TLC plate and explain how a student might improve their TLC technique so that their plates develop with spots like those on the right lane rather than those on the left lane
Score Student example
(1) Limited: identifies a technique for improving the TLC plate “Make the sample spot at starting line as small as possible.”
(2) Basic: identifies a technique for improving the TLC plate with standardized knowledge as an explanation “Multiple applications with time left to dry in between leads to a smaller, more concentrated spot.”
(3) Developing: expands on the explanation “By making the mixture you are spotting more dilute you decrease the amount of compound on the plate so your spots are less likely to smear together. Also, by spotting quickly so that minimal amounts of liquid are transferred onto the plate you also minimize the amount of compound on the plate”
(4) Transforming CK to PCK No examples

Table 10 PCKQ3: How would you help students to understand how to identify solvents of high or low polarity?
Score Student example
(1) Limited: identifies a method for the students to test low vs. high polarity “Let them adjust solvent mixture based on TLC results.”
(2) Basic: identifies a method, and provides standardized knowledge as an explanation “Presence of electronegative atoms often imparts polarity on a compound. Types of intermolecular interactions (hydrogen-bonding, dipole–dipole, etc.) govern the polarity of compounds. Explain to the student the motifs that give rise to certain interactions and why these interactions are strong or weak”
(3) Developing: uses dynamic teaching, such as drawings “I would draw out the molecule on the board, and ask them if there is an inherent dipole on the molecule. If there was, then maybe compare it to a solvent they know to be highly polar (water), and ask them to compare the relative strengths of those dipoles. This may give them a better understanding of how polar their solvent choices are”
(4) Transforming CK to PCK: recognizes student prior knowledge “Hopefully, their knowledge of polarity and electronegativity differences does exist. If not, start from there. Identify strongly polar bonds such as C–O or C–N and explain if necessary effects of symmetry if discussion dichloromethane vs. chloroform vs. carbontetrachloride. Also identifying polarity changes with length of alkyl chains of molecules. Presenting solvent miscibility and polarity tables that are documented is also a good strategy.”

Table 11 PCKQ4: What aspects of TLC do you think are most challenging for students to master?
Score Student example
(1) Limited: identifies a way in which TLC might be challenging with no further explanation “Choosing the right solvent”
(2) Basic: identifies a way in which TLC might be challenging, and provides standardized knowledge as an explanation “Finding the right developing solvent to use to get a good separation of solutes”
(3) Developing: expands on the explanation “Understanding the importance of dilution, and what gives good separation. It's not all polar or non-polar, it's what is more polar than what”
(4) Transforming CK to PCK “Students don't understand that surfaces can be just as reactive as molecules in solution. Especially difficult is the visualization of a glass surface as being ‘seen’ by the solvent as an ocean of hydroxyl groups”

Acknowledgements

We acknowledge the UM President's Postdoctoral Fellowship and the UM CSEI|UM Future Faculty program for funding.

Notes and references

  1. Alvarado C., Canada F., Garritz A. and Mellado V., (2015), Canonical pedagogical content knowledge by CoRes for teaching acid–base chemistry at High School, Chem. Educ. Res. Pract., 16, 603–618.
  2. Anderson R. D. and Mitchner C. P., (1994), in Gabel D. L. (ed.), Handbook of research on science teaching and learning, New York: Macmillan, pp. 3–44.
  3. Anderson L. W., Krathwohl D. R. and Bloom B. S., (2001), A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives, New York: Allyn and Bacon.
  4. Anderson W. A., Banerjee U., Drennen C. L., Elgin S. C. R., Epstein I. R., Handelsman J., Hatfull G. F., Losick R., O'Dowd D. K., Olivera B. M., Strobel S. A., Walker G. C. and Warner I. M., (2011), Changing the culture of science education at research universities, Science, 331, 152–153.
  5. Ausubel D. P., Novak J. D. and Hanesian H., (1978), Educational psychology: a cognitive view, Chicago, IL: Holt, Rinehart and Winston.
  6. Baxter J. A. and Lederman N. G., (2001), in Gess-Newsome J. and Lederman N. G. (ed.), Examining Pedagogical Content Knowledge, Boston: Kluwar Academic Publishers.
  7. Bhattacharyya G. and Bodner G. M., (2014), Culturing reality: how organic chemistry graduate students develop into practitioners, J. Res. Sci. Teach., 51, 694–713.
  8. Bond T. G. and Fox C. M., (2013), Applying the Rasch Model: Fundamental measurement in the Human Sciences, New York: Routledge.
  9. Bond-Robinson J., (2005), Instruments to drive effective constructivist teaching, Chem. Educ., 10, 154–162.
  10. BSCS, PCK Summit Dissemination Site, http://pcksummit.bscs.org, (accessed March 11, 2016).
  11. Carlson R. E., (1990), Assessing teachers' pedagogical content knowledge: item development issues, J. Pers. Eval. Educ., 4, 157–173.
  12. Cochran K. F., (1993), Pedagogical content knowing: an integrative model for teacher preparation, J. Teach. Educ., 44, 263–272.
  13. Cooper M. M., Grove N., Underwood S. M. and Klymkowsky M. W., (2010), Lost in Lewis structures: an investigation of student difficulties in developing representational competence, J. Chem. Educ., 87, 869–874.
  14. Davis E. A. and Krajcik J., (2005), Designing educative curriculum materials to promote teacher learning, Educ. Res., 34, 3–14.
  15. Edmondson K. M. and Novak J. D., (1993), The interplay of scientific epistemological views, learning strategies, and attitudes of college students, J. Res. Sci. Teach., 30, 547–559.
  16. French D. and Russell C., (2002), Do graduate teaching assistants benefit from teaching inquiry-based laboratories? Bioscience, 52, 1036–1041.
  17. Garritz A., (2015), PCK for dummies. Part 2. Personal vs. canonical PCK, Educ. Quimica, 26, 77–80.
  18. Grossman P. L., (1989), Learning to teach without teacher education, Teach. Coll. Rec., 91, 191–208.
  19. Hill H. C., (2008), Unpacking pedagogical content knowledge: conceptualizing and measuring teachers' topic-specific knowledge of students, J. Res. Math. Educ., 39, 372–400.
  20. Juttner M., Boone W., Park S. and Neuhaus B. J., (2013), Development and use of a test instrumnet to measure biology teachers' content knowledge (CK) and pedagogical content knowledge (PCK), Educ. Asse. Eval. Acc., 25, 45–67.
  21. Kind V., (2009), Pedagogical content knowledge in science education: perspectives and potential for progress, Stud. Sci. Educ., 45, 169–204.
  22. Kober L., (2014), Reaching students: what research says about effective instruction in undergraduate science and engineering, Washington, DC: The National Academies Press.
  23. Lederman N. G. and Gess-Newsome J., (1999), in Examining Pedgogical Content Knowledge, Gess-Newsome J. and Lederman N. G. (ed.), Boston: Kluwer Academic Publishers.
  24. Lowery-Bretz S., (2001), Novak's Theory of Education: Human Constructivism and Meaningful Learning, J. Chem. Educ., 78, 1107.
  25. Luft J. A., Kurdziel J. P., Roehrig G. H. and Turner J., (2004), Growing a garden without water: teaching assistants in introductory science laboratories at a doctoral/research university, J. Res. Sci. Teach., 41, 211–233.
  26. Mack M. R. and Towns M. H., (2015), Faculty beliefs about the purposes for teaching undergraduate physical chemistry courses, Chem. Educ. Res. Pract., 17, 80–99.
  27. Martin C. B., Schmidt M. and Soniat M., (2011), A survey of the practices, procedures, and techniques in undergraduate organic chemistry teaching laboratories, J. Chem. Educ., 88, 1630–1638.
  28. Mavhunga M. E., (2012), Explicit inclusion of topic specific knowledge for teaching and development of PCK in preservice science teachers, University of Witwatersrand.
  29. Mavhunga E. and Rollnick M., (2011), The development and validation of a tool for measuring topic specific PCK in chemical equillibrium, African Journal of Research in Mathematics, Science and Technology Education, 17(1–2), 113–125.
  30. Mavhunga E. and Rollnick M., (2013), Improving PCK of chemical equillibrium in pre-service teachers, African Journal of Research in Mathematics, Science and Technology Education, 17, 113–125.
  31. McManus D., (2002), Developing a teaching assistant preparation program in the School of Oceanography, J. Geosci. Educ., 50, 158–168.
  32. Neumann I., Neumann K. and Nehm R., (2011), Evaluating instrument quality in science education: Rasch-based analyses of a nature of science test, Int. J. Sci. Educ., 33, 1373–1405.
  33. Novak J. D., (1985), in West L. H. T. and Pines A. L. (ed.), Learning how to learn, Florida: Academic Press, p. 190.
  34. Nyquist J., Abbott R. and Wulff D., (1989), in Teaching assistant training in the 1990s, vol. 39.
  35. Padilla K. and Van Driel J., (2011), The relationships between PCK components: the case of quantum chemistry professors, Chem. Educ. Res. Pract., 12, 367–378.
  36. Park S. and Oliver J. S., (2008), Revisiting the conceptualization of pedagogical content knowledge (PCK): PCK as a conceptual tool to understand teachers as professionals, Res. Sci. Educ., 38, 261–284.
  37. Pavia D. L., Lampman G. M., Kriz G. S. and Engel R. G., (2013), A microscale approach to organic laboratory techniques, Belmont, CA: Brooks/Cole.
  38. Robinson J., (2000), New teaching assistants facilitate active learning in chemistry laboratories: promoting teaching assistant learning through formative assessment and peer reivew, J. Grad. Stud. Teach. Assist. Develop., 7, 147–162.
  39. Roehrig G. H., Luft J. A., Kurdziel J. P. and Turner J. A., (2003), Graduate teaching assistants and inquiry-based instruction: implications for graduate teaching assistant training, J. Chem. Educ., 80, 1206–1210.
  40. Rollnick M. and Davidowitz B., (2015), Topic Specific PCK of Subject Matter Specialists in Grade 12 Organic Chemistry, in Huillet D. (ed.) Proceedings of the 23rd Annual Meeting of the Southern African Association for Research in Mathematics, Maputo: Science and Technology Education, Eduardo Mondlane University, short papers, pp. 243–250.
  41. Shulman L. S., (1986), Those who understand: knowledge growth in teaching, Educ. Res., 15, 4–14.
  42. Shulman L. S., (1987), Knowledge and teaching: foundations of the new reform, Harvard Educ. Rev., 57, 1–22.
  43. Singer S. R., Nielsen N. R. and Schwiengruber H., (2012), Discipline-based education research: Understanding and improving learning in undergraduate science and engineering, Nat. Acad. Press.
  44. Smith S. and Banilower E., (2012), PCK Summit Paper, http://pcksummit.bscs.org/sites/default/files/SmithBanilower EP.pdf (accessed March 23, 2016).
  45. Taber K. S., (2001), Building the structural concepts of chemistry: some considerations from educational research, Chem. Educ. Res. Pract., 2, 123–158.
  46. Talanquer V., (2014), DBER and STEM education reform: Are we up to the challenge?, J. Res. Sci. Teach., 51, 809–819.
  47. Teichert M. A., Tien L. T., Anthony S. and Rickey D., (2008), Effects of context on students' molecular-level ideas, Int. J. Sci. Educ., 30, 1095–1114.
  48. Veal W. R. and MaKinster J. G., (2001), Pedagogical content knowledge taxonomies, http://unr.edu/homepage/crowther/ejse/vealmak.html.
  49. Williams L. C., Underwood S. M., Klymkowsky M. W. and Cooper M. M., (2015), Are noncovalent interactions an achilles heel in chemistry education? A comparison of instructional approaches, J. Chem. Educ., 92, 1979–1987.
  50. Willis G., Lessler J. T. and Capar R. A., (1999), Cognitive interiewing: a “how to” guide, University of Iowa.
This journal is © The Royal Society of Chemistry 2016