How chemists handle not-knowing in reasoning about a novel problem

Abstract

The most obvious feature of expertise in chemistry is content knowledge, which defines the primary objectives of instruction. Research in chemistry education, and STEM education more broadly, has also devoted attention to students’ developing scientific practices of reasoning, investigation, and learning. In this study, we set out to investigate how expert chemists reason about an unfamiliar question. We conducted semi-structured, think-aloud interviews with fourteen chemists, all of whom found the problem novel. In this article, we focus on how the chemists handled the situation of not-knowing. We analyzed the moments when they said “I don’t know” (IDK), taking that as a clear, systematic marker of their not-knowing. The results elucidate two general dimensions of the chemists’ reasoning and experience. First, their identifying what they do not know served substantive roles in their reasoning, including to mark that they needed to search for insight or information, or to mark a boundary to the problem space. Second, IDK statements served to help the chemists manage what they experienced socially and emotionally, such as to hedge or distance themselves from ideas they considered, or to forestall their own—or the interviewer's—negative judgments. We discuss both aspects of our findings, and we consider possible implications for instruction and for further research.

---

Introduction

Research on learning and instruction in chemistry, and other sciences, has shifted in emphasis over the past several decades. The long-held and still stable focus of chemistry instruction remains on traditional content, but the Chemistry Education Research (CER) community presses for greater attention to “practices” of learning. This shift had prominent representation in the Next Generation Science Standards (2012), which has helped drive the community toward reform (Melanie et al., 2015). Some arguments for the shift of emphasis frame it as pedagogical, claiming that “active learning” is more effective for learning the content (Allison, 2001; Minhas et al., 2012; Roberts, 2017). Other arguments frame practices as essential targets for instruction in their own right: Chemists are professionals at learning, and learning the discipline must mean, in part, learning how to learn (Talanquer and Pollard, 2010).

CER has contributed to this view in various ways, significantly through studies of how students learn, focusing attention, for example, on argumentation (Erduran, 2019), mechanistic reasoning (Caspari et al., 2018a; Caspari et al., 2018b), imagistic reasoning (Jeppsson et al., 2015), as well as on how canonically incorrect ideas may be productive aspects of learning progressions (Sevian and Talanquer, 2014). Some researchers have begun developing instructional practices to elicit, support, attend, and respond to students’ reasoning (Flood et al., 2015).

The empirical work has focused almost entirely on students, what happens during courses, measurements of learning outcomes, and so on. Understandings of disciplinary practices by scientists have derived mainly from scholarship in science studies and the history and philosophy of science (Longino, 1990; Nersessian, 1992; Pickering, 1993; Firestein, 2012; Latour and Woolgar, 2013). That work was not conducted with educational aims, but Discipline-Based Education Research (DBER) scholarship has drawn heavily on it, citing evidence from these studies to support arguments for reconceptualizing the substance of science education (Council et al., 2012).

There have also been many studies with educational aims to understand expert knowledge to inform curriculum and instruction for students, such as by studying how experts understand and respond to familiar situations as evidence of how their knowledge is structured. But, there has been less work by education researchers directly to study disciplinary practices in experts.

Studies in mathematics (Schoenfeld, 1985; Schoenfeld, 2016), physics (Clement, 1994a; Clement, 1998; Singh, 2002; Clement, 2008; Clement, 2013), engineering (Cross, 2001; Ball et al., 2004), and history (Wineburg, 1991) have involved respective discipline-based education researchers presenting experts with novel, unfamiliar situations, specifically with the aim of revealing experts’ practices of reasoning. Historians for example, encountering material outside their familiarity, showed expertise in their strategies and inclinations for assessing the reliability of sources; mathematicians in the frequency and quality of metacognition, as they chose and assessed paths toward solutions; physicists and engineers in their use and assessment of analogies and heuristics. In all these cases, the authors drew implications from their findings for curriculum and instruction oriented toward supporting students’ approaches to reasoning.

Similar expectations, that it is not only what experts know but also what they do that matters, have motivated conceptualizing learning-teacher relationships as “cognitive apprenticeships” (Collins et al., 2018). Rather than expecting a relationship founded on a transfer of information, this view is of a collaboration that includes the learner observing the teacher doing the work themselves, apprenticing in the disciplinary approaches to learning. To be sure, students in any chemistry class watch their instructors' solving problems and performing demonstrations, and what they see has an effect on their learning. In almost all classes, however, the problems and the demonstrations are well-structured and pre-planned. Courses seldom afford students opportunities to see chemists doing chemistry themselves, to see them formulating their own questions, discovering, and confronting novel situations and naturally ill-structured challenges. By these arguments, students in science courses would benefit from witnessing how their expert teachers guide themselves through tackling scientific problems.

Authors’ positionalities and the formation of this study

The first author, John, began work on this study as an undergraduate, having been an (outstanding) biochemistry student with the second author, Diren. At the time of the interviews, he was a rising-senior, with an interest in chemistry education. He had often noticed, and at times felt frustrated by, his peers’ saying “I don’t know,” in preface to their expressing excellent ideas. During the course of this research, Mr Button saw experts’ doing something quite similar, which made him interested in what IDK was doing for them, and perhaps for his peers.

Diren is a senior lecturer in the Department of Chemistry, with 12 years of teaching experience, long-standing interest in developing curriculum and practices of instruction, and emerging interest in chemistry education research. For several years she has participated in The Listening Project (Tobin), which focuses on helping instructors listen to and interpret student reasoning, as evident in assignments and discussions in their classes, along with collaborating with the Institute for Research on Learning and Instruction (IRLI).

David is a professor in Education and Physics, one of the leaders of the Listening Project and of IRLI. His research has focused for many years on students’ learning in STEM, and he regularly teaches introductory physics. Much of his work has focused on how students frame what it is they are doing as learners (Hammer et al., 2005). In recent years he has come to focus attention on the role of student affect and social interactions on disciplinary engagement, both in his research and in his teaching (Jaber and Hammer, 2016; Watkins et al., 2018; Radoff et al., 2019; Appleby et al., 2021).

During a meeting of the Listening Project, discussing chemistry students’ thinking, Diren and David discovered that an apparently simple question was novel for chemists: Why doesn’t the molecule FBO exist naturally as a stable compound? At first, they saw providing an opportunity to involve students in inquiry without concern for their arrival at a canonical outcome. Diren posed the question in her course, saw that students were able to come up with ideas, and then became curious to see what experts would do. She tried it out informally with a colleague, found the conversation fascinating, and brought the idea to David. For him, the question was an opportunity to study expert reasoning on novel problems, analogous to the work we cite below. Working together in IRLI, Diren and David formulated the plan for a study and recruited John for an IRLI Summer Scholar undergraduate research assistantship.

The initial purpose was to study the strategies chemists use in reasoning about a novel problem. To that end, John and Diren recruited 14 PhD chemists to participate in one-on-one interviews lasting about 50 minutes. They asked each chemist to consider a hypothetical compound made up of three atoms, one each of fluorine, boron, and oxygen (FBO).

What stood out for John, though, was that every chemist said “I don’t know” (IDK), some many times. Counting later, he saw the range was from 2 to 38 times, with an average of 13 IDK statements over the course of a 50 minute interview. Finding experts doing something similar to what he had seen in his peers had John curious. Diren was also familiar with students’ saying IDK, when they responded to in-class questions. To her, it seemed natural: students are learning novel concepts, and therefore they may really “not know” yet. Seeing IDK come up so often for experts, she shared John's curiosity. John and Diren came to wonder, what does saying “I don’t know” (IDK) accomplish, for chemists or for students? This became our focus for research.

Research question: What does saying IDK accomplish for experts, as they reason about a novel problem?

The structure of this article

Overall, we found that the chemists in this study used IDK in both epistemic and affective ways. The epistemic roles of IDK were to mark moments they experienced a gap in their knowledge, often as a target for their searching and sensemaking, other times as a constraining boundary for what they expected to accomplish. The affective roles of an IDK statement seemed to be as shields from negative judgment, their own or the interviewer's, similar to the use John attributed to his fellow students.

We thus begin the next section by briefly reviewing prior research on expert practices of reasoning, including the studies that motivated our initial interests as well as work related to epistemic and affective aspects of not-knowing. We then turn to the methods of this study, including a description of how we recruited the participants, conducted the interviews, and carried out the analysis. Following the methods, we present our findings, with examples from interviews and some numerical summaries. In the conclusion section, we summarize the work and discuss possible implications for instruction and for further research.

Literature review

The initial motivation for this study was to study how expert chemists reason about novel problems, following prior work in other disciplines. We review that prior work here. We then turn to another line of previous research focused on students’ and scientists’ experiences of not-knowing. That research provides a theoretical framework to guide our analysis in this paper.

Studies of experts’ reasoning on unfamiliar problems

An extensive history of research exists on the nature of expertise, with famous examples in domains of chess, nursing, firefighting (Klein, 2017), and physics (Larkin et al., 1980; Chi et al., 1981a). Most of this work focused on the nature of expert knowledge, which scholars took to as the key distinction between experts and novices: “It has become increasingly clear in recent years that the quality of domain-specific knowledge is the main determinant of expertise in that domain” (Chi et al., 1981b). Much of that work concerned observations of experts’ solving problems that have familiar form and substance, connecting with their experience and intuition. Over the past several decades, attention in education has shifted from this view to include disciplinary practices (Council et al., 2012): Chemists, physicists, mathematicians are, in essence, professional learners, and their expertise manifests not only what they know but, also in how they approach learning.

There has also been important work in DBER to study expert reasoning in clinical interviews. These have been structured similar to the earlier studies (Larkin et al., 1980; Chi et al., 1981a), but with the key difference being that the researchers selected problems to be problematic: What do experts do when their “intuition fails” (Singh, 2002)?

Schoenfeld studied what experts did with problems that challenged them, finding evidence of heuristics (Pólya, 1985) and metacognitive monitoring and control, in contrast with what he observed in students from conventional math courses. Wineburg (Wineburg, 1991) studied how historians approached learning about events in areas outside their familiarity, again identifying heuristics that marked their expertise, in how they assessed evidence on their way to forming an understanding. Clement (Clement, 1994b) and Singh (Singh, 2002) gave physicists problems that fell outside their intuitive grasp, to see them enact strategies of generating analogies, thought experiments, mental simulations, and limiting cases.

We designed our study based on these prior approaches, using the FBO question we discovered chemists found problematic. All of our interviewees had an immediate intuitive response that the molecule could not be stable, but for everyone in this set, explaining why took some time and effort. In this sense, the question seemed to straddle the aspects of “fractionated expertise” (Kahneman and Klein, 2009), showing evidence both of expert knowledge and of expert grappling with novelty. In other work to follow, we will analyze the interview data in these respects, what the evidence shows about the chemists’ intuition as well as their heuristic strategies for learning. Here we focus on their recognition and experience of not-knowing.

Studies of not-knowing

The first author's initial interest in the data centered on how expert chemists seemed to say IDK in ways similar to students, as a kind of ‘shield’ against their own or others’ criticism. Discussing this perception with the IRLI summer scholars brought a connection in the literature: Conlin & Scherr (Conlin and Scherr, 2018) identified students’ “softening their stances through hedging, joking, quoting, and other shifts of footing,” moves they described as “epistemic distancing”:

“One of the possible functions of this distancing is for students to protect themselves from the affective risks of evaluating each other's ideas and having their own ideas critically evaluated.” (p. 22)

In their account, the need for this distancing can fade as students come to see not-knowing as low-risk, in the contexts of their work together. Our notion of shielding, which we describe below, connects with this perspective, here identifying many experts’ similar need to “protect themselves from the affective risk” of evaluation.

A great deal of writing about scientists has described the centrality of not-knowing. Firestein's (2012) popular account, Ignorance: How it Drives Science, for example, describes the pervasive and productive experience of not knowing for scientists (Firestein, 2012). He described it as “exhilarating,” and commented that it needs more attention in education:

“The undone part of science that gets us into the lab early and keeps us there late, the thing that “turns your crank,” the very driving force of science, the exhilaration of the unknown, all this is missing from our classrooms.” (p. 4)

Science education researchers have argued for helping students learn to engage with not-knowing. Manz & Suárez (Manz and Suárez, 2018), for example, argued for helping elementary school teachers learn to support their students’ engaging with uncertainty. Watkins et al. (Watkins et al., 2018) looked for common patterns across instances of students’ engagement in scientific inquiry and found every case included at least some of the participants’ positioning themselves as not-understanding.

Across these and other studies, the experience of not-knowing is both epistemic (as a metacognitive assessment of one's knowledge) as well as affective (connected with emotions). Jaber & Hammer (Jaber and Hammer, 2016) review accounts of not-knowing as pleasurable and motivating for scientists and mathematicians and presents evidence of “epistemic affect” in students. Radoff, Jaber and Hammer (Radoff et al., 2019) describe one student's shift in her experience of not-knowing, from intensely unpleasant and debilitating to pleasurable and motivating. This shift was coupled with a shift in her epistemological framing, that is her sense of what is taking place with respect to knowledge, in the introductory physics course: At first, she framed the work as applying algorithms provided by authority; by the end of the course she framed it as wondering about and making sense of phenomena.

The doing of science, of course, depends critically on not-knowing; often, the work for scientists is in determining and articulating precisely what they do not know (Phillips et al., 2017; Phillips et al., 2018). The studies we cited above (Schoenfeld, 1985; Clement, 1994a; Singh, 2002; Schoenfeld, 2016) all recount experts’ metacognitive awareness of not-knowing. On Firestein's (Firestein, 2012) and others’ accounts (Keller, 1983), scientists’ affective experiences of not-knowing have driven their work forward, including to overcome profound obstacles, whether technical or societal (Prescod-Weinstein, 2021). We are not aware of prior work showing that, for experts as for students, the experience of not-knowing can generate feelings of social-emotional risk.

Theoretical framework

In sum, looking across prior work, we adopt the following as elements of our theoretical framework:

• Expertise involves rich, domain-specific knowledge, which drives quick intuitive judgments.

• Expertise also involves metacognitive practices for monitoring, assessing, and refining knowledge, including for recognizing when quick intuitive judgments may need deliberate, effortful examination.

• What people perceive, how they reason, and how they experience not-knowing all reflect and influence how they frame what is taking place.

• Experience of not-knowing is both epistemic and affective.

These theoretical ideas guide our methods and analyses as we work to understand the research question that emerged from our first look at the data, “What does saying IDK accomplish for experts, as they reason about a novel problem?” Thus, we expect the chemists’ participation in the interview reflects their framing of what is taking place in it, including their experience of not-knowing. That experience will not be purely epistemic, which has us looking not only for the particular gaps or inconsistencies they see, but also for how the chemists manage the feeling that arise. In studying the data, we attended to evidence both of knowledge and metacognition. We turn now to describe the interviews and our methods of analysis.

Design & methods

Participants

We sent emails to 18 PhD chemists, all from the second author's professional network, asking them to participate in a study on how chemists reason about a novel problem. The group included people currently active in academic chemistry research, as well as people working in related fields in industry. Of the 18 invitations, 16 accepted; the two who declined did so for logistical reasons. One of the 16 was not available on a time frame for this study. Another found the interview uncomfortable, so they and the interviewer decided to stop. In total, 14 interviews comprised the data for this study.

The participants were almost all academics at various levels, from post-doc to tenured professors; one was from industry. Among the 14 participants, three described themselves as organic chemists, one a physical chemist, one an inorganic chemist, and another a chemical engineer. The rest did not describe a specific concentration. We withhold further details to avoid giving clues to the participants’ identities.

Interviews

The first author conducted 8 of the 14 interviews, the second author 6, all in the summer of 2021. Table 1 provides an overview of the interview data. We used Zoom to conduct and record the sessions, with participants’ consent as approved by the Tufts IRB.

Table 1 Summary of interview data. The interviewer was either John (J) or Diren (D)

	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10	#11	#12	#13	#14
Length (mins)	72	54	47	69	47	40	67	54	53	50	64	54	49	69
Interviewer	D	D	D	J	J	J	D	J	D	D	J	J	J	J
Total IDKs	13	3	8	2	38	4	6	19	7	17	14	37	6	7

---

The interviews were semi-structured, using “think-aloud” methods, a staple of research in the cognitive and learning sciences (Ericsson and Simon, 1998). Think-aloud methods have been used in chemistry education research to study student thinking (Bowen, 1994), and they were the approach in previous studies of expert thinking cited above. The timing of the study necessitated conducting the interviews remotely, by Zoom. With the participants’ permission, we recorded the interviews for subsequent analysis.

A feature of think-aloud methods in general, is that they make it possible to follow the participant's line of reasoning. That was important for this study, as the chemists’ strategies were different. For example, some chemists immediately reached out for a pencil and paper, some asked if they could use their iPad to do some drawing, and some others wanted to utilize ChemDraw (2012). Across the interviews, participants could draw on materials they had available in their offices, such as books and papers. All, of course, had immediate access to the internet, with the interviews taking place by Zoom, which let them, for example, search databases they frequently use on their computers. Since we wanted to observe the natural flow of the experts operating under uncertainty and wanted to witness their use of resources when solving the problem, we simply requested that they show what they were doing to the camera or share their screen.

The interviews all opened with a brief review of the purpose, emphasizing that we were interested in their thinking processes and that we were not looking for or even aware of any particular correct answer. We asked that they try to think aloud and posed the initial question.

“Let's imagine a compound that is made up of boron, oxygen, and fluorine. This compound only has three atoms, with a single atom from each element. How would you chemically describe it?”

We worked to conduct the interviews as naturally as possible, which meant informal presentation of the task as well as follow up questions and conversation. This meant the interviewer did not read the task from a page, so there were small variations in the precise wording. The remainder of the interview was responsive to the interviewee's responses. We had a general plan to move on to a second question, to say that this molecule does not exist in a stable form and ask the participant to consider why that might be. In one form or another, every interviewee expressed discomfort about the molecule, positing that it could not exist as described, that it must be unstable, or simply that it was strange. This discomfort made a natural flow for the interviewer to post the second question, to ask the expert about their initial sense.

Data analysis

We transcribed and deidentified every interview. As we noted above and explain further in our results, the phrase “I don’t know” (IDK) stood out. We decided to use that as a simple, systematic approach to selecting excerpts for analysis. Thus, our initial step in systematic analysis was to go through all the transcripts to mark IDK statements. To be clear, that participants said IDK was a marker for relevant moments, like a radioactive tag on a molecule. Of course, there are many other markers of uncertainty, including tone of voice, pauses of speech, facial expressions and/or, other phrases such as “I’m not sure.” For our purposes in this analysis, we chose to limit our selection of data to IDK, which had the virtues of being clear and prevalent across the full corpus.

From there, the first two authors, John and Diren, analyzed the selected data through methods of constant comparison and grounded theory (Henwood and Nick, 2003; Saldana, 2013; Mills and Gay, 2016). This began with each going through the IDK statements independently in a pass of open coding, to describe what each saw as the primary meaning and function of IDK for the participant. They then came back together to compare and connect their respective descriptions, to agree on an initial set of categories and give these categories names.

As suggested by Glaser and Strauss’ use of the term grounded theory, our coding scheme evolved in iteration with our applying it to the data (Glaser and Strauss, 1967). Thus, John and Diren returned to the data, using both the transcripts and the video, to code independently using the initial set of categories. They compared across their codings, considered agreements and disagreements, and revised the categories and names to a set of eight. Each returned to the data, coded independently, and found a 90% inter-rater reliability. We consider that sufficient agreement to support our qualitative claims regarding participants’ uses of IDK, but we do not make quantitative comparisons or claims of statistical significance (Hammer and Berland, 2014).

Having identified these 8 categories evident in the data, John and Diren presented their findings to the third author, David. This served as a “peer-debriefing” check of the analyses (Guba and Lincoln, 1989). With small exceptions, David could see the claims supported in the evidence. As well, he suggested connections to his own and others’ prior research on not-knowing. We drew on that work for the theoretical framework we presented above.

David raised questions about the original eight categories: One of the categories did not seem clear in the data, and some of the distinctions did not seem important for the purposes of this article.† We condensed the eight categories to five, summarized in Table 2.

Table 2 Summary of IDK codes

IDK categories	Brief explanation
Shielding	Protecting oneself against possible negative evaluation
Searching	Demonstrating engagement, or need for engagement in effortful reasoning
Reflecting	Assessing the past ideas and reviewing
Positioning	Identifying one's place in the community, e.g. as within a certain subdiscipline
Marking	Indicating a missing piece of information as a constraint

---

We now turn to present data and analyses from the study, beginning with where we began, with examples of experts’ uses of IDK as they appeared in interviews, then in the subsequent section presenting the five categories of our coding scheme.

Example episodes from chemistry experts

We begin with three excerpts from three different interviews, to show representative examples of the dynamics of how IDKs emerged. The participants varied significantly in the number of times they said IDK, from 2 to 38 times over the hour. The examples here are from (1) the chemist who said IDK 38 times, (2) 17 times, and (3) only 6 times. We chose the episodes where IDKs were concentrated in a short amount of time. As a set, they capture a representative range of what we saw across interviews in how chemists used IDK statements, as we discuss in sections to follow.

Participant #5

Participant #5 said IDK 38 times over the interview; only participant #12 was comparable (Table 1). Like most of the experts, #5 moved quickly at the start of the interview to draw the following Lewis diagram. We start the excerpt there. The times marked are from the beginning of the interview. Ellipses (“…”) indicate brief pauses of speech.

01 : 33 In principle, [drawing simultaneously] it's… something like this (Fig. 1).

Fig. 1 The first chemical structure drawn by participant #5.

Although… boron tends to make matrices, in truth. So, you end up with lots of these all matrixed together somehow, I would imagine… [pauses, raises shoulders and eyebrows]

John: Oh…yeah, so, what…I’m sorry, what's a matrix?

Well, I mean… So, like borane… it doesn't just exist as BH ₃ . It, it's see: boron has an empty—it only has three electrons, right? So, it makes these three bonds by sharing with these atoms. Those atoms are electronegative, so they can donate electrons back into boron to give it the full octet, but if you have a matrix, right, you can have, [attempts to draw up in the air first] you know this… I, I don’t know how to draw it… like this would be a linear structure, but then you would have to… um… A matrix has many of these that are, that are bonded together. So, [picks up the pen starts drawing] you know, oxygen would probably be the more likely donor of electrons [adds the electron pair on the existing structure's oxygen] (Fig. 2).

Fig. 2 Participant #5 adds electron pairs to the first structure as a start for drawing a more complex structure in the next figure.

So, I don't… you might end up with something like you know… [starts drawing a new structure] …And then another boron here… And another oxygen, and another fluorine… [adds and elongates the structure] but on and on and on… (Fig. 3)

Fig. 3 Participant #5 draws the new structures to build a more complex construct.

You’d have this all matrixed together. It's just the way… I, I would just imagine—and the reason is that boron, uh… even when it has two bonds, it doesn't have a complete octet. So, it wants, it wants to make more bonds, so this is actually a positive… [adds charges to the existing drawing] (Fig. 4)

Fig. 4 Participant #5 adds charges to the structure as a result of a mid-thought reasoning.

But, you know you don't have to have a full bond here, you can have a partial bond or you can just have some of your oxygen electron density in there, so it's… I don't know how it would look, in truth, but, I would imagine it's complicated and not just one, not, not just this [goes back to underline the first structure], floating around in space.” (Fig. 5)

Fig. 5 Participant #5 goes back to the first structure and reasons over why it may not exist as written.

John: Oh, okay that's interesting. So, it's not just going to exist in space by itself, because boron is, is, is wanting more electrons?

Yeah, it doesn't have a full octet. Like, here, it doesn't have a full octet, right? It only has six electrons in its shell. So, it needs more. It wants more. So, it will tend to complex with other [gestures with hands a coming-together action], other electron donors. And, the most, most likely electron donor here is an oxygen. So, you know, this may actually look like… [starts drawing another new structure] (Fig. 6)

Fig. 6 Participant #5 restarts another structure, this time to build further.

Oxygen, boron… fluorine… oxygen… You know, for boron, you may get a… how about something like this? [completes the structure] You get a six-member ring (Fig. 7).

Fig. 7 Participant #5 completes a complex structure of a six-membered ring.

Maybe, it lives like that. [Looks at the structure and starts speaking and adding to it again] So, you've got like partial positive charge here, and partial negative charge there. I don't know. I’m totally guessing… just wild, wild guess (Fig. 8).

Fig. 8 Participant #5 adds charges to the atoms of the six-membered ring and pauses, adding IDK to their conclusion.

For #5 as for others, IDK seemed to serve several roles. One was as a metacognitive flag, to mark something #5 saw as uncertain and needing thought. For example, the first utterance, “I don’t know how to draw it…” marked and led directly into their drawing a diagram. There was no significant pause in their speech, between #5's saying IDK and getting to work. Thus, we infer, #5 momentarily experiences not-knowing, which prompted their searching for possible solutions.

After mulling over the structures for about a minute, the participant said “I don't know how it would look, in truth, but, I would imagine…” We see this, again as a metacognitive flag, and, in this case a hedge on the idea they were about to express. We suggest this use is both epistemic and affective, as a kind of protection from expectations for their upcoming claim. That is, saying IDK helped #5 remain uncertain –making epistemic, affective, and perhaps social space for a tentative decision, much as Conlin & Scherr (Conlin and Scherr, 2015) described observing in students. Embracing this perceived not-knowing helps the expert to stay with the problem, continue to tackle it, and establish new ideas.

The IDK at the end of this episode reinforces our interpretation of this mix of roles, epistemic and affective. #5 has already gone through the thinking process, come up with a detailed chemical structure and an interesting proposal of a dynamic reaction mechanism for FBO. Yet, they chose to emphasize their uncertainty, “I don't know. I’m totally guessing just wild, wild guess.” From the evidence of their reasoning, it was not so much a “wild, wild guess.” Instead, #5 had a sense that their answer might be wrong, and felt a need to emphasize that sense.‡ Following Tannen (Tannen, 1993) we see the negative statement as evidence of the speaker's sense of an expectation that they should know. That is, we take this, and similar usages of IDK in other interviews, as evidence of the participant's framing, that they should be saying things they “know” in the interview. With IDK, they can shield against those expectations.

Participant #10

Participant #10, interviewed by Diren, also said IDK more than the average, 17 times over the hour. Several IDK's came in a cluster about 10 minutes after the start.

10 : 35 And so, and so it started, because I know how like boronates make, so make borons make boronate esters, I sort of know that there's at least, there could be two oxygens there. I kind of, I kind of started with like, “Okay, maybe there's a ring that's formed, that the boron comes through and, and, and bridges two oxygens together.” And so, so like what else is off the boron—I don't know. It probably needs a few more bonds to it, probably at least two (Fig. 9).

Fig. 9 Participant #10 uses descriptive hand gestures to explain the structure they imagined.

And then, you know, just thinking about like how to, how to, you know, if there's a ring there, like plopping a fluorine sort of one bond away from the oxygen, I don't know how to… That, without having any real reasoning whatsoever, that doesn't feel right to me. That feels like that would be very difficult to, to produce. And so, for me, I think, I would expect the fluorine to be further away. And so, and so I drew this (Fig. 10).

Fig. 10 Participant #10 shows the drawing of the structure that they previously described by gestures.

Diren (interviewer): Well, how cool is that!

Right. Right. So, there's the boron, there, bridging two oxygens. I don't know what else is going on up here. And then, the fluorine somewhere away from… what is here, so this could just be another methylene group, this could be an aryl group, this could be something else. But, it's not, it's not so direct and I don't, I don't know why I wouldn’t—I’m feeling that. Other than I know that the, the couple of times I’ve looked into literature of trying to site-specifically fluorinate anything the reactions are bananas. And I just… I just, you know? That's chemistry I’m… I’m uncomfortable with.

We found this particular moment especially complex and interesting. The first occurrence of IDK was unusual in the data set. It did not seem to be saying that this was something they personally did not know but rather that there were, in principle, multiple possible answers. That is, they were saying confidently, that something “else is off the boron,” and it could be many things. The participant describes the boron at the center of the hypothetical structure with their hands, making two bonds to two oxygens.

We arrived at this sense by analyzing both what #10 said, their diction, and by how they said it, their prosody. From their words, #10 was recounting how they had been reasoning (“I kind of started with like…”), rather than reasoning live. “What else is off the boron” was a rhetorical question, which they answered in a brusque, assertive tone, “I don’t know,” in the same way they might have said “there's no telling” or “it doesn’t matter.” What did matter was the need for more bonds, “probably at least two.” That is, #10 was using IDK to make an assertion about the chemistry, speaking in this instance with authority.

The next IDK was quite different, evidenced by the participant's diction, body language, and prosody. #10 said IDK very slowly, staring into space away from the interviewer, and without finishing the sentence: “I don't know how to…” They had started to present an idea, “plopping a fluorine sort of one bond away from the oxygen,” and the IDK here seems be a kind of narration of their current state, not knowing but simultaneously looking in their mind for something that might help. The search seemed difficult to narrate in words; perhaps it took place in images. When they spoke again it was to hedge, “without having any real reasoning whatsoever, that doesn’t feel right.”

They continued in a similar tone, explicitly marking their uncertainty and now, not-knowing, connected to statements about what they felt. They searched their mind for where they had seen fluorination reactions before, a move that seems to have emerged from their analysis of the structure they drew. IDK again marks uncertainty and their sense of incompleteness, which they seem to be trying to address at the time: “I don't know what else is going on up here. And then, the fluorine somewhere away from… what is here.”

Participant #13

Participant #13 was interviewed by John. They only used IDK six times during the interview. Two of those IDKs appeared close together at a moment when #13 was trying to make sense of two chemical formulae they had found in an online chemical database. At this moment, about 17 minutes into the interview, they were sharing their computer screen with the database pulled-up, showing the formulas F–B=O and (BOF)₃. The latter expression is of a trimer, meaning a compound with three identical units of BOF.

17 : 24 My first initial guess, and this is totally first-draft thinking, would be that… would…

I’m wondering if the borons are all in the center and the and so the borons are giving, are giving… the borons are just giving up their electrons to oxygen and fluorine so they're all positive. I don't know…

Why would it form a trimer? So, boron is giving up three electrons… But, I don't know. So, if they’re all… I mean you wouldn't think the positive sides would interact—electrostatically, they don't want to.

If it's just boron on one side and oxygen and fluorine on the other, and there's no reason to think of it would necessarily form a trimer. It could be anywhere from trimers to “N” with just, with just chains, if it's positive-negative, positive-negative. “Let's figure this out. Let's look up trifluoroboroxin.”

Participant #13's initial search in the database came out of their sense of not-knowing particular information they knew how to retrieve. Having found, at least partially, the information for which they were looking, #13 tried to make sense of why FBO should form trimers. IDK here follows the expert's reasoning about the boron's “giving up” electrons to oxygen, which they introduced with hedging—“this is totally first-draft thinking.” As in other examples, we see IDK as evidence of framing, similar to #5's marking a deviation from expectations of knowing. Similar to how we saw other participants employ the phrase, saying IDK seems to help this expert speak and think more freely than they could if they were constrained by expectations of knowing. That is, IDKs serve as the temporary shields against the default framing.

One could see these IDK's as interruptions of the expert's thought process, but the evidence suggests the contrary: They arrive at moments of especially rich idea-generation. The dual purpose of protecting while motivating is evident as well in the sentences following the second IDK: “Let's figure this out. Let's look up trifluoroboroxin.”

Categories of how chemists used IDK

In this section, we present categories that came out of our analysis of IDKs across the full data set, through qualitative methods of constant comparison and grounded theory (Henwood and Nick, 2003; Saldana, 2013). The question was novel for all participants, and everyone expressed not-knowing at some point. Table 1 gives a summary of the instances of IDK across the interviews.

The range in IDKs was from 2 to 38, with an overall average of 13. There are several likely contributions to the range. One is the chemists’ varying backgrounds: While they all had PhD's in chemistry or chemical engineering, they had a variety of current positions and specializations, none focused on boron chemistry. A second likely contribution is that some were interviewed by John, an undergraduate, and others were interviewed by Diren, a faculty member in chemistry: The average number of IDKs for John was 16 and the average for Diren was 9. We speculate that the difference between speaking with an undergraduate and speaking with a fellow chemist might affect participants’ framing what was taking place, including with respect to expectations of knowing.

Finally, as we noted in 3.3 above, it is clear people express not-knowing in many ways other than saying IDK, such as in other common phrases (“I’m not sure”), pauses of speech, gestures and facial expressions, as well as in idiosyncratic statements. One interviewee, for example, began by saying:

“Okay, I'll start out by saying I should have, should have remembered more of my inorganic chemistry.” (#14)

That is clearly an expression of not-knowing, one we would consider evidence of “shielding” as we describe below, but with our simple methodology it would not be included in the data unless it came near a statement of IDK. In further work, there will be value to developing a more thorough methodology, one that could allow quantification, but we expect that development will be difficult. Going forward, we see possibilities in collaborations with data scientists (Çınar et al., 2020; Jiang et al., 2020; Fussell et al., 2022).

For now, hoping in part to motivate those efforts, we refrain from quantitative analyses. We use IDK as a simple, systematic marker of relevant data for analyzing how chemists handle not-knowing, and we limit our claims to identification of functions in evidence. We do not doubt that other aspects of the data would serve a similar purpose. Table 2 summarizes the categories we identified in the meanings and functions of IDK for chemists across the interviews. Note these categories are not mutually exclusive; an individual use of IDK might well have more than one function.

Table 3 shows the coding for each interviewee. The counts reflect our interpretation of the primary or most obvious function of IDK in that moment. We did not code statements of IDK that did not evidently bear on chemistry, such as “I don’t know where I put my phone.” (See the section ‘Not coded’).

Table 3 Frequency of IDK code

IDK categories	Interviewees														∑
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10	#11	#12	#13	#14
Total	13	3	8	2	38	4	6	19	7	17	14	37	6	7	181
Shielding	2	1	1	0	11	2	0	10	0	4	3	13	0	1	48
Searching	6	0	2	0	12	0	2	3	2	6	1	5	2	1	42
Reflecting	2	1	0	0	0	0	0	1	1	2	1	2	0	1	11
Positioning	0	1	3	0	2	1	1	2	0	0	1	7	0	1	19
Marking	2	0	2	2	7	1	2	1	3	3	1	5	2	1	32
Not coded	1	0	0	0	3	0	1	2	1	2	6	1	1	2	20

---

Shielding

The first pattern that struck the first author, we noted above, was of the chemists saying IDK in ways that reminded him of other students in his classes: They would say IDK, but immediately continue to produce something valuable. John's first description was that saying IDK seemed like a “shield” that could prevent judgment about what the participant had said or was about to say. We have noted several instances we interpret as shielding in the examples above:

#5: “IDK. I’m totally guessing.”

#10: “IDK how to… That, without having any real reasoning whatsoever, that doesn't feel right to me.”

#13: “My first initial guess, and this is totally first-draft thinking […] IDK.”

In each of these instances, the chemist elaborated IDK with disclaimers. We suggest that their feeling the need for these disclaimers reflected their sense of tacit expectations that they should know, or that what they said would be taken as their answer. Shielding allowed them to consider an idea without the fear of judgement—by the interviewer and perhaps by themselves—if their idea later turns out to be wrong or incomplete. As Conlin & Scherr put it, the shield is “making space to sensemake” (Conlin and Scherr, 2018).

In some cases, the shielding came as a preparation for the idea to follow, perhaps to forestall further questioning. More often, IDK as a shield appeared after the chemist had presented a line of reasoning.

#12, for an additional example, drew a diagram and explained their reasoning about the bond angles: “Again, I know, boron makes, does funky bonding, but I didn't really like it. I thought this space, the space of this seemed little bit more reasonable, you know, electron-wise. Like, just gave it more space to, you know? I don't know. But that's 120 or something degrees and it gives it more space.” (Fig. 11)

Fig. 11 Participant #12 presents an improved electron arrangement in a newly drawn structure.

IDK here served as a kind of a flag, an alert, that the reasoning they had just developed is uncertain. It is clear the expert has knowledge; saying IDK allows them to use that knowledge to develop an idea, shielding them and it from an expectation of certainty.

Searching

This category of IDK usage was metacognitive: The participant said IDK as an explicit comment on their knowledge-state to indicate they were searching, or saw a need to search, within their minds for ideas that might related to the question. In our coding, we distinguished two forms: Concurrent, indicating they were in the process of searching, and cuing, indicating their sense of a need to search.

Participant #10's saying IDK above, “…like plopping a fluorine sort of one bond away from the oxygen, I don't know how to…” is an example of what we coded as concurrent searching, with the evidence of slow speech and gaze.

Another example was from participant #12. They had noticed an error in the formal charge they had assigned to tetravalent boron, which prompted the following:

“I was… didn't, I did, I did, I didn't count the electrons very well. And, I’m looking at here, yeah, so there's an extra hyd—there's an extra hydroxide, that's what it was. Yeah… [draws a new, correct structure] So, then this would again, I don’t know, what we’d have to here [draws the counter ion] maybe we could do a fluoronium? Ha! I don’t know. I mean, I’m making stuff up here. I don't know, but this could… [Continues to draw/work] you could do that, but it's unusual. I’d probably have, rather have sodium in there. Right? Rather than a fluoride.” (#12)

The participant said the first IDK very slowly, in the middle of expressing thought about the chemistry, evidence of concurrent searching, probably for what they could use as a counter-ion to balance charges. We coded the second IDK as shielding, interrupting the flow with “Ha” and following with the elaboration “I’m making stuff up here.” #12 then returned to working on the problem, with the third IDK again narrating their state of mind, as if to say, “working on it.” (Fig. 12)

Fig. 12 Participant #12's concurrent searching for ideas while using IDK.

In other instances, IDK seemed to signal the need to search. The participant would say IDK quickly, and then show signs of engagement in effortful thought. Participant #7 had started to consider trifluoroboroxine, in response to the initial question about FBO, and remarked that nature favors six-membered rings. When the interviewer asked why, the participant immediately responded with “I don’t know,” then worked to construct a line of reasoning:

“I don't know. I mean eventually you end up with … it's… Well, part of it, I think, has to do… with the bond geometries. So, if we think about the open and closed forms of glucose, and we'll just… [draws structures] for any generic pyranose sugar, doesn't really matter… Right? We lose entropy when we cyclize. But, we gain enthalpy because we're forming… essentially, two sigma bonds at the expensive of one pi bond—Oh, one Sigma bond, at the expensive of a pi bond. So, you have that enthalpic contribution that helps stabilize things.” (#7) (see Fig. 13)

Fig. 13 Participant #7's drawn explanation and proposals following quickly after IDK.

Here, saying IDK served as a kind of launch into the participant's generating several ideas, including an assertion that the bond geometries of this structure may be important in the creation of the different forms of this strange molecule. That is, here IDK seemed to mean “I need to think about that.”

We coded the following IDK from #11 as searching as well, but it was harder to decide whether to call it concurrent or cuing:

“I mean, I think that, like you know BF ₄ ⁻ , so that's like literally… I think I just don't really see boron with too many double bonds, in general. I’m just trying to think. I don’t know. But, even when you—like… [drawing BF₄⁻structure]. And this would have a formal negative charge. And then, when… I’m trying to remember boronic acids, but…” (#11) (Fig. 14)

Fig. 14 The idea of BF₄⁻ and its structure are produced by #11 after IDK.

Like a concurrent IDK, the statement came in the middle of a line of reasoning, accompanied by their saying “I’m just trying to think.” Here, #11 spoke it quickly, as if they took a quick break in the work of reasoning, said IDK, and then resumed.

In these ways, Searching IDK statements served chemists the metacognitive purpose of monitoring their epistemic state.

Searching and Shielding were the most evident functions for IDK in the data set, between them making up more than half of the codes.

Reflecting

In a relatively small number of instances, 11 across our data set, a participant used an IDK statement to pause and take stock of their thinking in the context of the interview, to assess what else they might like to contribute, or to check with the interviewer if they were meeting expectations. We called this function “reflecting.”

There were no instances of Reflecting in the example episodes. Here is one from Participant #2:

“So… I don’t know what I want to add here other than I think uh… my view of bonding is essentially based on just this idea that you’re bringing two things together—or three, or four…uh… But eventually, the attractive forces have to win out on the repulsive forces otherwise, you don’t get bonding. And, it's the same tr—the same is true even for ionic bonding—it's no different. Uh… and there, I always think about whether the ionization potential and electron affinity values tell you whether that's even feasible.” (#2)

The participant uses an IDK statement to think about what they would like to contribute to the interview. We note this instance might also serve as Searching, that in taking stock of whether they had given an adequate response for the interview, the participant was also noting a need to search for other possible relevant knowledge. In fact, this brought the participant to a grander discussion of bonding principles.

Another example was in Participant #11's interview, following their marking a piece of information as missing, another function of some IDKs we describe below. Here, #11 used IDK to reflect on what might be helpful in the context of the interview:

“It'd be helpful if I remembered actually how to determine formal charge—which I don't feel like—I mean, I can look up, if you need me to, but I really don't know if it'll be super helpful, for me.” (#11)

#11 first expressed not knowing “how to determine the formal charge,” but the IDK was specifically to express their doubts over whether or not the addition of formal charges is worth pursuing in the context of the interview.

Positioning

We called it “positioning” when a participant used an IDK statement to delineate their expertise in chemistry. This was often connected with their identifying other chemists to whom they would turn for help.

Participant #8 for example commented:

“…Oh, I’ve also seen boron clusters, but, that is not my specially because I’m more of an organic chemist rather than an inorganic chemist, so I don't know too much about like boron-boron bonds and like a boron double bonded to a boron that, for me, that was weird.” (#8)

The participant uses IDK to say that they are unfamiliar with boron chemistry, because they are an organic chemist, implying that an inorganic chemist might know more. Here and in other instances, we saw Positioning IDKs also serving the function of Shielding, protecting themselves from expectations. We distinguish Positioning as a particular function of chemists expressing their own knowledge of their expertise in relation to others’.

Marking

Finally, participants used IDK statements simply to mark information they were missing, and that they did not expect they could find by searching in their minds. It was an example of Marking when #10 said, “What else is off the boron—I don’t know,” in example episodes above. In that moment, they were marking information they did not know but deemed irrelevant. Another example was in #5's interview, later than the excerpt we quoted above, when they said IDK about information that might be relevant (Fig. 15).

Fig. 15 The bonds of F–B and B=O in participant #5's drawing.

“But, I don't know. I’d have to look up the bond energies.” (#5)

Another example of Marking was #3's IDK, about whether a bond in FBO is known by chemists to exist:

“I mean one thing that would be… [pauses, thinks with hand on chin] I mean, is the boron double bond oxygen a known bond at all? I don't know that… [thinks just briefly] But, is there a bond strength that's generally associated with it?” (#3) (Fig. 16)

Fig. 16 B=O double bonds and the chemical scheme participant #3 is working on at the time of IDK. This participant deliberately drew the top structure in red and the bottom in black.

Participant #3 chose to not to dwell on the exact bond strength of B=O. After remarking that they do not know, they moved rather quickly to explore another relevant bond strength that could be “associated with it”. This expert used the factual IDK productively to explore analogic solutions of the specific bond strength of an unfamiliar bond.

Factual IDK statements seem to allow experts to address particular details within their lines of reasoning. The ideas broached by factual IDK statements were not enough to stop experts from reasoning, whereas it might be for students. Understanding the transition between this sort of comfort of admitting to “not knowing” and “still being productive” should be the subject of future work.

Not coded

Finally, for completeness, we note that participants said IDK, so we counted the instance by our methodology, but the function of that IDK was irrelevant. For example, when #5's iPad left the Zoom call, they commented “I don't know why it dropped off.” Not including inconsequential IDK statements, our participants spoke between 2 and 35 IDK statements with an average of 12 IDKs across the fourteen interviews.

Limitations

There are several limitations to this study, as we have also noted in the methods section above. One is that we solely focused on expert chemists’ use of the phrase “I don’t know.” That made for a simple method of finding moments of not knowing, but, of course, we recognize there is a plethora of other phrases that may convey the same meaning. That may be especially important for non-native speakers of English. Further work could expand the data analyses to investigate other phrases as well as non-verbal clues of not-knowing.

In addition, having two different researchers interview the participants may have influenced the participants’ framing of expectancies, as noted above.

In our next iteration of this study, we will build member check-ins into our methods, which would allow us an additional mechanism of validating or possibly disconfirming our interpretations. It is possible that in some instances, for examples, we have misinterpreted what a participant meant to be saying. Member check-ins will be especially important as we deepen our study of the particular and varying dynamics within individuals’ reasoning and experience. That said, looking across the evidence, we argue that the data we have from this study supports the core claims we are making, that expert chemists say IDK to serve various functions including Shielding and Searching as we described above and review below. We note as well that some interviews included the sort of evidence member check-ins would provide, for instance, when participants reflected explicitly on their uncertainty. That is how the “reflecting” category emerged.

Finally, we do not claim that our list of functions is exhaustive. Our sample set of fourteen experts merged on some clear ways of using IDK. With the current data, there is no knowing if a larger and perhaps more representative sample of chemistry experts would change the number or the types of these categories.

Conclusions, discussion, and implications

Conclusions

The core questions in CER have concerned understanding how to help students make progress toward expert knowledge in chemistry (Stains and Talanquer, 2008; Cooper and Stowe, 2018). We had clear evidence of some of that knowledge in our interviews: Every participant had an immediate sense of FBO as a strange idea for a molecule. Like accounts of expertise in other fields (Klein, 2009; Klein, 2017), they had intuitive knowledge readily available that let them see quickly that something was wrong. It would be interesting to study the nature of that knowledge, to build on other empirical studies (Kozma and Russell, 1997; Heyworth, 1999; Stieff and Raje, 2010) of chemists’ expertise.

Our focus here, however, has been on chemists’ not-knowing, which was also clearly in evidence across these interviews. None of the participants had a ready answer to the question of why FBO felt so strange—why it would not form as a stable molecule. That is, our focus has been on what chemists know about and how they experience the state of not-knowing.

Of course, chemists are familiar with that state. The ordinary experience of a chemist, doing chemistry, is not-knowing; chemists are professional learners. In our original design of these interviews, we set out to study the reasoning processes of chemists when presented with an unfamiliar problem, emulating previous work in other disciplines including mathematics (Schoenfeld, 2016), physics (Clement, 1994b), and history (Wineburg, 1991). In this article, we have focused on the social-emotional and epistemic experience for chemists of not-knowing. We identified moments to study with the very simple method of searching transcripts for the words “I don’t know” (IDK), and for each instance we analyzed what saying IDK seemed to accomplish for the chemist.

Shielding against expectations

Our first finding is that one of the main functions of IDK was what we call Shielding, in particular against what the chemists sensed as an expectation that they should have an answer to our question. Most participants—10 out of 14—said “I don’t know” at least once with the primary function of Shielding. In this, the experts used of IDK was similar to what Conlin & Scherr (Conlin and Scherr, 2018) reported in physics students, who needed to “make space to sensemake.” IDK played that role for chemists here, freeing them to consider ideas and possibilities.

There was appreciable variation among the chemists with respect to this function, from person to person as well as between John (an undergraduate) and Diren (a faculty member) as interviewers. There may also have been differences related to participants’ backgrounds and current positions that may certainly bear further study.

For now, we are hesitant to draw conclusions about individuals that depend on quantifying our coding, for two main reasons. The first is the simplicity of our methods, analyzing only moments when participants say IDK. It is an obvious next step to look for other articulations of not-knowing—“I’m not sure,” “Beats me,” and so on—which different people may use more or less often, in general or for perceptions of the context.

More important, we suggest that what is most significant about this first finding is not what it says about the individual participants but what it says about the context of academia. Taking the set of chemists as informants, the evidence points strongly toward a shared, default expectation of knowing; not-knowing can be embarrassing. Prior work has identified students’ expectations and discomfort with confusion and uncertainty (Chen, 2020; Chen and Qiao, 2020; Phillips et al., 2021); to our knowledge this is the first evidence that experts can have similar expectations and discomfort.

Metacognitive monitoring

Our second finding is that chemists used IDK as an expression of metacognitive monitoring and control, similar to aspects what Schoenfeld (Schoenfeld, 1985; Schoenfeld, 2016) found in expert mathematicians. Participants thus used IDK to flag aspects of what they were saying or thinking as uncertain or incomplete, often in conjunction with Shielding against tacit expectations. Often this flag cued them to begin a search in their minds for possibly relevant information or ideas, which we called Cuing as one of the forms of Searching. Sometimes they used IDK to narrate their current state of trying to work something through, which we described as Concurrent Searching. Sometimes, too, their monitoring allowed them to decide that they were missing information, deciding to proceed without it or to consult with another chemist or reference, in a manner we called Marking.

This was another strong pattern, evident in 13 out of the 14 interviews of the participant chemists using IDK at least once as a form of metacognitive expression. As others found in previous studies of experts in other fields (Schoenfeld, 1985; Wineburg, 1991; Clement, 2012; Schoenfeld, 2016), the chemists in this study showed expertise in how they monitored for and recognized what they did not know. During their interview, participant #5 put it this way: “You got to know what you don’t know.”

IDK helped chemists refrain from settling on an answer; it gave them freedom to generate and to consider a variety of ideas, to rethink and unstick themselves, to prolong their persistence in reasoning.

Not-knowing, across participants, is part of doing science, and part of their expertise. No chemist knew, at the outset of the interview, why FBO would not exist as a stable molecule (although every chemist at least suspected it would not). But having seen they do not know, every chemist persisted in reasoning about the question.

Discussion

In CER, as in other STEM education research, conceptualization and research on expertise has focused almost exclusively on knowledge (Cooper and Stowe, 2018). Following work in other disciplines (Schoenfeld and Herrmann, 1982; Schoenfeld, 1985; Clement, 1994a; Singh, 2002; Randles and Overton, 2015) and shifting attention toward disciplinary practices as essential objectives, we set out to study the expertise of chemists in grappling with what they do not know. Thus, we chose a question that was likely to be unfamiliar, and indeed it was.

It is clear that for students (Manz, 2015; Manz and Suárez, 2018; Watkins et al., 2018; Sundstrom and Cardetti, 2021) as for scientists (Firestein, 2012), not-knowing is a staple of experience, which motivates inquiry and underlies the practices in the discipline. We had evidence in this study, of not-knowing motivating the participants to engage in extended sensemaking—in fact, many of the participants stayed longer than the appointed time (50 minutes for the interview), and some followed-up later over email with more ideas. Students and chemists have this in common: Their primary occupation is to learn.

What our interviews showed was that students and chemists also have in common a discomfort with not-knowing, with the possibility of being judged wrong. Several accounts in college physics have described students needing to manage that discomfort, in collaborative groups (Conlin and Scherr, 2018) or in personal study (Radoff et al., 2019). In retrospect, perhaps we should not have been surprised to see similar discomfort for the PhD chemists in our study; it is easy for us to empathize, imagining ourselves as research participants in these interviews, feeling like we should know. In the contexts of these interviews, which sit within the context of university learning and teaching, there is an expectation of knowing.

That expectation, associated with formal learning and teaching in school, contrasts with the contexts of chemists’ professional lives, in which not-knowing is central and productive. There are similar concerns raised in other areas. Ilgen et al., (Ilgen et al., 2019) for example, discussed the importance of medical clinicians having “comfort with uncertainty”—and we note that a significant fraction of students enrolled in chemistry courses do so as part of preparation for careers in medicine. For us, the findings raise the questions of how chemistry education might expand and shift its objectives, not only to impart knowledge but to help cultivate the expertise of not-knowing.

Implications for classroom learning

There is growing interest in science instruction helping students to develop disciplinary abilities and inclinations for grappling with uncertainty, mainly as part of growing interest in their development in disciplinary practices of science (Council et al., 2012) “Asking questions” is widely recognized as a practice in doing science (Phillips et al., 2017); there are arguments as well for problematizing as “the work of identifying, articulating, and motivating” a “gap of understanding” (Watkins et al., 2018). Learning to do chemistry means learning to engage with not-knowing.

Almost all chemistry instruction, including almost all assessment of student learning, has focused on what students know. Only recently have courses begun to adopt for example, the use of ill-structured problems (Jonassen, 1997; Landa et al., 2020; Yeong, 2021) to promote students’ more sustained, engagement. This study supports the need for that shift of pedagogy and emphasis, in evidence of how experts’ recognition of not-knowing motivates and guides their engagement. At the same time, it shows evidence of how the shift will be challenging to enact.

Discomfort with uncertainty seems to be a widespread phenomenon in the community and institution. Most of the chemists in these interviews felt the need to “make space to sensemake” (Conlin and Scherr, 2015), against a general expectation of knowing. To be clear, we do not see this as a feature of individuals in our study. Rather, that the phenomenon of shielding is evident across members of the community suggests it reflects the broader community: in the contexts of learning and teaching at the university, not-knowing is an embarrassment. The study participants, like college students, have all engaged, for many years, in a system that frames education as a transfer of knowledge to learners.

We suggest that this emphasis on knowledge transfer has important implications for chemistry education reform. In a famous essay, The culture of education, Bruner (Bruner, 1996) quoted the African proverb: “The fish will be the last to discover water.” The success of reform efforts to support students’ developing comfort with uncertainty may depend on educators and learners seeing the water. That is to say, the framing is pervasive that confusion and uncertainty are troubles to be resolved, rather than opportunities for intellectual activity. The change we envision will need learning at the scale of the institution, by community as a whole. In other words, chemistry education reform needs to consider how to shield the classroom from the surrounding expectations of certainty.

Thus, we suggest two sorts of implications for chemistry teaching. At one level, there is work for reform in the substance of curriculum, to incorporate ill-structured, open-ended questions that do not have single correct answers as targets, like our FBO question. Such reforms allow instructional attention to students’ developing the sorts of strategies that contribute to expertise. At another level, we argue for attention to the “feelings” of not-knowing as inherent in doing science (Jaber and Hammer, 2016), which apparently need shielding. This work is in instructors’ tuning-in to and valuing expressions of uncertainty and confusion, supporting students’ positioning themselves as not-knowing (Watkins et al., 2018).

This, our findings suggest, will entail educators’ learning to see—and guide students to see—confusion and uncertainty as opportunities. The discovery of problems and the formation of questions should be seen as credit-worthy accomplishments. Indeed, the utility of “not knowing,” in this study, lay in the experts’ shared abilities to continue working on a novel problem. Instruction should encourage students to do the same. For students as for experts, this will involve attention to a mix of epistemic and social-emotional aspects of the experience. To that end, an immense value exists in listening to one another. Hearing “IDK” and understanding its utility and plasticity will help students and experts alike to engage with challenging material and to support each other in further inquiry.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

The authors of this paper would like to thank Dr Milo Koretsky for his suggestions, Dr Kristin Wendell, the participants who were all generous with their time, Dr Ira Caspari and her research team in the Tufts Chemistry Department for their help with interview strategies, and Dr Desen Ozkan for her comments on the manuscript. Certainly, too, we are grateful to all of the participants for their generosity with their time. This project was funded by the Tufts University Institute for Research on Learning and Instruction (IRLI).

References

1. Allison J., (2001), A Model for Substantial Deviations from the Traditional Lecture Format for Graduate and Upper-Level Undergraduate Courses in Science—Lecture and Learning Classes, J. Chem. Educ., 78, 965.
2. Appleby L., Dini V., Withington L., LaMotte E. and Hammer D., (2021), Disciplinary significance of social caring in postsecondary science, technology, engineering, and mathematics, Phys. Rev. Phys. Educ. Res., 17, 023106.
3. Ball L. J., Ormerod T. C. and Morley N. J., (2004), Spontaneous analogising in engineering design: a comparative analysis of experts and novices, Des. Stud., 25, 495–508.
4. Bowen C. W., (1994), Think-Aloud Methods in Chemistry Education: Understanding Student Thinking, J. Chem. Educ., 71, 184–190.
5. Bruner J. S., (1996), The culture of education, Cambridge, Mass: Harvard University Press.
6. Caspari I., Kranz D. and Graulich N., (2018a), Resolving the complexity of organic chemistry students' reasoning through the lens of a mechanistic framework, Chem. Educ. Res. Pract., 19, 1117–1141.
7. Caspari I., Weinrich M. L., Sevian H. and Graulich N., (2018b), This mechanistic step is “productive”: organic chemistry students' backward-oriented reasoning, Chem. Educ. Res. Pract., 19, 42–59.
8. ChemDraw Professional, ver. 22.2.0.3348, Perkin Elmer Inc.
9. Chen Y.-C., (2020), Dialogic Pathways to Manage Uncertainty for Productive Engagement in Scientific Argumentation: A Longitudinal Case Study Grounded in an Ethnographic Perspective, Sci. Educ., 29, 331–375.
10. Chen Y.-C. and Qiao X., (2020), Using students' epistemic uncertainty as a pedagogical resource to develop knowledge in argumentation, Int. J. Sci. Educ., 42, 2145–2180.
11. Chi M. T. H., Feltovich P. J. and Glaser R., (1981a), Categorization and representation of physics problems by experts and novices, Cognit. Sci., 5, 121–152.
12. Chi M. T. H., Glaser R. E. and Rees E., (1981b), Expertise in Problem Solving.
13. Çınar A., Ince E., Gezer M. and Yılmaz Ö., (2020), Machine learning algorithm for grading open-ended physics questions in Turkish, Educ. Inform. Technol., 25(5), 3821–3844.
14. Clement J., (1994a), Imagistic simulation and physical intuition in expert problem solving, Proceedings of the 16th Annual Conference of the Cognitive Science Society, pp. 201–206.
15. Clement J., (1994b), Use of physical intuition and imagistic simulation in expert problem solving, Implicit Explicit Knowl., 204–244.
16. Clement J. J., (1998), Expert novice similarities and instruction using analogies, Int. J. Sci. Educ., 20, 1271–1286.
17. Clement J., (2008), Creative model construction in scientists and students: the role of imagery, analogy, and mental simulation, Dordrecht: Springer.
18. Clement J., (2012), Informal Reasoning and Education, pp. 345–362.
19. Clement J. J., (2013), Parallel roles for nonformal reasoning in expert scientific model construction and classroom discussions in science, 2013 Annual Meeting of the National Association for Research in Science Teaching, University of Massachussetts, Amherst.
20. Collins A., Brown J. S. and Newman S. E., (2018), Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser, pp. 453–494.
21. Conlin L. and Scherr R., (2015), Making Space to Sensemake: Epistemic Distancing in Small Group Physics Discussions.
22. Conlin L. D. and Scherr R. E., (2018), Making Space to Sensemake: Epistemic Distancing in Small Group Physics Discussions, Cogn. Instr., 36, 396–423.
23. Cooper M. M. and Stowe R. L., (2018), Chemistry Education Research From Personal Empiricism to Evidence, Theory, and Informed Practice, Chem. Rev., 118, 6053–6087.
24. Council N. R., Education D. O. B., Social Sciences A., Education B. O. S., Standards C. O. A. C. F. F. N. K. S. E., Keller T., Quinn H. and Schweingruber H., (2012), A framework for K-12 science education: Practices, crosscutting concepts, and core ideas, Washington, DC: National Academies Press.
25. Cross N., (2001), Elsevier Science, pp. 79–103.
26. Erduran S., (2019), Argumentation in Chemistry Education: Research, Policy and Practice.
27. Ericsson K. A. and Simon H. A., (1998), How to Study Thinking in Everyday Life: Contrasting Think-Aloud Protocols With Descriptions and Explanations of Thinking, Mind, Culture, Activity, 5, 178–186.
28. Firestein S., (2012), Ignorance: How It Drives Science, Ignorance: How It Drives Science.
29. Flood V. J., Amar F. G., Nemirovsky R., Harrer B. W., Bruce M. R. M. and Wittmann M. C., (2015), Paying Attention to Gesture when Students Talk Chemistry: Interactional Resources for Responsive Teaching, J. Chem. Educ., 92, 11–22.
30. Fussell R., Mazrui A. and Holmes N., (2022), Machine learning for automated content analysis: characteristics of training data impact reliability.
31. Glaser B. G. and Strauss A. L., (1967), The discovery of grounded theory: strategies for qualitative research, Chicago: Aldine Publishing.
32. Guba E. G. and Lincoln Y. S., (1989), Fourth generation evaluation, Thousand Oaks, CA, US: Sage Publications, Inc.
33. Hammer D. and Berland L. K., (2014), Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning, J. Learn. Sci., 23, 37–46.
34. Hammer D., Elby, A., Scherr, R. E. and Redish, E. F., (2005), in Mestre J. (ed.), Transfer of Learning from a Modern Multidisciplinary Perspective, Greenwich, CT: Information Age Publishing, pp. 89–119.
35. Henwood K. L. P., Nick F., (2003), Qualitative research in psychology: expanding perspectives in methodology and design, 1st edn, Washington, DC: American Psychological Association, pp. 131–155.
36. Heyworth R. M., (1999), Procedural and conceptual knowledge of expert and novice students for the solving of a basic problem in chemistry, Int. J. Sci. Educ., 21, 195–211.
37. Ilgen J. S., Eva K. W., de Bruin A., Cook D. A. and Regehr G., (2019), Comfort with uncertainty: reframing our conceptions of how clinicians navigate complex clinical situations, Adv. Health Sci. Educ., 24, 797–809.
38. Jaber L. Z. and Hammer D., (2016), Learning to Feel Like a Scientist, Sci. Educ., 100, 189–220.
39. Jeppsson F., Haglund J. and Amin T. G., (2015), Varying Use of Conceptual Metaphors across Levels of Expertise in Thermodynamics, Int. J. Sci. Educ., 37, 780–805.
40. Jiang R., Gouvea J., Hammer D. and Aeron S., (2020), Automatic coding of students' writing via Contrastive Representation Learning in the Wasserstein space.
41. Jonassen D. H., (1997), Instructional Design Models for Well-Structured and Ill-Structured Problem-Solving Learning Outcomes, Educ. Technol. Res. Devel., 45, 65–94.
42. Kahneman D. and Klein G., (2009), Conditions for Intuitive Expertise: A Failure to Disagree, Am. Psychol., 64, 515–526.
43. Keller E. F., (1983), A feeling for the organism: the life and work of Barbara McClintock, San Francisco: W.H. Freeman.
44. Klein G. A., (2009), Streetlights and shadows: searching for the keys to adaptive decision making, Cambridge, MA: MIT Press.
45. Klein G. A., (2017), Sources of Power, 20th Anniversary Edition: How People Make Decisions, Cambridge, United States: MIT Press.
46. Kozma R. B. and Russell J., (1997), Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena, J. Res. Sci. Teach., 34, 949–968.
47. Landa I., Westbroek H., Janssen F., van Muijlwijk J. and Meeter M., (2020), Scientific Perspectivism in Secondary-School Chemistry Education: Integrating Concepts and Skills in Chemical Thinking, Sci. Educ., 29, 1361–1388.
48. Larkin J., McDermott J., Simon D. P. and Simon H. A., (1980), Expert and novice performance in solving physics problems, Science, 208, 1335–1342.
49. Latour B. and Woolgar S., (2013), Laboratory life: The construction of scientific facts.
50. Longino H. E., (1990), Science as social knowledge: values and objectivity in scientific inquiry, Princeton, N.J.: Princeton University Press.
51. Manz E., (2015), Resistance and the Development of Scientific Practice: Designing the Mangle Into Science Instruction, Cogn. Instr., 33, 89–124.
52. Manz E. and Suárez E., (2018), Supporting teachers to negotiate uncertainty for science, students, and teaching, Sci. Educ., 102, 771–795.
53. Melanie M. C., Marcos D. C., Diane E.-M., Cori L. F.-H., Sarah E. J., Joseph S. K., James T. L., Rebecca L. M., Lynmarie A. P. and Sonia M. U., (2015), Challenge faculty to transform STEM learning: Focus on core ideas, crosscutting concepts, and scientific practices, Science (Am. Assoc. Advancement Sci.), 350, 281–282.
54. Mills G. E. and Gay L. R., (2016), Educational research: competencies for analysis and applications, London: Pearson Education.
55. Minhas P. S., Ghosh A. and Swanzy L., (2012), The effects of passive and active learning on student preference and performance in an undergraduate basic science course, Anat. Sci. Educ., 5, 200–207.
56. National Research Council (NRC), (2012), A framework for K-12 science education: Practices, crosscutting concepts, and core ideas, The National Academies Press.
57. Nersessian N. J., (1992), How do scientists think? Capturing the dynamics of conceptual change in science, Cogn. Models Sci., 15, 3–44.
58. Phillips A. M., Watkins J. and Hammer D., (2017), Problematizing as a scientific endeavor, Phys. Rev. Phys. Educ. Res., 13, 020107.
59. Phillips A. M., Watkins J. and Hammer D., (2018), Beyond “asking questions”: Problematizing as a disciplinary activity, J. Res. Sci. Teach., 55, 982–998.
60. Phillips A. M., Sundstrom M., Wu D. G. and Holmes N. G., (2021), Not engaging with problems in the lab: Students' navigation of conflicting data and models, Phys. Rev. Phys. Educ. Res., 17, 020112.
61. Pickering A., (1993), The Mangle of Practice: Agency and Emergence in the Sociology of Science, Am. J. Soc., 99, 559–589.
62. Pólya G., (1985), How to solve it; a new aspect of mathematical method, Princeton, N.J.: Princeton University Press.
63. Prescod-Weinstein C., (2021), The disordered cosmos: a journey into dark matter, spacetime, and dreams deferred, New York: Bold Type Books.
64. Radoff J., Jaber L. Z. and Hammer D., (2019), “It's Scary but It's Also Exciting”: Evidence of Meta-Affective Learning in Science, Cogn. Instr., 37, 73–92.
65. Randles C. A. and Overton T. L., (2015), Expert vs. novice: approaches used by chemists when solving open-ended problems, Chem. Educ. Res. Pract., 16, 811–823.
66. Roberts D., (2017), Higher education lectures: From passive to active learning via imagery? Active Learn. Higher Educ., 20, 63–77.
67. Saldana J., (2013), The coding manual for qualitative researchers, Los Angeles, Calif., London: SAGE.
68. Schoenfeld A. H., (1985), Mathematical Problem Solving, Academic Press.
69. Schoenfeld A. H., (2016), Learning to Think Mathematically: Problem Solving, Metacognition, and Sense Making in Mathematics (Reprint), J. Educ., 196, 1–38.
70. Schoenfeld A. H. and Herrmann D. J., (1982), Problem perception and knowledge structure in expert and novice mathematical problem solvers, J. Exp. Psychol.: Learn., Memory, Cogn., 8, 484–494.
71. Sevian H. and Talanquer V., (2014), Rethinking chemistry: a learning progression on chemical thinking, Chem. Educ. Res. Pract., 15, 1–23.
72. Singh C., (2002), When physical intuition fails, Am. J. Phys., 70, 1103–1109.
73. Stains M. and Talanquer V., (2008), Classification of chemical reactions: Stages of expertise, J. Res. Sci. Teach., 45, 771–793.
74. Stieff M. and Raje S., (2010), Expert Algorithmic and Imagistic Problem Solving Strategies in Advanced Chemistry, Spatial Cogn. Comput., 10, 53–81.
75. Sundstrom M. and Cardetti F., (2021), Exploring the introductory physics classroom through the lens of intellectual humility: Handling what you do not know, Phys. Rev. Phys. Educ. Res., 17, 020135.
76. Talanquer V. and Pollard J., (2010), Let's teach how we think instead of what we know, Chem. Educ. Res. Pract., 11, 74–83.
77. Tannen D., (1993), in Tannen D. (ed.), Framing in Discourse, New York, NY: Oxford University Press, pp. 14–56.
78. Tobin R., (PI) The Listening Project, Funded at Tufts University by the Howard Hughes Medical Institute.
79. Watkins J., Hammer D., Radoff J., Jaber L. Z. and Phillips A. M., (2018), Positioning as not-understanding: The value of showing uncertainty for engaging in science, J. Res. Sci. Teach., 55, 573–599.
80. Wineburg S. S., (1991), Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence, J. Educ. Psychol., 83, 73–87.
81. Yeong F. M., (2021), Using asynchronous, online discussion forums to explore how life sciences students approach an ill-structured problem, Teach. Learn. Inquiry, 9, 138–161.

---

Footnotes

† The initial eight categories included one of “validating,” but in our peer-debriefing it was difficult to support as distinct from “reflecting.” The eight also included three versions of “shielding,” distinguished by timing: in advance of an explanation, during or after. The timings suggested subtle differences of function, but explaining those differences was making the manuscript too long and complex. We decided the important claim to support is that of the combined category shielding, which we describe below.

‡ Interestingly, this trimeric FBO matrix model described by the participant has actually been synthesized.

---