Exploring the shift to online open-book final exams: the impact on question types and the experiences of students and academics

Abstract

The COVID-19 pandemic prompted the shift to online learning, including a change from paper-based (Type-P) exams to online, open-book (Type-O) exams. This study investigated the impact of the sudden shift from Type-P to Type-O exams on the nature of exam questions, and the students’ and academics’ experiences. Type-P and Type-O exams were analysed, focussing on question type, and Bloom's Taxonomy classifications. Type-O exams had a decrease in drawing questions in favour of short answer questions, and a slight shift to lower-order thinking was seen. Other changes were mostly insignificant. Semi-structured interviews with students revealed the main origins of stress for Type-O exams related to technological failures. Students noted they prepared notes for ease of searching for Type-O exams, and optimal memorisation for Type-P exams. Students who had taken both exam types revealed a preference for Type-P exams, as they preferred drawing answers. Semi-structured interviews with academics revealed that writing questions for Type-O exams required more thought to avoid questions with searchable answers. However, academics enjoyed the conveniences of the online format, which includes the automatic marking of multiple-choice questions. Academics appeared to have an astute awareness of the students’ experience with the differing exam types. Our findings suggest that Type-O exams could be successfully integrated into university systems, however there are important considerations that should be addressed.

---

Introduction

As identified by Archer's (2017) assessment purposes triangle, assessments typically serve three main purposes:

• identifying knowledge gaps allowing students to re-evaluate their learning strategies and teachers to adapt their teaching strategies;

• providing evidence of learning to hold universities accountable;

• and certifying the student's attainment of knowledge and skills.

As such, it is important that assessments accurately reflect a student's understanding of course content in order to cater to these purposes. In general, there are two main types of assessments:

• formative assessments such as ongoing weekly quizzes;

• and summative assessments such as final examinations (Williams and Wong, 2009).

Traditionally, final examinations are closed-book and paper-based and are usually completed in-person under invigilator supervision (herein referred to as Type-P) (Williams and Wong, 2009; Nsor-Ambala, 2019). However, due to the COVID-19 pandemic, students could no longer attend in-person exams. Instead, exams shifted to formats that did not require face-to-face contact, such as online written exams (Dicks et al., 2020). Due to the remote nature of these exams, many were transitioned to online open-book (herein referred to as Type-O), granting students access to notes and textbooks, with some allowing open-internet access. The risk for academic dishonesty prompted universities to implement online invigilation services, such as ProctorU (Wuthisatian, 2020; Vazquez et al., 2021). However, students often raise concerns for safety and privacy around the use of external proctoring companies (Petillion and McNeil, 2020).

Question types in final exams

Final exams generally consist of a variety of questions presented in various formats. The types of questions that can be used are summarised in Table 1.

Table 1 Types of questions typically found in final examinations

Type of questions	Description
a Only possible in online examinations.
Multiple choice	Question with multiple answers to choose from
Multiple choice with calculation	MCQ requiring numerical calculations
Fill in the blank	Filling in tables, blank spaces; one- or two-word answers
Short answer	Answers is a few sentences to a paragraph
Calculation	Numerical calculations including working out
Drawing	Drawing diagrams, structures, arrows, labelling, etc.
Equation	Requiring formulas, laws, equations, expressions etc.
Drop-down box^a	Box with a select number of answers to choose from
Hybrid	Two or more types of questions combined
Hybrid with drawing	Hybrid questions requiring drawings
Proof	Deriving or proving laws, expressions, etc.

---

Further to this, each question can be assessed using the revised blooms taxonomy (RBT) (Anderson and Krathwohl, 2001) which classifies questions into six categories. Based on this, questions can be further classified as either lower- or higher-order of thinking (Agarwal, 2019; Stringer et al., 2021), as shown in Table 2.

Table 2 Revised Bloom's taxonomy (RBT) levels and their corresponding order of thinking

RBT level	Order of thinking	Description
Remember	Lower	Recognising and recalling facts
Understand	Lower	Explaining what facts mean
Apply	Higher	Using formulas, rules, or ideas
Analyse	Higher	Breaking down information and finding relationships between parts
Evaluate	Higher	Judging the value, validity, or reliability of information
Create	Higher	Combining parts to make new wholes

---

The student's experience of exams

The experience of students taking various types of exams has been documented in literature. Students taking Type-P exams revealed they prepared notes for efficiently memorising information (Theophilides and Koutselini, 2000; Dave et al., 2021), while those taking Type-O exams geared their notes towards ease of access to information during exams (Theophilides and Koutselini, 2000; Myyry and Joutsenvirta, 2015; Dave et al., 2021). Students taking Type-O exams were shown to spend less time preparing for Type-O exams as they could rely on their notes for information (Myyry and Joutsenvirta, 2015; Johanns et al., 2017). However, studies on differences in learning retention after examination are mixed, with findings indicating that this is largely dictated by the quality of assessment (Williams and Wong, 2009; Gharib et al., 2012; Nsor-Ambala, 2019). No known research into strategies of students during these online exams has been documented to our knowledge.

Students have also reported differing sources of stress depending on the type of exam taken. Pre-exam, Type-O students reported lower levels of pre-exam anxiety when compared to closed-book exams, stating access to notes as the main reason (Myyry and Joutsenvirta, 2015; Nsor-Ambala, 2019). During exams, those taking Type-O exams reported additional stresses about technological failures such as battery issues, and difficulties downloading or uploading sections of the exam (Washburn et al., 2017; Shraim, 2019; Petillion and McNeil, 2020). One study that surveyed 342 students showed that 75% of participants noted online exams increased their stress levels (Shraim, 2019).

Additionally, a study by Iannone and Simpson (2014), showed that most students surveyed from two universities stated a preference for closed-book paper-based exams for mathematics, citing reasons such as less opportunity for cheating. Those who disagreed felt that closed book exams favoured those with a better memory (Iannone and Simpson, 2014). It also appears that preference for computer-based and paper-based exams may depend on the discipline being examined (Okocha, 2017).

The academic's experience of exams

Academic exam preferences have been briefly explored in the literature, showing mixed opinions. One study found that 52.6% of surveyed anatomy academics preferred paper-based exams due to concerns about cheating and collusion, and technical issues with online exams (Dulohery et al., 2021). Despite these concerns, many academics prefer marking online exams due to reduced workload, since some questions are automatically marked by the online system (Wibowo et al., 2016; Pagram et al., 2018; Cao et al., 2019; Dulohery et al., 2021).

Context

At the School of Chemistry in the University of Sydney, Type-P exams took place in large exam halls in a space shared by all students completing the relevant unit and involved live invigilators. Students were given an examination booklet detailing the exam questions and separate paper for written answers. Students were not allowed access to notes or devices during these exams. Conversely, Type-O exams were conducted on an online learning management system (LMS) which could be accessed by the student's device of choosing, and at a location of the student's choosing. Students were permitted access to notes, and open access to the internet, however these exams were not open to collusion. Type-O exams were invigilated using a third-party, online proctoring service, ProctorU. Students were asked to upload scans or images of their answers to longer calculation questions and were given 25 minutes after their exam to do so.

Research scope

At the School of Chemistry in the University of Sydney, a shift from Type-P to Type-O exams occurred in 2019–2021 for all three levels of undergraduate chemistry (first-, second-, and third-year) due to the COVID-19 pandemic. This shift was unprecedented and thus prompted an investigation into the impact of these changes on various aspects of the exam.

Chemistry exams often require students to draw structures and mechanisms. As such, transitioning to Type-O exams introduces various complexities, particularly in integrating drawing questions in an online format. Therefore, we sought to investigate the changes on the nature of exam questions between Type-P and Type-O exams. Additionally, this was an unexpected change for both students undertaking exams and academics writing them. Therefore, we also aimed to examine the impact on the experience of students and academics to gain insights into the suitability of Type-O exams in future. To this end, we sought to answer the following questions:

(1) How did the shift to Type-O exams impact the exam questions, if at all?

(2) How did the shift to Type-O exams impact the experiences of students, if at all?

(3) How did the shift to Type-O exams impact the experiences of academics, if at all?

Methods

Analysis of exam questions

Analysis of exam questions was performed on all three levels of undergraduate chemistry (first-, second-, and third-year) between 2019 and 2021. Type-P exams were used in 2019, followed Type-O exams for the first time in 2020, and again in 2021. First-year exams analysis was limited to core units of study to limit variables such as conflation of topics by off-semester units of study (i.e. those taking unit B during semester one and vice versa). All second- and third-year exams were used as there were no off-semester units of study. Replacement exams were not considered due to their similarity to the main exam. A total of 18 first-year, 12 second-year, and 22 third-year exams were analysed.

Categorisation of exam questions

Questions were categorised by their question type, RBT level, and order of thinking. Questions with multiple parts were treated as separate questions.

Analysis of one exam was performed by three different researchers independently. Discussions were had until all category guidelines were agreed upon by all three researchers. The remaining 51 exams were analysed by one researcher, with regular meetings to ascertain consistency.

Categorisation of exam question types were assigned according to Table 1. Classification of questions was performed by utilising the RBT (Anderson and Krathwohl, 2001), and further separated into higher- or lower-order according to divisions used in other studies (Agarwal, 2019; Stringer et al., 2021) according to Table 2.

The percentage of exam items assigned to each category in an exam was calculated using the marks allocated to each question type.

Statistical analyses

Quantitative analysis was conducted using SPSS statistics. Pearson's chi-squared tests of associations were used to determine the significance of changes in the percentage exam items coded to each of the four categories between years (2019–2020, 2020–2021, 2019–2021) and their equivalent semesters (e.g. 2019 Semester 1 and 2020 Semester 1).

A confidence interval of 95% was applied, with a Bonferroni correction to prevent Type I error as comparisons were performed on all four categories. Cramer's V was utilised to determine the effect size of any significant results (Kim, 2017).

Recruitment

For recruitment of interview participants, non-random, convenience, and voluntary sampling were used. In total, we recruited nine students who had undertaken both Type-P and Type-O exams (Group P/O), eight students who had only undertaken Type-O exams (Group O), and six academics involved in the preparation and marking of both exam types.

Interviews and their analyses

Semi-structured interviews (Bodner and Orgill, 2007) were conducted over Zoom with a set of pre-determined prompts. Interviews for students and academics were written with different prompts to explore their relevant perspectives (ESI†).

Audio and video recordings were taken during the interviews, but only audio was analysed. Audio recordings were transcribed using the AI Transcribe tool in Microsoft Word before being cleaned by listening to recordings and correcting the generated transcript.

Inductive thematic analyses were conducted on the interviews (Braun and Clarke, 2006). Three researchers independently analysed one interview to develop the first set of codes which were discussed and condensed until 100% agreement was achieved. To validate this codebook, two researchers independently coded a second interview and compared agreement. 90% and 77.8% agreement were reached for the student and academic interviews, respectively. To reach the 80% agreement threshold, the academic codebook was further refined and condensed until 100% agreement was reached. A third researcher then audited the coding of the second interview to validate the new codebooks. Using these codebooks, one researcher coded all interviews and generated three and six additional codes for the student and academic interviews, respectively. Another audit was conducted on all interviews to determine the validity of the coding. The final codebooks contained 14 and 36 codes for the student and academic interviews, respectively.

Ethical considerations

Ethics approval was obtained for this study (University of Sydney, 2021/180) in adherence with the regulations defined in the National Statement on Ethical Conduct in Human Research (National Statement on Ethical Conduct in Human Research, 2015).

Results and discussion

Changes to exam questions

Changes to question type. The investigation of the proportion of exam items coded to each question type revealed a statistically significant difference, with all significant differences showing a large effect size (Table 3). This was evident both when comparing exams by year, and by equivalent semesters (Table S1, ESI†).

Table 3 Pearson's chi-squared tests for the percentage of exam items coded to different question types between 2019–2020, 2020–2021 and 2019–2021 for first-, second- and third-year exams

		First-year	Second-year	Third-year
a Significant p-value (<0.05). b Large effect size.
2019–2020	p-Value	<0.001^a	<0.001^a	<0.001^a
	Cramer's V	0.406^b	0.435^b	0.337^b
	df	8	10	8
2020–2021	p-Value	<0.001^a	0.997	<0.001^a
	Cramer's V	0.242^b	0.038	0.224^b
	df	8	8	10
2019–2021	p-Value	<0.001^a	<0.001^a	<0.001^a
	Cramer's V	0.242^b	0.446^b	0.405^b
	df	8	10	10

---

The percentage of exam items coded to each question appears to change between Type-P and Type-O exams (Fig. 1). A decrease in drawing questions and an increase in short answer questions in 2020 were attributed to university mandates to avoid file uploads during exams.

Fig. 1 Percentage of exam items coded to different question types for (A) first-year (B) second-year and (C) third-year chemistry exams from 2019–2021. * = significant p-value <0.05.

After the lifting this mandate in 2021, some drawing questions reappeared in first- and third-year exams but not in second-year exams, with no significant difference being shown between second-year exams in 2020 and 2021. Despite the return of some drawing questions, there remained a difference in question types between Type-P and Type-O exams, possibly owing to decisions made by the second-year academics.

The increase in drop-down questions between 2019 and 2020 remained between 2020 and 2021. First-year exams favoured drop-down questions more than second- and third-year exams, likely due to the increased exam difficulty in later years which necessitates longer text responses.

There was a notable increase in multiple-choice questions (MCQs) between 2019 and 2020, due to the lack of MCQs in some 2019 units. In 2020 and 2021, MCQs were included in these units, possibly due to ease of implementation or changes in university guidelines on the number of MCQs allowed per exam.

Similarly, other universities also reported resorting to non-handwritten exam questions, such as “pick the correct statement”, for questions that would traditionally require hand-drawn responses (Dicks et al., 2020). Those who used file uploads to accommodate drawing questions reported several technical problems (Clark et al., 2020; Rodríguez Núñez and Leeuwner, 2020), with Clark et al. (2020) resorting to the use of chemical drawing tools, whereby students were able to convert chemical structures into SMILES strings to denote their final responses. Along with our findings, this suggests that drawing questions could be replaced with non-handwritten equivalents.

In summary, there was a difference in common question types between Type-P and Type-O exams. Type-O exams inhibited drawing questions in favour of short answer questions, a trend that persisted even after file uploads were permitted in 2021.

Changes to RBT item type. Next, changes to RBT item types were assessed (Anderson and Krathwohl, 2001).

Type-O exams saw a shift from ‘apply’ to ‘understand’ questions, indicating lower-order thinking (Fig. 2). Significant differences were observed between 2019–2020 and 2019–2021, though first- and second-year changes had small and medium effect sizes, respectively (Table 4). This may have been due to the difficulty of including calculation and drawing question in Type-O exam types, which are classified as ‘apply’ questions (Crowe et al., 2008; Pugh and Gates, 2021). It is also possible that the academics aimed to reduce student stress and anxiety by lowering the RBT item type. However, these changes were relatively minor, as indicated by their effect sizes.

Fig. 2 Percentage of exam items coded to each of the RBT levels for (A) first-year (B) second-year and (C) third-year chemistry exams during 2019–2021. * = significant p-value <0.05.

Table 4 Pearson's chi-squared test values for the percentage of exam items coded to different RBT levels between 2019–2020 and 2020–2021 for first-, second- and third-year exams

		First-year	Second-year	Third-year
a Significant p-value < 0.05. b Large effect size. c Medium effect size.
2019–2020	p-Value	0.026	0.001^a	<0.001^a
	Cramer's V	0.104	0.145^c	0.173^c
	df	4	5	5
2020–2021	p-Value	0.024	0.131	<0.001^a
	Cramer's V	0.115	0.093	0.235^b
	df	4	4	5
2019–2021	p-Value	0.005^a	<0.001^a	<0.001^a
	Cramer's V	0.12	0.197^c	0.230^b
	df	4	4	5

---

Third-year exams showed a significant shift in RBT item type with a large effect size, both between 2019–2020 and 2020–2021, possibly due to syllabus changes in 2021. A significant difference was also noted from semester 1 of 2019 to semester 1 of 2020 (Tables S2 and S3, ESI†), likely due to the rapid transition to Type-O exams. However, no significant difference was found between semesters 2 of 2019 and 2020, suggesting that academics rapidly adapted to the new format (Tables S2 and S3, ESI†).

A study by Davies et al. (2022) also found that the rapid shift to online exams during COVID-19 resulted in exams with minimal changes to question RBT type. However, their questions were limited to “single best answer” questions in the first three levels (i.e. Remember, Understand, and Apply).

Of interest to note, there was a tendency toward lower-order thinking questions in Type-O exams, despite the open-book format. This could pose a problem for long-term learning retention, as higher-order questions are associated with better retention (Johanns et al., 2017). The rapid transition to Type-O exams likely resulted from time constraints for academics, limiting their ability to write higher-order questions that are more suited to an online open-book format. However, this tendency toward lower-order thinking questions seemed to pervade even at higher levels of chemistry (i.e., through second- and third-year units). As such, this tendency could further be explained by a lack of formal training in writing effective exam questions. While there is no consensus on how Type-O exams should be written, the literature suggests benefits from using higher-order questions carefully designed to assess genuine critical thinking skills and avoid easily searchable answers, for example, through utilising authentic assessment principles (Eilertsen and Valdermo, 2000; Brightwell et al., 2004; Williams and Wong, 2007; Johanns et al., 2017).

Overall, questions in all exam types favoured those of a lower-order thinking. Any statistically significant results were found to have a small effect size, despite the minor shift toward ‘understand’ type questions.

Student interviews

Semi-structured interviews were conducted with students who had taken both Type-P and Type-O exams (Group P/O) and those who had only taken Type-O exams (Group O).

Emotional and mental states during Type-P exams. Overall, students mentioned differing sources of stress when taking Type-P or Type-O exams. For those in Group P/O, the most common source of stress (raised by six of the nine students) was the exam hall environment and the presence of invigilators. Three of these students cited that having their Type-O exam at home felt more relaxing. However, one student stated that while the environment of the Type-P exam was more stressful, they found that it helped them:

‘perform better as well because. this is a more formal event, this is something that's important and they’re having everyone come in and people watch you.’ – Student 2 (Group P/O).

One student revealed that memorising information for closed-book Type-P exams was stressful, mirroring findings seen in the literature (Nsor-Ambala, 2019). Another source of stress raised by students taking Type-P exams was organising transport and locating the exam venue which, cited by three of the nine students:

‘I gotta make sure I catch a few trains early just in case there's delays, and then finding the room’ – Student 4 (Group P/O).

Emotional and mental states during Type-O exams. Next, Group P/O as well as Group O students were asked about their emotional and mental states while taking Type-O exams. The most prominent stressor cited was the use of ProctorU, the online proctoring service used in Type-O exams at the University of Sydney, which was raised by 15 of the 17 students interviewed. Students mainly raised concerns for privacy while being recorded taking Type-O exams. One student expressed concern about ProctorU not working during their exam. While they ‘didn’t have a specifically bad experience with it’ they had heard of issues affecting other students, and that added an ‘additional stress that you don’t normally have with a paper-based exam.’ Concerns about proctoring services during Type-O exams have been briefly touched on in the literature and are in line with our findings (Petillion and McNeil, 2020).

The second-most prominent stressor raised by these students was the time limit. Nine of the 17 students interviewed believed that the time allotted for Type-O exams was insufficient, which resulted in feelings of stress. For example, students noted that typing out answers for calculation questions required more time than it would have for a written exam:

‘I think there was a bit of a push for time, especially with the calculations. It obviously took a bit longer than just writing it down ‘cause we had to type it up.’ – student 9 (Group P/O).

Seven of the 17 students mentioned auditory distractions in their domestic environments as a source of stress during Type-O exams, with one student recounting:

‘One exam, my neighbours, it was Friday afternoon or Thursday afternoon, it was 3:00 PM, they started having a dance party, like an absolute raging party, and I sat there and listened to house music for the most of my exam. And it just made me so angry, like why? Why do I have to deal with this?’ – Student 5 (Group P/O).

Other stressors raised by the students were primarily related to technology. Most prominently, concerns about internet failure were raised by six students, aligning with findings from other studies (Elsalem et al., 2020). Additionally, four students noted that uploading handwritten responses was a source of stress when taking Type-O exam including a lack of appropriate technology, failure to upload, and one student stating they ‘honestly nearly forgot.’ Two students also commented on issues with the exam interface, with one stating:

‘You couldn’t flick through the questions very easily, so… you didn’t really know the overview and how to pace the exam properly.’ – Student 17 (Group O).

These concerns of technological failures align with findings seen in multiple other studies (Washburn et al., 2017; Shraim 2019; Clark et al., 2020; Elsalem et al., 2020).

In addition, four students further noted that they were initially unsure of the expectations of a Type-O exam, citing the lack of past papers and sample exam questions as a significant concern:

‘Cause you basically have no clues about how the online exams will look like… when it transfers to online, the uni teaching staff, they also don't have the past exam paper of online exam, so they can't provide you any clues. Sometimes they don't provide you the sample. The sample they give you, they put nonsense questions there, just gave you a look of the online exam, how online exam feels like. But the questions aren’t really related.’ – Student 7 (Group P/O).

However, after this initial adjustment period, 12 out of the 17 students stated that they had eventually adjusted to the Type-O exams. These findings are in line with what have been reported in the literature, whereby students who had only been exposed to paper-based exams initially found online exams more stressful, but believed that familiarising themselves with the exam format would decrease their stress (Wibowo et al., 2016; Afacan Adanır et al., 2020).

Therefore, it appears that students experience differing origins of stress depending on whether they are taking Type-O or Type-P exams.

Differences in exam preparation strategies. Students in Group P/O were then asked about their exam preparation strategies for both Type-P and Type-O exams. Four of the nine students in Group P/O noted that for Type-P exams, their notes were prepared in a way to assist them with memorisation. These students then indicated a difference in note preparation for Type-O exams, stating that they compiled as much information as possible into their notes to facilitate quick searching of answers during the exam. These findings mirror what has been noted in the literature (Theophilides and Koutselini, 2000; Dave et al., 2021). Two students further revealed that they prepared less for their Type-O exams:

‘Because it's online exam and is mostly open book, I study less 'cause I can have all the information on my computer.’ – Student 14 (Group O).

As such, it appears that the closed-book nature of Type-O exams affects the strategies used by students to prepare for their exams.

Differences in strategies used during exams. Students were then asked about the strategies they used during exams for both Type-P and Type-O exams. Eight out of nine students in Group P/O reported completing their exams in chronological order. Similarly, all students in Group O followed the same approach.

Eight out of nine students in Group P/O reported skipping questions they were unsure of during both Type-P and Type-O exams. Similarly, all eight students in Group O who had only taken Type-O exams reported the same strategy. However, their methods for flagging skipped questions varied due to the exam format. Four of the nine students in Group P/O mentioned circling the question number, leaving it blank, or holding the page during Type-P exams, while six out of eight students in Group O either left questions blank, used the LMS flagging function, or wrote question numbers on paper during Type-O exams:

‘When we used to do the exams on paper… I had these like little tabs I would attach that would be different colours that meant different things… like, this is a question I just haven't done… But that's not an option with online… Like I would write down on a piece of paper that I need to definitely go back to this question 'cause I haven't done it, so that was slightly more convoluted.’ – Student 3 (Group P/O).

Three additional exams strategies were identified by students who took Type-O exams. Because of the online, open-book, open-internet (herein referred to as open-access) nature of Type-O exams, students had access to external materials for use in their exams. All 17 students interviewed mentioned looking at their notes at some point in the exam – either for factual recall, or to check their answers after writing a response – with one student remarking:

‘I think there were some questions which are like, sort of fact checking… And these are the sort of questions I would probably look up or check a book, 'cause I feel like they don't really test your sort of understanding of the topic.’ – Student 1 (Group P/O).

Our interviews further revealed that students’ Type-O exam strategies were influenced by two conflicting motivations: the desire to finish on time vs. the desire for legibility. Five out of nine students in Group P/O and five out of seven‡ students in Group O mentioned that they used full sentences, and believed that full sentences conveyed their ideas more clearly despite taking longer to write. Four students stated that they used a mix of full sentence and dot point responses depending on time constraints. The remaining two students (one from each group) mentioned they used dot points for efficiency, especially if they were running out of time. For questions that involved calculations, 11 of 17 students remarked omitting lines of working for sake of time, while the remaining six noted including all lines of working for the sake of clarity. Regarding the tension between these conflicting motivations, one student noted:

‘I think in general I did full sentences except for when I was really short for time or if it was a question that I thought was one of the sorts of questions where they really were just wanting to see one or two essential points.’ – Student 3 Group (P/O).

Considering the complexities of typing chemistry responses, three students from Group P/O and five students in Group O mentioned they skipped formatting as they found it time consuming. The remaining nine students chose to include formatting for the sake of legibility, despite acknowledging that it was time-consuming:

‘It probably was just a visual thing, because I can see something in my brain, it can mean like something to me, but it might not mean the same thing to the marker… I have to make sure that it's right and that what I'm trying to convey is coming out.’ – Student 2 (Group P/O).

Furthermore, four students in Group P/O revealed that the convenience of typing allowed them to edit their exam responses. However, some remarked that this slowed them down during the exam:

‘…we might do a lot more self-editing and rewriting stuff which we didn't even need, but it's more just, if you give them the option, then we naturally go for it, so that could be a factor that slows down our speed in the online exams.’ – Student 8 (Group P/O).

Therefore, it appears that students maintained some common exam strategies between Type-P and Type-O exams. However, students also revealed adopting new strategies in order to enhance their performance in Type-O exams, which was mainly influenced by the open-access format, and the conflicting desire to either complete exams on time or maintain legibility.

Exam type preferences. Finally, students were asked about their exam type preference. Eight students out of nine in Group P/O expressed preference for Type-P exams, citing that it was easier to show working for answers to calculation and drawing questions on paper. These findings correlate to findings seen in literature, whereby students who had taken both types of exams revealed a preference for paper-based exams (Wibowo et al., 2016). Surprisingly, two students in Group O also stated preference for Type-P exams, despite not being prompted. One explained:

‘when teachers see your paper, they can see like your process of working out’ (Student 12, Group O).

Previous studies have revealed that students who were taking chemistry exams showed lower acceptability of computer-based exams when compared to those taking other courses (Okocha, 2017). This could be attributed to the need for chemical drawings, reaction diagrams, calculations, and special symbols in the field of chemistry, which lends itself better to paper-based responses.

Regarding Type-O exams, all 17 students stated that the potential for technical failures, and issues with uploading responses made this exam format more stressful. Despite this, six of the nine students in Group P/O still enjoyed having the option to draw and upload, suggesting the potential to successfully implement this feature in Type-O exams if technical issues could be mitigated. Interestingly, only one of the students from Group O preferred having drawing questions in Type-O exams:

‘…it can be really helpful to some students who need that other way to show “I do know what you're talking about let me show you.”’ – Student 11 (Group O).

All 17 students also expressed preference for un-proctored Type-O exams due to both privacy concerns and technical issues. However, five students recognised the need for proctoring, as Type-O exams were perceived to be more susceptible to academic dishonesty. This has been noted in the literature, whereby cheating increased with the shift to online exams (Newton and Essex, 2024).

Despite the issues students expressed about Type-O exams, five students in Group P/O and six students in Group O enjoyed the open-access format, and appreciated not having to memorise large amounts of information. Additionally, four students in Group P/O and two students in Group O enjoyed the convenience of the remote nature of Type-O exams:

‘Uhm, in the paper-based, I think definitely the drawing questions good, 'cause then it's just so nice just to draw straight onto the exam papers… I still prefer doing it at home, in the comfort of my own home, it still outweighs going into uni and doing it in-person.’ – Student 4 (Group P/O).

In light of this, it appears that if technological errors were eliminated, Type-O exams may have the potential to be successfully implemented. This is also suggested by Wibowo et al. (2016) who found that students would enjoy online exams if technical issues were alleviated, and mock exams were provided for preparation.

Academic interviews

Experiences writing Type-O exams. Academics were first asked about their experiences writing Type-O exams. Six academics mentioned that the open-access nature of the exam posed extra challenges when writing questions. Academics mentioned the initial mandate of not including uploads restricted the types of questions they could write:

‘In organic chemistry we use mechanisms a lot, and so obviously you're restricted to be able to do that in an essay-type based question, or you would have to frame it so that students gave descriptions on what is happening rather than drawing it.’ – Academic 1.

Academics also mentioned the importance of writing questions that were not readily searchable online or within students’ notes, to effectively gauge students’ understanding. These findings have been reflected in other studies, where academics mentioned checking if search engines would identify solution to their questions (Johnston and O’Farrell, 2020).

One academic mentioned the need to write multiple Type-O exams to reduce the risk of academic dishonesty and prevent unauthorised disclosure of questions online. However, the academic noted this created an additional workload that is not typically encountered when writing Type-P exams, because the replacement exam is usually recycled and never published.

In addition, two academics stated they felt less supported when writing Type-O exams, noting that for Type-P exams, the education support team oversaw the formatting exams. However, for Type-O exams the academics were assigned the extra requirement of formatting and editing questions for use in the LMS themselves and found it time consuming.

Perception of in factors affecting question type. Academics were then asked about their perceptions of the changes in exam questions.Before being shown the results of the previous section, academics were asked about whether the question types they used in Type-O exams differed from those used for Type-P exams. All six academics mentioned similarities between question types for both exam formats, but commented on the loss of drawing questions in 2020 (due to uploads not being permitted until 2021), and the addition of drop-down box questions in Type-O exams.

After being presented with Fig. 1, all six academics commented on the university mandates restricting their use of drawing questions – a factor also raised prior to being shown the results. Academics also raised that it was the online delivery of Type-O exams that made it more difficult to implement drawing and calculation questions.

Additionally, two academics raised that changes in the curricula may have contributed to the changes in question types, irrespective of the change in exam type.

Perception of in factors affecting RBT item type. When asked about their perception on the RBT item type of questions in Type-O exams prior to being shown Fig. 2, three academics mentioned that regardless of exam format, they used a mixture of higher and lower orders of thinking in order to differentiate students:

‘Well, so I'd say the online exam wasn't any different in a sense to the paper-based exams as there were both types. And of course the higher order questions are the ones that I use as the HD and distinction barrier questions. And the low order ones for the pass and credit.’ – Academic 2.

This is consistent with other studies that have shown that at least 50% of exam questions were lower order, allowing students to pass and higher performing students to be distinguished (Dicks et al., 2020). Two academics believed that Type-O exams contained more higher order questions because of the open-access format, which would allow students to answer ‘recall’ questions by searching their notes or the internet. In contrast, one academic believed that all exams contained mostly lower order questions, regardless of format:

‘I feel many of our students struggle with anything higher order. So if there's a bias, it would be more towards lower order questions. Uhm, yeah, genuinely applying knowledge in an unfamiliar context is something that happens very seldom in USYD exams in my experience…it's to do with how our students are trained, it's to do with their expectations coming in.’ – Academic 4.

Academics were then shown Fig. 2. All academics suggested that the restrictions caused by Type-O exams and university mandates limited their ability to incorporate higher-order question types, which contributed to the decline in RBT item type:

‘I think again, consistent with the modes of delivery. So clearly yeah, 2020 where we went online and only had the short answer response perhaps changed… Uhm, we couldn't probe a lot of things because it was all sort of descriptive. That sort of looks like it made a little bit of a comeback in 2021 where we could have some file uploads. And so yeah, higher order questions could be slotted back in again.’ – Academic 1.

Five academics also raised that the introduction of new curricula may have also contributed to the lower RBT item types. Three suggested that changes in exam writers may have had an influence, suggesting that different examiners may write questions with different levels of thinking:

‘Different people teach different courses. Different people ask different types of questions.’ – Academic 4.

Experiences marking Type-P vs. Type-O exams. Next, academics were asked about their experience marking Type-O in comparison to Type-P exams. The main advantages mentioned regarding marking Type-O exams were based on convenience.

Four academics raised that the main advantage was the automatic marking of questions by the LMS. However, another four academics noted that negative aspects of marking Type-O exams were due to technological issues such as manually verifying answers to check for errors by the LMS, and that markers could not access an exam at the same time without the risk of overwriting each other's marking.§

Two academics enjoyed the convenience not handling physical papers and being able to mark from home. However, two academics believed online marking was more time-consuming as they could not write on the exams. One found it challenging as they preferred marking one question at a time to ensure the marking scheme is applied consistently across students:

‘So when I mark, if I've got to mark six questions in the paper, I like to mark all of question one first and then all of question two first and then all of question three so that I can be consistent more easily across the papers. But that was much more difficult in an online format, so I didn't do that in an online format. So I did spend a lot more time going back and checking that I had applied my marking scheme equally to the first marked papers and the last marked papers. ‘ – Academic 3.

Perception of the students’ emotional experience. Academics were then asked about their perception of the students’ emotional experience of Type-O exams. Four academics believed the most common stressors were likely due to technological issues, such as internet connectivity, proctorU complications, and uploading responses. This was in line with the students’ experience, whereby all 17 students raised technological issues as a stressor in Type-O exams. Three academics separately mentioned: time pressures due to having to type responses; being unable to prepare due to a lack of past exam papers; and feeling isolated doing an exam at home. These perceptions align with many of our students’ reported experiences. In contrast, one academic stated:

‘the trepidation of exams will be the same regardless’ – Academic 6.

Perception of the students’ exam strategies. Next, academics were asked to comment on their perception of the students’ exam strategies. Three academics mentioned they noticed shorter response lengths in Type-O exams, as they had instructed students to use dots points for efficiency, both for the students and the marker. Three others believed that response lengths generally varied between students. Indeed, this was shown in our student interviews, where students appeared to decide between time efficiency, and clarity to determine response length.

Regarding the students’ time management strategies, four academics believed that this was specific to the individual student.

Three academics mentioned the chronological approach as they noticed questions being left empty towards the end of the exams. Two mentioned it was possible that the questions toward the end of the exam may have been due to students spending time formatting their responses, or looking through their notes.

Overall, it appears that most academics had astute perception of the students’ experience of Type-O exams.

Limitations

This study was confined to the School of Chemistry at the University of Sydney. As such, experiences noted in this study may differ from those in other institutions. It should also be noted that this study compares Type-P and Type-O exams that occurred during the COVID-19 pandemic. It is possible that additional factors related to the pandemic may have affected the exams, and the students’ and academics’ experiences. Additionally, this study focuses on analysis of Type-P exams in 2019 only, which may differ from Type-P exams used in previous years. Our interviews also have small sample sizes which may limit their generalisability to the broader population. In addition, those who volunteered for interviews may have had different experiences to others who did not.

Conclusions and future considerations

Overall, we observed a minor shift in question types with Type-O exams which showed a decrease in drawing questions, and an increase in short answer questions. Interviews with academics suggested that the online format of the Type-O exam was the primary motivation for this change. This could be attributed to the specific requirements of chemistry, which often require chemical drawings, reaction diagrams, calculations, and special symbols which are more easily accommodated in paper-based responses. Despite this, the RBT item type of the exams remained consistent. Studies have shown that open-book exams with equivalent RBT types to their closed-book counterparts do not result in any significant changes to student results, suggesting that both exam formats can adequately measure students’ understanding of course material (Brightwell et al., 2004; Davies et al., 2022). However, it is important to note that these studies only assessed questions in the first three RBT levels. Future studies exploring the implication of utilising questions at higher RBT levels should be considered.

Regardless, it is crucial to consider the impact of exam type on the experiences of both students and academics. Student interviews revealed a shift in the origins of stress when comparing both exam types. The stressors for Type-O exams were mainly related to technological failures, such as internet dropouts. Despite the mentions of stress undertaking Type-O exams at home, it was also noted that the Type-P environment was also a cause of some students’ stress. It may be possible for universities to offer alternative locations for students with difficult home environments to take their Type-O exams. Furthermore, this study identified the potential for reducing students’ stress regarding Type-O exams by providing them with adequate preparatory material to help familiarise them with the delivery and format of Type-O exams. These findings align with literature reports where students who had only been exposed to paper-based exams found online exams more stressful but believed that familiarising themselves with the exam format would decrease their stress (Wibowo et al., 2016; Afacan Adanır et al., 2020). This could be easily implemented by providing students with mock exams on the same LMS used in final exams, allowing students the opportunity to familiarise themselves with the new format. While they voiced a preference for Type-P exams overall, students demonstrated their ability to quickly adapt to the unfamiliar exam format, by adjusting their note preparation style, modifying their response style to suit the Type-O format, and largely maintaining exam strategies between the two exam types.

As for academics, the two main setbacks that academics raised about their experiences with Type-O exams, were due to the open-access nature of the exams and their perceived lack of support. Academics attributed changes in exam questions to changes in curricula, and the inability to add drawing questions due to a lack of upload capabilities in the first year. Despite this, our analyses of the exam questions revealed that academics continued to write assessments at similar level of RBT item type. However, since current literature suggest the benefits of writing higher-order questions for Type-O exams to enhance learning retention and ensure answers are not easily searchable, it is important to ensure that academics are adequately trained in writing such questions.

As such, this study has shown that Type-O exams have the potential to be implemented in the university system without significantly impacting the student and academic experience; however, it is essential to consider key factors to ensure they uphold exam integrity.

Translation to a post-COVID world

As we transition to an education landscape beyond the pandemic, it is essential to evaluate the long-term viability and effectiveness of Type-O exams, and assess how they might endure in this new educational environment.

A major area for consideration is the rise of artificial intelligence (AI), such as large language models or generative AI, which presents both challenges and opportunities in the post-COVID landscape. For instance, unauthorised use of AI tools by students could pose a significant risk for academic dishonesty. Online proctoring tools like ProctorU offer potential solutions by providing remote invigilation services, but their effectiveness and associated privacy concerns require careful consideration (Hussein et al., 2020; Selwyn et al., 2023; Kuleva and Miladinov, 2024). On the other hand, AI technology itself could be employed to minimise the likelihood of academic dishonesty through the use of AI-based proctoring or AI-generated adaptive exams (Nigam et al., 2021; Surahman and Wang, 2022). Additionally, AI could be used to automate marking and to provide efficient feedback, potentially reducing the workload for academics.

As such, balancing the benefits and potential risks associated with the rapid evolution of AI in relation to Type-O exams is a critical area for future research.

Furthermore, it is important to consider technological solutions that could address several of the issues raised in this study. For instance, the challenges with incorporating drawing questions or uploading answers in Type-O exams could be mitigated by permitting the use of digital drawing devices, such as tablets. However, this approach raises concerns about equitable access, as not all students may have personal access to such devices. To address this, universities could establish digital testing centres equipped with digital drawing devices. Such centres could ensure that students have equal access to necessary technology, thereby maintaining fairness in the Type-O examination process. These testing centres could potentially reduce students’ stressors relating to technological failure, however, may reintroduce the stressors of travelling to the required testing venue. The implementation of digital testing centres could also facilitate the reinstatement of in-person invigilation, thus combining the benefits of Type-O exams while maintaining the academic integrity of the Type-P exam setting. Further investigation would be required to understand the implications of this strategy.

Author contributions

Alice Lu completed this work as part of a capstone honours thesis. As such, the bulk of the research work, and original writing, was undertaken by her. Reyne Pullen and Stephen George-Williams acted as co-supervisors throughout that time. Finally, Joy Ghrayche joined the research group to undertake the final writing/conversion steps to push this study through to publication.

Data availability

Data collected for this study have not been made publicly available due to ethical considerations.

Conflicts of interest

The authors report there are no competing interests to declare.

Acknowledgements

This work was supported by the University of Sydney under an Education Innovation grant. The authors would like to acknowledge all interviewees, both staff and students. Without their input, this work simply could not have been conducted.

References

1. Afacan Adanır G., İsmailova R., Omuraliev A. and Muhametjanova G., (2020), Learners’ Perceptions of Online Exams: A Comparative Study in Turkey and Kyrgyzstan, Int. Rev. Res. Open Distrib. Learn., 21(3), 1–17 DOI:10.19173/irrodl.v21i3.4679.
2. Agarwal P. K., (2019), Retrieval Practice & Bloom's Taxonomy: Do Students Need Fact Knowledge before Higher Order Learning? J. Educ. Psychol., 111(2), 189–209 DOI:10.1037/edu0000282.
3. Anderson L. W. and Krathwohl D. R., (2001), A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Longman.
4. Archer E., (2017), The Assessment Purpose Triangle: Balancing the Purposes of Educational Assessment, Front. Educ., 2, 41 DOI:10.3389/feduc.2017.00041.
5. Bodner G., and Orgill M., (2007), Theoretical Frameworks for Research in Chemistry/Science Education, Educación química.
6. Braun V. and Clarke V., (2006), Using Thematic Analysis in Psychology, Qualit. Res. Psychol., 3(2), 77–101 DOI:10.1191/1478088706qp063oa.
7. Brightwell R., Daniel J.-H. and Stewart A., (2004), Evaluation: Is an Open Book Examination Easier? Biosci. Educ., 3(1), 1–10 DOI:10.3108/beej.2004.03000004.
8. Cao Y., Porter L., Liao S. N. and Ord R., (2019), Paper or Online? A Comparison of Exam Grading Techniques, Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, pp. 99–104 DOI:10.1145/3304221.3319739.
9. Clark T. M., Callam C. S., Paul N. M., Stoltzfus M. W. and Turner D., (2020), Testing in the Time of COVID-19: A Sudden Transition to Unproctored Online Exams, J. Chem. Educ., 97(9), 3413–3417 DOI:10.1021/acs.jchemed.0c00546.
10. Crowe A., Dirks C. and Wenderoth M. P., (2008), Biology in Bloom: Implementing Bloom's Taxonomy to Enhance Student Learning in Biology, CBE Life Sci. Educ., 7(4), 368–381 DOI:10.1187/cbe.08-05-0024.
11. Dave M., Dixon C. and Patel N., (2021), An Educational Evaluation of Learner Experiences in Dentistry Open-Book Examinations, Br. Dent. J., 231(4), 243–248 DOI:10.1038/s41415-021-3258-7.
12. Davies D. J., McLean P. F., Kemp P. R., Liddle A. D., Morrell M. J., Halse O., Martin N. M. and Sam A. H., (2022), Assessment of Factual Recall and Higher-Order Cognitive Domains in an Open-Book Medical School Examination, Adv. Health Sci. Educ., 27(1), 147–165 DOI:10.1007/s10459-021-10076-5.
13. Dicks A. P., Morra B. and Quinlan K. B., (2020), Lessons Learned from the COVID-19 Crisis: Adjusting Assessment Approaches within Introductory Organic Courses, J. Chem. Educ., 97(9), 3406–3412 DOI:10.1021/acs.jchemed.0c00529.
14. Dulohery K., Scully D., Longhurst G. J., Stone D. M. and Campbell T., (2021), Emerging from Emergency Pandemic Pedagogy: A Survey of Anatomical Educators in the United Kingdom and Ireland, Clin. Anat., 34(6), 948–960 DOI:10.1002/ca.23758.
15. Eilertsen T. V. and Valdermo O., (2000), Open-Book Assessment: A Contribution to Improved Learning? Stud. Educ. Eval., 26(2), 91–103 DOI:10.1016/S0191-491X(00)00010-9.
16. Elsalem, L., Al-Azzam N., Jum’ah A. A., Obeidat N., Sindiani A. M. and Kheirallah K. A., (2020), Stress and Behavioral Changes with Remote E-Exams during the Covid-19 Pandemic: A Cross-Sectional Study among Undergraduates of Medical Sciences, Ann. Med. Surgery, 60, 271–279 DOI:10.1016/j.amsu.2020.10.058.
17. Gharib A., Phillips W. and Mathew N., (2012), Cheat Sheet or Open-Book? A Comparison of the Effects of Exam Types on Performance, Retention, and Anxiety, J. Psychol. Res., 2(8), 469–478 DOI:10.17265/2159-5542/2012.08.004.
18. Hussein, M. J., Yusuf J., Sandhya Deb A., Fong L. and Naidu S., (2020), An Evaluation of Online Proctoring Tools, Open Praxis, 12(4), 509–525 DOI:10.5944/openpraxis.12.4.1113.
19. Iannone P. and Simpson A., (2014), Students’ Preferences in Undergraduate Mathematics Assessment, Stud. High. Educ., 40(6), 1046–1067 DOI:10.1080/03075079.2013.858683.
20. Johanns B., Dinkens A. and Moore J., (2017), A Systematic Review Comparing Open-Book and Closed-Book Examinations: Evaluating Effects on Development of Critical Thinking Skills, Nurse Educ. Pract., 27, 89–94 DOI:10.1016/j.nepr.2017.08.018.
21. Johnston J. and O’Farrell C., (2020), Rewriting the (Exam) Script? Assessing Student Learning in an Unusual End-of-Year Context, All Ireland J. High. Educ., 12(2) DOI:10.62707/aishej.v12i2.457.
22. Kim H.-Y., (2017), Statistical Notes for Clinical Researchers: Chi-Squared Test and Fisher's Exact Test, Restor. Dent. Endod., 42(2), 152–155 DOI:10.5395/rde.2017.42.2.152.
23. Kuleva M. and Miladinov O., (2024), Exploring the Efficacy of Online Proctoring in Online Examinations: A Comprehensive Review, Environ. Technol. Resour. Proc. Int. Sci. Pract. Conf., 2, 192–196 DOI:10.17770/etr2024vol2.8058.
24. Myyry L. and Joutsenvirta T., (2015), Open-Book, Open-Web Online Examinations: Developing Examination Practices to Support University Students’ Learning and Self-Efficacy, Active Learn. High. Educ., 16(2), 119–132 DOI:10.1177/1469787415574053.
25. National Statement on Ethical Conduct in Human Research, (2015), Canberra: National Health and Medical Research Council, https://www.nhmrc.gov.au/research-policy/ethics/national-statement-ethical-conduct-human-research.
26. Newton P. M. and Essex K., (2024), How Common Is Cheating in Online Exams and Did It Increase During the COVID-19 Pandemic? A Systematic Review, J. Acad. Ethics, 22(2), 323–343 DOI:10.1007/s10805-023-09485-5.
27. Nigam A., Pasricha R., Singh T. and Churi P., (2021), A Systematic Review on AI-Based Proctoring Systems: Past, Present and Future, Educ. Inf. Technol., 26(5), 6421–6445 DOI:10.1007/s10639-021-10597-x.
28. Nsor-Ambala R., (2019), Impact of Exam Type on Exam Scores, Anxiety, and Knowledge Retention in a Cost and Management Accounting Course, Acc. Educ., 29(1), 32–56 DOI:10.1080/09639284.2019.1683871.
29. Okocha F., (2017), Student Perception and Acceptance of Computer Based Testing, J. Digital Innov. Contemp. Res. Sci. Eng., 5(1), 25–32.
30. Pagram J., Cooper M., Jin H. and Campbell A., (2018), Tales from the Exam Room: Trialing an E-Exam System for Computer Education and Design and Technology Students, Educ. Sci., 8(4), 188 DOI:10.3390/educsci8040188.
31. Petillion R. J., and McNeil W. S., (2020), Student Experiences of Emergency Remote Teaching: Impacts of Instructor Practice on Student Learning, Engagement, and Well-Being, J. Chem. Educ., 97(9), 2486–2493 DOI:10.1021/acs.jchemed.0c00733.
32. Pugh S. L. and Gates J., (2021), The Application of Bloom's Taxonomy to Higher Education Examination Questions in Physics, New Direct. Teach. Phys. Sci., 16 DOI:10.29311/ndtps.v0i16.3674.
33. Rodríguez Núñez, J. and Leeuwner J., (2020), Changing Courses in Midstream: COVID-19 and the Transition to Online Delivery in Two Undergraduate Chemistry Courses, J. Chem. Educ., 97(9), 2819–2824 DOI:10.1021/acs.jchemed.0c00781.
34. Selwyn N., O’Neill C., Smith G., Andrejevic M. and Gu X., (2023), A Necessary Evil? The Rise of Online Exam Proctoring in Australian Universities, Media Int. Aust., 186(1), 149–164 DOI:10.1177/1329878X211005862.
35. Shraim K., (2019), Online Examination Practices in Higher Education Institutions: Learners’ Perspectives, Turk. Online J. Distance Educ., 20(4), 185–196 DOI:10.17718/tojde.640588.
36. Stringer J. K., Santen S. A., Lee E., Rawls M., Bailey J. Richards A., Perera R. A. and Biskobing D., (2021), Examining Bloom's Taxonomy in Multiple Choice Questions: Students’ Approach to Questions, Med. Sci. Educ., 31(4), 1311–1317 DOI:10.1007/s40670-021-01305-y.
37. Surahman E. and Wang T.-H., (2022), Academic Dishonesty and Trustworthy Assessment in Online Learning: A Systematic Literature Review, J. Comput. Assist. Learn., 38(6), 1535–1553 DOI:10.1111/jcal.12708.
38. Theophilides C. and Koutselini M., (2000), Study Behavior in the Closed-Book and the Open-Book Examination: A Comparative Analysis, Educ. Res. Eval., 6(4), 379–393 DOI:10.1076/edre.6.4.379.6932.
39. Vazquez, J. J., Chiang E. P. and Sarmiento-Barbieri I., (2021), Can We Stay One Step Ahead of Cheaters? A Field Experiment in Proctoring Online Open Book Exams, J. Behav. Exp. Econ., 90, 101653 DOI:10.1016/j.socec.2020.101653.
40. Washburn, S., Herman J. and Stewart R., (2017), Evaluation of Performance and Perceptions of Electronic vs. Paper Multiple-Choice Exams, Adv. Physiol. Educ., 41(4), 548–555 DOI:10.1152/advan.00138.2016.
41. Wibowo, S., Grandhi S., Chugh R. and Sawir E., (2016), A Pilot Study of an Electronic Exam System at an Australian University, J. Educ. Technol. Syst., 45(1), 5–33 DOI:10.1177/0047239516646746.
42. Williams J. and Wong A., (2007), Closed Book, Invigilated Exams versus Open Book, Open Web Exams: An Empirical Analysis, ASCILITE 2007 – Australasian Soc. Comput. Learn. Tertiary Educ., 1079–1083.
43. Williams J. B., and Wong A., (2009), The Efficacy of Final Examinations: A Comparative Study of Closed-book, Invigilated Exams and Open-book, Open-web Exams, Br. J. Educ. Technol., 40(2), 227–236 DOI:10.1111/j.1467-8535.2008.00929.x.
44. Wuthisatian, R., (2020), Student Exam Performance in Different Proctored Environments: Evidence from an Online Economics Course, Int. Rev. Econ. Educ., 35, 100196 DOI:10.1016/j.iree.2020.100196.

---

Footnotes

† Electronic supplementary information (ESI) available. See DOI: https://doi.org/10.1039/d4rp00290c

‡ During an interview with a student in Group O, the researcher forgot to ask this prompt. Thus, responses were only obtained from seven students instead of eight.

§ Of note, the LMS marking issues were eventually addressed with the implementation of Gradescope. However, this did not occur until after these interviews were conducted.

---