Thinking Differently - Students’ Cognitive Processes When

Assignment Task

Background

Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students’ cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions.

Methods

We undertook a ‘think aloud’ study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used.

Result

Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students.

Conclusion

Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.

Introduction

Single best answer questions (SBAQs) have dominated written medical examinations for many years (Coderre et al. 2004; Heist et al. 2014) and have several advantages including high internal consistency and ease of marking (Wass et al. 2001; Pugh et al. 2019). However, SBAQs have been criticised for not being an authentic representation of clinical practice, because candidates may use the cues provided in the answer options to arrive at the correct answer (McCoubrie 2004; Raduta 2013; Heist et al. 2014). The very shortanswer question (VSAQ) format provides one potential solution to the cueing and lack of authenticity posed by SBAQs. In the VSAQ format, the candidate is still presented with a clinical vignette and lead-in question, however they are required to independently generate a very short answer (of 1–5 words) rather than choosing the correct answer from a list of answer options. Furthermore, the development of a computeradministered VSAQ assessment has allowed for semi-automated, time-efficient and consistent marking, making them a feasible alternative to SBAQs.

Quantitative studies have demonstrated that students’ scores on SBAQs are significantly higher than on comparable VSAQs (Sam et al. 2016, 2018, 2019), which suggests that students can obtain useful information from the provision of answer options in SBAQs, or are able to enhance their scores through guessing. These hypotheses can be explored in qualitative think-aloud studies. Previous work in this area (Coderre et al. 2004; Heist et al. 2014; Durning et al. 2015; Surry et al. 2017) has focused on the thought processes used for SBAQs. These studies have consistently shown that students and doctors use both analytical and non-analytical reasoning strategies to answer SBAQs; an approach that aligns with the ‘processes expected in real-world clinical reasoning’ (Surry et al. 2017) (p.1075, our emphasis). However, these studies also reveal the use of ‘test-taking’ behaviours that reduce the authenticity of this type of question, such as the deliberate elimination of alternatives. The test-taking behaviours associated with the use of the answer options in SBAQs would simply not be possible with VSAQs. As VSAQs are a novel question format in medical education, there is a dearth of literature comparing the cognitive approaches to different question formats. A previous study compared the reasoning strategies used to answer clinical questions presented in either short answer or extended matching question formats (Heemskerk et al. 2008). The authors report more use of scheme-inductive reasoning (data to diagnosis) for short answer questions and of hypothetico-deductive reasoning for extended matching questions. In addition to comparing the generic thought processes employed to answer SBAQs with those for VSAQs, it is also useful to investigate whether there are differences in approach between high and low performing students. Previous work has suggested low performers are more likely to exhibit premature closure and less likely to engage in a process of ruling out alternatives when answering SBAQs compared to high performers (Heist et al. 2014). Further exploration of these differences could help medical educators understand areas of cognitive error and thus plan more effective teaching. We undertook a ‘think-aloud’ study of SBAQs and VSAQs with final year medical students in order to explore the following research questions:

1. Do students use different methods to answer VSAQs compared to SBAQs?

2. Is there an association between student performance (as measured by scores in a linked quantitative study (Sam et al. 2019)) and the methods used to answer questions?

Methods

Our description of the research methods employed in this study is based on the structure suggested by O’Brien et al. (2014) in their Standards for reporting qualitative research

Qualitative approach

We used a ‘think aloud’ study design, whereby participants are asked to voice the thoughts that occur to them as they complete a task (Hevey 2012), in this case answering SBAQs and VSAQs. We subsequently used a content analysis approach and derived our initial content themes from a previous think aloud study undertaken by Surry et al. (2017).

Context

Interviews were conducted in private rooms at each of the five UK medical schools in which this study took place.

Ethical review

This study was approved by the Medical Education Ethics Committee at Imperial College London (Reference MEEC1718-100). All students were provided with information about the study at least 48 hours in advance of the interview day and gave informed written consent to participate on the day of the interview. All interviews were conducted using participant codes rather than names. Participants were given a copy of all the questions used (with answers and explanations).

Data collection methods

Face-to-face interviews were held on one day per school during the period October to December 2019, each conducted by one study investigator (AHS, RWi, CB, RWe) who was not connected to the medical school in question. All interviews were digitally recorded for subsequent transcription. Students were first given up to 16 minutes to complete eight VSAQs under exam conditions, followed by the interview (described below), then up to 12 minutes to complete eight SBAQs, again followed by the interview. Students who were entitled to extra time in their university examinations were given an additional 25% of the time allowed. Like others (Durning et al. 2015; Surry et al. 2017), we used a retrospective approach (i.e. students answered the questions and then explained their thought processes) to mimic a real-life examination and to avoid the first explanations influencing how subsequent questions were answered.

Data collection instruments

Each student answered eight VSAQs and eight SBAQs, written by members of the research team specifically for this study (Supplement Appendix 1). Four questions (Q1-4) were used in both formats (i.e. five answer options were added to make the question into SBAQ format from the VSAQ format) as the first four VSAQs and last four SBAQs on each student’s paper. A further four pairs of questions were written (Q5-8), which could all be answered in either VSAQ or SBAQ format. The two questions within each pair were ‘matched’ on the basis of specialty and approximate difficulty, as determined subjectively by the question writers. Students answered one question in each pair as a VSAQ and one as an SBAQ, with the version determined at random. The four questions within each block were presented to each student in random order. Students were then asked about their approach to each question in turn using the prompts ‘What steps did you take to answer the question?’ and ‘What were your thoughts during each step?’ Students were asked to explain their thoughts if necessary, for example, why they eliminated certain answers, why they highlighted particular words.

Units of study The transcripts were coded at question-level i.e., each cognitive method (see below) was marked as present or absent for each student-question combination. In this respect, we quantitised the qualitative data using categorisation, as previously described (Onwuegbuzie and Teddlie 2003), to enable us to compare the relative use of each cognitive method for each question format and between the two student performance groups.

Data processing

Interview transcripts were anonymised prior to analysis, such that coding was undertaken blind to each student’s medical school and performance group. Coding was undertaken using NVivo as described below.

Data analysis

Surry and colleagues (2017) identified four key themes in their work (analytical clinical reasoning, nonanalytical clinical reasoning, test-taking behaviours and reactions to the question), and we used these themes as our sensitising framework. Based on these themes, we identified the methods used to answer each question within four main ‘method groups’ (Figure 1). CB and RWi iteratively developed the first version of the codebook for this study through collaboratively coding one randomly-selected transcript, followed by independent coding and discussion of two further randomly-selected transcripts and development of the second version of the codebook. Independent coding of a further three randomly-selected transcripts was then undertaken by four team members and the results discussed at a team meeting. The resulting revised codebook was applied independently by at least three team members to two further randomly selected transcripts in turn. At this stage, no further changes to the codebook were made, and this was subsequently applied by four team members to code the eighth transcript, with discussion of any discrepancies. All transcripts (including those used in codebook development) were then coded by RWi using the final codebook, which also gives examples of each method as found in the transcripts (Supplement Appendix 2). CB checked the coding of all transcripts, with any discrepancies resolved via discussion with AHS. A coding results table for each student was produced, identifying which methods were present for each question in each format.

To address research question 1 (comparison of methods used), we only used the question pairs (Q5-8) because the pairing provided approximate comparability between the VSAQs and SBAQs. We aggregated data first within each individual method and then within each method group (analytical reasoning, nonanalytical reasoning, test-taking behaviours and acknowledging uncertainty). We summed the number of times a method/method group was used across all student-question combinations for each question type and converted this into a relative frequency by dividing the total by the number of times each method/method group could have been used.

To address research question 2 (association between methods and performance), we used the four pairs of questions (Q5-8), plus the additional four items that were answered by all students as VSAQs (Q1-4). We chose not to use the results from the four SBAQs items that students had already answered as a VSAQ (Q1- 4), because we found that students’ descriptions of their thought processes for these questions were very brief and often focused on the VSAQ version. We present results for the relative frequency with which each method group was used (calculated as for research question 1). The results for each research question are presented graphically, using quotes to illustrate key findings.

Comparison of methods used to answer VSAQs and SBAQs

 The total uses of each method group/method as a percentage of the usage opportunities for that method group/method, respectively. We found that students demonstrated analytical reasoning more often when answering VSAQs compared to SBAQs, with 41% of usage opportunities taken when answering VSAQs compared to 35% for SBAQs. The biggest difference between question types within the analytical reasoning methods group was in the use of key features (38%/23% of VSAQ/SBAQ usage opportunities). The most common method overall was hypothetico-deductive reasoning (82%/74% VSAQ/SBAQ usage opportunities). There were similar levels of use of each non-analytical reasoning method across the two question formats. Students used test-taking behaviours more frequently when answering SBAQs, with 16% of usage opportunities taken when answering VSAQs compared with 27% for SBAQs (excluding the method SBA answer options from the denominator for VSAQs). Students made extensive use of SBAQ answer options (77% SBAQ usage opportunities), but students were also twice as likely to use buzz-words when answering SBAQs compared to VSAQs (33%/17% SBAQ/VSAQ usage opportunities). Students acknowledged uncertainty more with VSAQs (43%/32% VSAQ/SBAQ usage opportunities), generally due to differences in expressions of uncertainty as to the correct answer (25%/7% VSAQ/SBAQ usage opportunities).

The following three quotes illustrate the benefit students obtain from using the answer options for SBAQs (i.e., cueing) and how these reduce students’ uncertainty regarding the correct answer:

“I went through each specific one and thought to myself, how likely is this? Rule it out immediately or keep it in for now.” Student 1

“But when they’d given me this lovely option of serum electrophoresis that swung me definitely more towards multiple myeloma.” Student 2

“I think, if, if I'd have seen that option in a single best answer scenario, I may have been more confident, but the fact that there, there [sic] is essentially unlimited options between the two, it makes it slightly more tricky.” Student 7

The second of these quotes highlights how the answer options can help students form a diagnosis, as well as helping them arrive at the correct answer, as illustrated in the first quote.

Association between student performance and methods used

The relative frequency of use of each method group comparing results by question type and student performance group. The key differences between the two performance groups are that, compared to the low performing group, the high performing group tended to have greater use of analytical reasoning for both question types, greater use of non-analytical reasoning primarily for VSAQs and slightly less use of test-taking behaviours for SBAQs. The high performing group also acknowledged uncertainty more frequently than the low performing group for both question types, in particular about what the question is asking for VSAQs and about the correct answer for SBAQs.

Qualitative differences in cognitive approaches between high and low performing students are highlighted in the following example. One question asked students to name the most appropriate diagnostic investigation for a patient with features of multiple myeloma. Student 3, from the low performance group, used pattern recognition and gut instinct (and arrives at an incorrect answer), whilst Student 1, from the high-performance group uses hypothetico-deductive and pathophysiological reasoning (and arrives at a correct answer).

“And then I saw that his calcium was high, his creatinine was high, and his haemoglobin was quite low, so I again thought this was a sort of a cancer progressing sort of question. I thought the BPH [benign prostatic hypertrophy] had progressed into like a carc [carcinoma]… like an actual prostate cancer, that’s what I got from the question. So what investigation is going to confirm it and I just put serum PSA.” Student 3, low performing

“White cells are normal so it’s less likely to be infective, potentially less likely to be inflammatory as well. Platelets are normal and so also potentially not a clotting factor or a thrombocytopenic disease or necessarily a potentially an inflammatory disease. … At this point I was thinking that there seems to be some element of renal failure. Is that cause of his back pain and tiredness in the form of renal osteodystrophy? And, you know, his calcium is quite high so that could also be contributing to his tiredness, I think, at that point. So, I was thinking is CKD but with the ESR as well quite high I think that is also a red flag in this picture of back pain, renal failure and tiredness for multiple myeloma.” Student 1, high performing

Both performance groups used the answer options provided in the SBAQs to help them answer the questions – that is, benefitted from cueing - as illustrated by the following quotes:

“It seemed like a picture of gout and specifically the answers are cueing me towards that much more because there is no talk of antibiotics or joint arthroscopy or anything like that for septic arthritis.” Student 1, high performing group

“I knew that he needed imaging, and I did consider a CT thorax as I was reading, but then I saw that answer C specifically said high resolution CT, at which point I thought, Oh, that’s definitely the answer then.” Student 20, low performing group

Conclusion

VSAQs were more likely to be answered using analytical reasoning and generated less test-taking behaviours compared with SBAQs. These results suggest that students apply more authentic clinical reasoning strategies when answering VSAQs as opposed to SBAQs. Even in this very small sample of questions, there was evidence of positive cueing with SBAQs, where the answer options helped students arrive at the correct answer resulting in SBAQ scores being higher than VSAQ scores, as has been reported in our previous large-scale study (Sam, Westacott et al. 2019). This adds further evidence to suggest that VSAQs are more likely to promote deep rather than surface learning and increase the validity of written assessment.

This Education has been solved by our PhD Experts at My Uni Paper.