Breathlessness resulting from heart failure (HF) manifesting as acute cardiogenic pulmonary oedema (ACPO) is a potentially life-threatening condition routinely encountered in paramedic practice (Greaves, 2005). Appropriate diagnosis and treatment of HF can reduce mortality and morbidity (NHS Executive, 2000), however prehospital differential diagnosis of HF by paramedics is sometimes unreliable (Schaider et al, 1995) and may lead to therapeutic interventions being withheld (Jenkinson et al, 2008) or the initiation of inappropriate and potentially harmful treatment (Wuerz and Meador, 1992). Existing studies report accuracy rates for HF diagnosis by paramedics ranging from sensitivities of 42.6% in the UK (Jenkinson et al, 2008) and 55% in the US (Schaider et al, 1995), to levels of concordance with physician diagnosis of over 80% in other US systems (Tresch et al, 1983; Pozner et al, 2003; Ackerman and Waldron, 2006).
While there is some evidence that specific training interventions in other areas of paramedic practice can enhance diagnostic accuracy (Wojner-Alexandrov et al, 2005), no studies have evaluated the effect of participation in a HF training intervention on diagnostic accuracy among UK undergraduate paramedics assessed using mannequin-based clinical simulation.
Methods
A two-stage repeated measures study was conducted during March and April 2008 to investigate the effect of participation in a targeted training intervention on self-reported confidence in aspects of HF management and diagnostic accuracy assessed through a range of medical simulation scenarios. Participants were recruited from a group of final year full-time BSc paramedic science students. Ethics approval was granted by the University of Hertfordshire Health and Emergency Professions Ethics Committee.
Power calculations were performed using G*Power 3 (Faul et al, 2007). It was felt that participants might reasonably be expected to initially achieve results at a midpoint between the lower sensitivity levels discussed above (Schaider et al, 1995; Jenkinson et al, 2008), and that participation in a training intervention would potentially enable participants to achieve the higher accuracy rates reported in other studies (Tresch et al, 1983; Pozner et al, 2003; Ackerman and Waldron, 2006). A priori power calculation indicated that a minimum sample size of 18 participants would be required to detect an increase in sensitivity for HF from 50% to 80% (α=0.05, β=0.87).
A study specific questionnaire was designed to collect demographic data and assess self-reported confidence via five-point Likert scale responses to a range of statements relating to assessment and management of HF (1=strongly disagree, 2= disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree). These included confidence in initial assessment, clinical examination, acquiring a medical and drug history, using ECG findings and administration of medications currently approved for UK paramedics in the management of HF (Joint Royal Colleges Ambulance Liaison Committee (JRCALC), 2006).
Immediately following completion of the questionnaire, participants undertook three scenarios utilising three SimMan mannequins (Laerdal Medical Corporation, USA) in the University of Hertfordshire simulation centre (Alinier, 2007). Scenarios were developed to simulate HF, chronic obstructive pulmonary disease (COPD) and pneumonia, the latter being the two conditions most commonly confused with HF in the prehospital phase (Tresch et al, 1983, Jenkinson et al, 2008).
Each scenario consisted of a SimMan mannequin with preset heart and lung sounds and vital signs, a sheet of paper consisting of a brief history of presenting complaint and a set of vital signs, a collection of medication containers, and a printed 12-lead ECG (Figure 1 and 2). Mannequin speech functions were not used and participants were not expected to interact with the mannequin beyond performing a physical examination as described. Individual scenario details and lists of acceptable diagnostic terms were approved by a panel of clinicians and researchers.
The study was conducted in a ward area of the simulation centre, with each mannequin located in a hospital bed surrounded by curtains. Participants were given up to 10 minutes to examine each mannequin and scrutinize clinical data before recording a diagnosis on a study specific form and then moving to the next simulation in a linear fashion. The researcher was present for the duration of the study in order to marshal participants and intervene in the event of technical failure.
Responses were compared against the predetermined list of acceptable diagnostic terms and marked as correct or incorrect accordingly. Two weeks after undertaking the initial simulation sessions, participants attended a 90 minute targeted HF training intervention addressing the pathophysiology of HF, other common causes of breathlessness, focused history-taking, differential diagnosis, ECG interpretation and pharmacology. The session concluded with a series of case studies to test diagnostic reasoning. A further two weeks following completion of the training intervention, the Likert scale section of the questionnaire was re-administered and participants undertook the simulation exercise again, although the order in which cases were presented was changed to minimize case spotting.
Statistical calculations were performed using the statistical package for the social sciences (SPSS Inc Chicago, Illinois) version 14.0, with statistical significance set at P<0.05. Data from the first simulation session were initially grouped according to whether the participant recorded a correct diagnosis for every scenario or provided one or more incorrect answers. Chi-square analysis was performed to investigate the relationship between test performance and demographic variables and to evaluate pre- and post intervention performance.
Pre- and post-intervention performance in individual simulation scenarios was assessed using McNemar's change test. Mann-Whitney U tests were used to investigate the relationship between Likert scale responses and test performance. Reliability analysis was performed on the Likert scale data using Cronbach's Alpha, with a value of above 0.7 regarded as acceptable. Analysis of the Likert scale responses pre- and post-intervention was conducted using the Wilcoxon signed ranks test.
Results
Twenty participants were initially enrolled. Of these, two withdrew before the study commenced. One further participant withdrew after completing the initial questionnaire and undertaking the first simulation session.
Of the initial 18 participants, 61.1% (n=11) were female. The majority of participants were aged 20-29 years (94.4%, n=17), with one aged 30–39. Most commonly, the highest level of education received by individuals prior to commencing paramedic training was at A-Level (83.3%, n=15). Two participants (11.1%) had completed degrees, and one had undertaken an access to higher education course. All had completed a sandwich year in clinical practice prior to returning for their final year of degree studies. Initial Likert scale responses to statements regarding confidence in using various components of the assessment process to differentiate HF from other causes of breathlessness are shown in Figure 3.
Agreement with statements regarding confidence was generally high, with ‘agree’ constituting the modal response for all assessment related items. Responses to statements regarding initial assessment and obtaining a focussed history demonstrated a narrow distribution, with all participants selecting either ‘agree’ (n=15 and n=17) or ‘strongly agree’ (n=3 and n=1). A narrow distribution was also seen in responses to the statement regarding physical examination findings, with 15 participants selecting ‘agree’ and 3 selecting ‘neither agree nor disagree’. In contrast, responses to statements regarding the use of ECG findings and reviewing medication were more widely distributed.
In terms of statements relating to prehospital drug therapy, ‘agree’ constituted the modal response (Figure 4). All respondents indicated agreement (n=15) or strong agreement (n=3) with the statement regarding administration of oxygen. The majority of participants selected ‘agree’ (n=13) or ‘strongly agree’ (n=4) in relation to administration of Glyceryl Trinitrate (GTN), with one respondent selecting ‘disagree.’ Responses to the statement regarding furosemide were more widely distributed.
Analysis of internal consistency using Cronbach's Alpha yielded a low score (a= 0.132), indicating that responses to individual items were not closely correlated. It was therefore inappropriate to calculate a total score for further statistical analysis beyond that performed for individual items.
Following initial simulation testing, three responses were judged to be incorrect for each scenario. Incorrect diagnoses included respiratory/chest infection (n=2) and peritonitis (n=1) in the HF scenario, respiratory/chest infection (n=2) and HF (n=1) in the COPD scenario, and HF (n=2) and pericarditis (n=1) in the pneumonia scenario. Sensitivity and specificity for HF diagnosis were 83% and 91.67% respectively. Age, gender and level of education were not associated with test performance (P>0.05). There was no significant correlation between diagnostic accuracy and levels of confidence (P>0.05).
Subsequent data analysis was performed on the 17 participants who completed all phases of the study. Following the second phase of testing, all diagnoses recorded for the HF scenario were judged to be correct. Two incorrect diagnoses were noted in the pneumonia scenario, namely septicaemia (n=1) and myocardial infarction (n=1). Diagnostic accuracy for the COPD scenario remained unchanged, with three incorrect diagnoses of respiratory/ chest infection recorded. No participants recorded a diagnosis of HF in the COPD or pneumonia scenarios. Pre and post intervention sensitivity and specificity for each scenario are shown in Table 1.
| Sensitivity % | Specificity % | |||
|---|---|---|---|---|
| Pre | Post | Pre | Post | |
| Heart failure | 83 | 100 | 91.67 | 100 |
| COPD | 89 | 82 | 100 | 100 |
| Pneumonia | 83 | 88 | 91.67 | 94.1 |
Post-hoc power calculations indicated that a minimum sample size of 17 was required to detect a statistically significant difference (α= 0.04, β=0.84). There were no statistically significant changes in pre-and post-intervention diagnostic accuracy assessed through Chi-square analysis of participants with correct responses to all scenarios vs those with one or more incorrect answers.
Comparison of pre- and post-test performance in individual scenarios using NcNemar's test revealed a non-significant trend towards improved recognition of HF. Level of diagnostic accuracy for COPD and pneumonia remained largely unchanged, although none of the participants mistook these conditions for HF. There were no statistically significant changes in Likert scale responses, with the exception of a significant improvement in confidence in using ECG findings to differentiate HF from other causes of breathlessness (z= -2.309, P=0.021). A non-significant trend towards improved confidence in obtaining a focused history to differentiate causes of breathlessness was also observed (z= 1.890, P=0.59).
Discussion
Pre-test diagnostic sensitivity for HF was higher than expected when compared with results from a single clinical study conducted by Jenkinson et al (2008) in a UK setting (42.6% vs 83%), but on par with the diagnostic concordance levels of >80% reported in several US studies (Tresch et al, 1983; Schaider et al, 1995; Pozner et al, 2003; Ackerman and Waldron, 2006). This may refect participant demographics as there is evidence of improved clinical performance among graduate practitioners from other professions when compared with practitioners with less extensive education (Clinton et al, 2005).
Equally, it may refect the difficulties inherent in comparing results from simulation based assessment with those gained from clinical studies. In this study, COPD and pneumonia were the conditions most commonly mistaken for HF. This is in keeping with existing studies, which have reported similar results from the 1980s to the present day, thus highlighting the persistent nature of this problem in paramedic practice (Tresch et al, 1983; Schaider et al, 1995; Pozner et al, 2003; Ackerman and Waldron, 2006; Jenkinson et al, 2008).
Although calculated in the context of a repeated measures design involving a small number of mannequin simulation cases, both sensitivity and specificity increased in relation to HF. Corresponding values for COPD and pneumonia remained largely unchanged, although these conditions were not primarily targeted by the training intervention.
Inappropriate initiation of treatment regimes intended for HF to patients with COPD or pneumonia is associated with worsened outcomes (Hoffman and Reynolds, 1987; Wuerz and Meador, 1992), while administration of treatment for COPD or pneumonia in HF cases may mean that patients are denied the symptom relief afforded by appropriate prehospital drug therapy (Jenkinson et al, 2008). In contrast, prehospital management of COPD and pneumonia is largely confined to administration of oxygen and bronchodilators (JRCALC, 2006), therefore the diagnostic confusion observed between these two conditions in this study is arguably of negligible clinical significance.
Initial questionnaire results indicated broad agreement with statements relating to levels of confidence in various aspects of assessment. This is perhaps unsurprising given the level of undergraduate education participants had already received. Pre- and post-intervention Likert scale responses remained largely unchanged, with the exception of the use of ECG findings, where a statistically significant increase in self reported confidence was noted (z= -2.309, P=0.021). Pre-registration training requires that paramedics are able to identify a range of arrhythmias amenable to diagnosis via lead II alone and scrutinize 12-lead ECG's to identify features suggestive of acute ST elevation myocardial infarction (JRCALC, 2006; Ambulance Service Association, 2007).
There is no standard requirement for paramedics to use the ECG to identify other features that may be seen in association with HF, such as left ventricular hypertrophy (Morris et al, 2008). Although the study design limits the extent to which the inference can be made that this change was due to the study intervention, recognition of 12-lead ECG features seen in HF did form part of this training.
Questionnaire results relating to the administration of medications were more varied. Oxygen therapy is commonly provided by paramedics for a wide range of conditions (Greaves, 2005), therefore the high levels of confidence observed in this area are not unexpected. Confidence in administration of GTN, which is licensed for sublingual administration by paramedics in both HF and ischaemic chest pain, was also high (JRCALC, 2006).
This route of administration is minimally invasive, and student paramedics are permitted to administer this agent from an early stage in their training. This may in part provide some explanation for these findings, as chest pain and HF are common presenting complaints in paramedic practice and therefore the students might reasonably be expected to have gained a degree of experience in using this drug (Greaves, 2005). Responses relating to confidence in the use of furosemide were less evenly distributed, potentially indicating that participants were less confident in using this drug.
UK practice guidelines dictate that this drug is reserved for severe HF where an adequate response to GTN has not been achieved and transfer time to hospital is prolonged (JRCALC, 2006). In comparison to GTN, student paramedics are not permitted to administer furosemide until much later in their training, therefore it is reasonable to assume that study participants would have had less experience in using it.
Furthermore, evidence suggests that furosemide is particularly harmful when administered inappropriately, and there are some calls for it to be withdrawn from prehospital use (Holliman et al, 1992; Nieminen et al, 2005; McKinney et al, 2009). It is possible that some participants would have been aware of these controversies, and that this may have influenced individual responses.
‘Initial diagnostic accuracy of undergraduate paramedics assessed through clinical simulation was higher than anticipated’
Limitations
Clearly this study would have been strengthened through the use of a control group to enable more robust inferences to be made. The activities of the cohort could not be controlled between study phases, and participation in the initial phase of testing may have prompted participants to undertake relevant self directed study in addition to that provided through the training intervention.
Consideration must also be given to the use of clinical simulation in assessing diagnostic accuracy. A gold standard test might involve the use of live patients in real clinical scenarios. This approach was not considered appropriate due to the requirement for and challenges associated with obtaining informed consent from patients in prehospital emergency situations (Kuehl, 2002; Dubois, 2006), issues regarding patient safety, and the large number of patient encounters that would be required to obtain an adequate sample of HF patients on the basis of prevalence rates quoted in other studies (Jenkinson et al, 2008).
Simulated patients played by actors have been used to good effect in other areas of medical education, and permit students to be examined in areas such as communication, history-taking, and responses to non-verbal cues (Smee, 2003). These are aspects of the assessment process that cannot be readily assessed through mannequin-based clinical simulation alone, yet they are of the utmost importance in the diagnostic process (Chow and Naik, 2008).
Participants were subjected to the same simulation scenarios during each phase of testing, although the order in which they were encountered was changed in an attempt to minimize case spotting. Nonetheless, the potential for participants to recognize scenarios from the preceding phase of testing cannot be ignored. There is also evidence that participation in medical simulation may also improve future clinical performance (Okuda et al, 2009).
It would have been possible to develop different scenarios for the second phase of testing, but this would have been at the expense of repeatability and not in keeping with a true repeated measures design (Elwood, 2007). Each simulation scenario comprised a range of clinical data and diagnostic cues, yet the study did not investigate whether these had been correctly identified. Participant perception of the relative importance of each component part of individual clinical scenarios in terms of clinical decision making was also not addressed.
Conclusion
Initial diagnostic accuracy of undergraduate paramedics assessed through clinical simulation in this study was higher than anticipated and that reported in a previous UK clinical study (Jenkinson et al, 2008). A non-statistically significant trend towards increased diagnostic accuracy and a statistically significant improvement in self reported confidence relating to use of 12-lead ECG findings (z= -2.309, P=0.021) were observed following participation in a targeted HF training intervention. During the second phase of testing, all participants correctly identified HF and none mistook COPD or pneumonia for HF.
Application of these results to wider clinical practice suggests that clinical benefits could be realized for patients presenting with HF and breathlessness manifesting as ACPO with more widespread implementation of targeted HF training interventions and access to clinical simulation training facilities.
The performance of undergraduate paramedics in this study may also provide some of the reassurance sought by the Department of Health (DH) regarding the clinical value added by a transition to degree level education for paramedics (DH, 2008). Further research involving the wider paramedic profession and more clinically relevant methods of assessment of diagnostic accuracy other than simulation in relation to HF is warranted.