The use of strict assessment criteria or adopting a heavily prescriptive structure may improve the reliability factor of paramedic portfolios but would obliterate the essence of portfolios in terms of flexibility, personal orientation and authenticity (Driessen et al, 2005). There is also a recognition that a professional's competence is more complex than a neatly prescribed set of competencies:
‘The competence of professionals derives from their possessing a set of relevant attributes such as knowledge, skills and attitudes. These attributes which jointly underlie competence are often referred to as competencies. So a competency is a combination of attributes underlying some aspect of successful professional performance’
The above sentiment resonates with the work of Bloom (1956) who also cited knowledge (cognitive domain), skills (psychomotor focus) and attitudes (affective domain) as being central to any educational outcome (Bloom, 1956). Applying this revised taxonomy to the use of portfolios in paramedic practice, the author proposes one possible interpretation as follows:
No single method is appropriate for assessing clinical competence (Norman et al, 2002; Wilkinson et al, 2002) and as such, portfolios could be seen as effective at documenting both the competency-based and self-directed approaches to demonstrating a paramedic's fitness to practice.
Summative or formative in design
An alternative perspective on portfolios, however, is that they are either summative or formative in design. That is to say, portfolios either focus on displaying learning outcomes/competencies and contain evidence that these have been met (summative) or that portfolios demonstrate a learning process where no final mark or grade is awarded (formative) (Timmins, 2008).
It is when portfolios are used for both summative and formative purposes, especially a summative assessment tool, that confusion and tensions emerge (Wilkinson et al, 2002); although Menix (2007) stresses the importance of collecting and evaluating both summative and formative data (Menix, 2007).
This theory is developed further by Webb et al (2002) who have proposed different models of portfolios according to their intended purpose. Adopting a gastronomical theme, they have designed four different portfolio models with varying purposes—from a simple ‘tick box’ format to a portfolio which collates a body of evidence about the learning which has taken place (Webb et al, 2002).
Strength of portfolios
The strength of portfolios derives from their ability to offer rich and authentic evidence of a learner's achievement and/or a practitioner's professional development (Driessen et al, 2006) but within the health professions, there has been a paradigm shift from process-based to competency-based education and measurement of outcomes (Carraccio et al, 2002).
Yet, educational outcomes are more than simply the retention of hard ‘measurable’ facts (Bloom, 1956; Knowles, 1975; Kolb, 1984; Boud and Feletti, 1996; Burnard, 1998; Chambers and Wall, 2000; Anderson et al, 2001) and within paramedic practice and the allied health professions, there is a fundamental need for understanding in order to develop clinical skills and make informed clinical decisions based on underpinning knowledge and clinical reasoning.
Evaluation
To help evaluate the ability of paramedic portfolios to encompass these diverse themes, Steinaker and Bell's (1979) approach to evaluation is useful as it differentiates between process evaluation (what methods and strategies are affective) and product evaluation (a summative approach) (Steinaker and Bell, 1979). This is consistent with the earlier work of Donabedian (1999) who also highlighted the need to evaluate structure and process within health care systems, but also the outcome in terms of effectiveness of the interventions for enhancing health and wellbeing (Donadebian, 1966).
A number of evaluation models centred on health care place the emphasis on quantitative factors, such as Abruzzese's (1996) model which considers organizational effectiveness indicators in the form of retention, turnover and cost effectiveness (Abruzzese, 1996).
Cobb and Billings (2000) make reference to teaching and learning practices but also costs. A more generic education model is presented by Wagner and Weigand (2002) who take a broad view of the impact of training on an organization, linking behavioural changes with results (Wagner and Weigand, 2002).
The language of commerce does not sit easily with the author's perspective of health care, although this is perhaps a naïve view given the challenging financial climate that all areas of society are operating within at present. At least Lewis (2000) supports a model of evaluation which sees a return to language more readily associated with a ‘purist’ ideal of education, with an emphasis on learning outcomes (Lewis, 2000). Such an evaluation has an affinity with both competency-based and self-directed learning but as with any valuing theory, there is a need to determine criteria, set standards and measure the outcome (Scriven, 1980). Essentially, any evaluation needs to combine both qualitative and quantitative methodologies (Cantillon, 1999) although evaluating competence based on qualitative indicators is still evolving (Byrne et al, 2009).
Constraints
The primary constraint with relying on a set of defined competencies within the Knowledge and Skills Framework is that a paramedic's competence is more than the sum of each discrete element of their job role and the content and component parts of each competence is comparatively easy to construct than to predict the performance of each paramedic working to these competencies (Tuxworth, 1989).Defined competences are too narrow (Barnett, 1994) and requiring a paramedic to produce evidence of attainment of a particular competence is reductionist, ignoring the underlying attributes and the complexity of paramedic practice (Bloom, 1956; Manley and Garbett, 2000; McAlleer and Hamill, 2003).
It is also unclear how a competency-based approach accommodates those paramedics with learning difficulties. A paramedic may have difficulty documenting how they have met a specific competence but in practice, are fully able to apply that particular skill. A competency-based approach fails to measure the underlying cognitive and affective skills needed for effective practice (Savery and Duffy, 1995; Le Var, 1996; Lillyman, 1998) and technical skills are emphasized at the expense of knowledge and understanding (Ashworth and Morrison, 1991).
‘A paramedic may have difficulty documenting how they have met a specific competence but in practice, are fully able to apply that particular skill’
Competencies
Competencies are also representative of the behavioural approach which is individualistic in orientation (Ashworth and Saxton, 1990; Ashworth, 2003) but an integral part of a paramedic's job role, as with most health professionals, is to work within a team. Providing evidence of that aspect of competence could prove problematic if a portfolio is solely concerned with displaying quantitative evidence of a specific KSF dimension. There are also issues of transferability of competence, as a reliance on a performance approach to assess professional competence gives no guarantee that an individual will perform appropriately in a variety of contexts (Norris, 1991).
Self-evaluation
The importance of context is fundamental to the evidence provided by a paramedic through their personal reflections. This self-evaluation approach can be problematic in itself as the evidence presented by a paramedic through reflexivity is interpreted solely by the individual paramedic (Usher, 1993).
Boud and Walker (1993) have also identified a number of barriers to reflection and cite previous negative experiences or a lack of awareness of the individual adversely effecting the process of reflection. This observation is supported in the wider literature:
‘Individuals differ in their ability to self-reflect and self-evaluate. Some people may be too hard on themselves and focus too much on their own weaknesses. The portfolio may then come across as lacking in strength and substance and therefore not give a clear picture of the individual. Alternatively, an individual may concentrate too much on his or her self-identified strengths and therefore portray an image of himself or herself that is not entirely accurate’
Self-directed learning, by its very definition, will always involve a substantive element of individualism and a paramedic's own interpretation of experiences cannot be dismissed if reflection is to be considered as part of their evidence of professional competence.
Reflection
Considering the use of reflection from another perspective generates a concern over the very personal nature of reflections. The author's own portfolio does not contain a section on reflection as he considers his reflections private and personal to himself. Reporting on the qualitative findings of a study of 253 diploma of nursing students and their use of portfolios in clinical practice, McMullan (2008) noted a similar response from a number of nursing students who also felt that reflexivity was a very personal process and not one to be documented in a portfolio (McMullan, 2008).
The obvious problem with this approach is that if a paramedic's reflections are not documented and no evidence is included in a portfolio that a paramedic has engaged in reflexivity, then it is conceivable that any assessor viewing the portfolio could interpret the omission as being a failure to engage in any CPD (the principal reason that approximately 10% of paramedics selected for audit had their portfolios returned to them so that additional evidence could be documented) (HPC, 2010b)
To be most effective, however, self-directed learning needs to be used in conjunction with some variation of external competency evaluation (Lenburg, 2000) as any work-based learning needs to incorporate both critical reflection and more specific content learning (Onyx, 2001).
Recommendations
The author believes that within the paramedic profession, portfolios should be used in tandem with other assessment methods of competence, such as direct observation, to determine if a paramedic is competent (Wolf, 1989; Gonzi et al. 1994).
This would help to attain sufficient breadth of evidence, including reflection, to demonstrate the competence of a paramedic and provide a match between the evidence presented in a portfolio and clinical practice (Wailey, 2002).
To further ensure the validity of any evidence in a portfolio, the author argues that some form of verification of portfolio content prior to submission for audit would benefit the profession. This may involve an element of peer review, although issues concerning data protection could arise. A more practical solution to enhancing the validity, and reliability, of a paramedic's evidence of fitness to practice could lie with the introduction of a rating scale, as advocated by Byrne et al (2007) and Karlowicz (2010).
The author would also support a revision of the HPC's template portfolio and develop it to more closely mirror the guidance offered to state registered paramedics for completing their portfolio of CPD. In its current form, the author believes the HPC's template portfolio is too rudimentary and does not effectively capture the diverse array of CPD activities a paramedic can engage in to demonstrate their competence.
A further alteration to the current use of portfolios in paramedic practice regards the audit itself. The author is in favour of a greater percentage of the profession being audited in 2011 as the numbers holding a licence to practice as a paramedic in the UK increase.
Accepting the potential cost implications of such a development, increasing the number of paramedic portfolios being audited could result in a reduction (statistically) of failing to identify those paramedics who are not fit to practice and bring the audit process in line with other professions who have a far larger number of registrants on the HPC register. Similarly, as different roles and skill sets evolve within the paramedic profession, a review of portfolio format and what constitutes evidence of fitness to practice should be conducted.
A final recommendation proposed by the author is for a comprehensive review to be conducted of the 2009 audit of paramedic portfolios, involving all those who were involved in the process. Incorporating the principles of 360º feedback, such a review could lead to improved reliability and validity of the portfolio audit process (Morgeson et al. 2005). It is recognized within higher education that student involvement in the evaluation of learning programmes is the cornerstone of quality improvement (Hendry et al, 2001) but these principles are transferable outside of formal education (Bernardin and Beatty, 2005; London and Smither, 2005; Hoffman, 2005; Antonioni, 2005).
A more in-depth evaluation of the different models of portfolios and a more comprehensive look at the different approaches to assessing portfolio content would be beneficial. But what this evaluation has identified are the two core education strategies (competency-based vs self-directed learning) used within paramedic portfolios and the core concept of what constitutes as evidence of CPD.
Conclusion
There is an uneasy relationship between documenting CPD and assessing a paramedic's competence and questions exists as to whether this is even practical. These are two separate measurements, with one not necessarily implying the other has been met. The author believes a portfolio is best suited to documenting a paramedic's CPD. The challenge for the profession, and HPC in particular, is to find a way to retain this focus while incorporating a more summative context.
It is perhaps understandable that portfolios should be seen as a means to demonstrate a paramedic's fitness to practice. CPD undoubtedly forms part of that equation but the summative assessment of a paramedic's competence needs to be more robust than the compilation of a portfolio.
A combination of direct observation, observed structured clinical examinations or even written examinations (to incorporate the more theoretical or governance aspects of the paramedic role) should all be used in tandem with a portfolio.
The portfolio measures what is valued by the individual paramedic. When this changes to placing a value on what is measured, portfolios within paramedic practice assume a very different dimension.