Use of PBAs to derive educational value

One of the proposed educational values of PBAs as WBA tools is their ability to identify the trainee in difficulty;20 however, this was not necessarily the experience of trainees in this study:I am sure that certainly in my last placement there were times where I was a failing trainee in the sense that I didn't feel that I was getting the hang of a particular operation or I was frustrated in my training but PBA didn't rescue me.ST3 general surgery

This interviewee felt that PBAs are not being used in the manner in which they were designed and intended. In order for PBAs to be effective: (1) the trainer would have to observe this trainee undertaking the procedure; (2) the trainer would have to give feedback on the procedure; (3) this feedback would be specific and detailed; (4) the feedback would be formative, addressing this trainee's shortcomings in relation to the ‘particular operation’ and offer targeted direction to overcome any technical or non-technical deficiencies the trainee may feel that they have.

Another trainee described the challenges related to imposing a quota necessary for PBAs to be completed and that leading to a box ticking, educationally invalid experience; this view was shared by all subsequent interviewees. These points are echoed by the results of a survey of 500 medical trainees conducted by the Joint Royal Colleges of Physicians Training Board (JRCPTB) where 67% of trainees stated that WBAs were too time-consuming, with a large proportion concerned that their supervisors did not have time to complete the assessments, often leaving trainees to complete them—including writing feedback.21 Furthermore, a survey of supervisors by the JRCPTB showed that two-thirds did not find WBAs effective in identifying underperforming trainees21 and that many felt WBAs to be an unproductive ‘tick box’ exercise. Bindal et al22 identified time pressure as a major source of dissatisfaction, and claimed that WBAs were widely viewed by trainees as a summative tick box exercise with little educational value.23

Interviewee 10 provided the perspective of a trainee who is nearing the end of their training, who requires global feedback and focused training to meet the requirements of independent practice. This is educationally at odds with the ‘point-by-point’ feedback mandated by PBAs:I am wanting to be assessed more globally. A sort of overall impression of “am I almost ready for consultant practice or not?” rather than a specific nitty gritty “did you do this? Yes/No” and “did you do it” or on a particular point “satisfactory” or “ready for CCT”. I don't know. I think that they task down into so many tiny little constituent parts that sometimes the global overview is lost.ST7 general (colorectal) surgery

Indeed, from October 2013, the JRCPTB reverted to a multiple consultant report to offer structured feedback on trainees' overall clinical abilities from a range of supervisors.21 The use of assessments performed on multiple occasions, by multiple assessors, in this way allows for ‘triangulation’ of assessment, which is an important determinant of assessment reliability.20 ,23 ,24 Interviewee 10's desire to be assessed more globally resonates with concerns25 ,26 voiced internationally that the introduction of competency-based postgraduate medical training created a gap between the assessment of specific competencies and actual clinical practice, emphasising competency over expertise. In order to bridge this gap ten Cate and Scheele argue for the use of entrustable professional activities (EPAs), a process that involves assessing global performance in all the competencies required to carry out critical professional activities such as running an operating list or perioperative care.25 Essentially, EPAs assess an individual's readiness to be entrusted with professional responsibility.

Use of PBAs for assessment

Previous work has demonstrated the effectiveness of PBAs for summative assessment of specific procedures and performance correlates well with surgical experience.14 Based on this, the JCST has advised a minimum number of PBAs to demonstrate minimum competence in certain specified procedures. However, the summative role of PBAs appears to have expanded beyond this remit. Further quantifying numbers of PBAs may have the unintended consequence of trainees viewing all PBAs as summative assessments. A number of points and areas for concern were raised by participants in relation to the use of PBAs for assessment. First, some trainees felt that PBAs were not specific to a level of training. In particular, one senior trainee felt that they were not fit for purpose as an assessment tool for more senior trainees as, owing to their ‘point-by-point’ nature, a global overview is lost. Second, it was the opinion of some interviewees that the use of PBAs for summative assessment conflicted with and prevented their use as an effective formative learning tool. Third, in the experience of our cohort, many of their trainers had not been trained to complete PBAs, or told trainees to complete their own PBAs. This may reduce the validity of the PBA outcomes for formative or summative assessment.

Interviewee 5 in particular raised many interesting points about the use of WBA as tools for formative feedback rather than summative assessment and the validity of the PBA as a tool for summative assessment:Whereas I suspect that if that aspect of their use was completely put to the side and you were just told this is a purely formative tool for your interest etc., I suspect trainees would apply them a bit more like they were designed …So I still fundamentally think that a report from a trainer over 6 months or a year or your progress and general competency in multiple fields is by far the most powerful tool although it's difficult to quantify numerically or whatever like a workplace based assessment does. So my personal view is workplace based assessments should firmly be formative tools that aid in the teaching process or the training process and perhaps their number can be used as a surrogate of engagement with the process but I do not think that this should be used as a summative tool.ST6 general (vascular) surgery

The view of interviewee 5 was supported by interviewee 3 with regard to the importance of PBAs for formative purposes:I think the most important thing about all of these things is the feedback. It's not the mark you get at the end of the day.ST3 general (colorectal) surgery

There appears to be a misalignment of conception between those who developed WBAs (for formative use) and the JCST which has implemented WBAs (for a combination of formative and summative purposes).27 This quote is one of a number of examples given that relate to concerns over implementation of PBAs, as opposed to, primarily, an issue with the content and format of PBAs.

In a recent survey of orthopaedic surgeons, less than half of trainees and trainers had received training in the use of PBAs.28 Interestingly, 53% (67% of consultants and 44% of registrars) were unsure whether PBAs had a formative or summative assessment purpose, and 13% (10% of consultants and 15% of registrars) believed they were summative.28 The article concluded with the important reminders that all users should familiarise themselves with PBAs, that PBAs were designed to be formative and that successful completion of a PBA at any level does not constitute a ‘licence to operate’.28

Some argue that the key to the success of the PBA is careful implementation.29 Provided trainees receive feedback and have confidence in the assessment system, improvement in competence will take place.29 ,30 However, this link between feedback and actual improved practice is not given. Indeed, it has been reported that only 8% of feedback or areas for development identified through mini-CEX were converted into an action plan.11 ,31

Currently, trainers also fulfil the role of assessors for PBAs. Many trainers have very little or no training in assessment and, although training might help trainers to make better judgements in assessment, there remains the issue of the trainer being both a teacher and assessor, with the potential for conflict of interest. Being both teacher and assessor are components of an integrated educational programme, but it is preferable that teaching and summative assessments are not carried out by the same assessor.32 However, if PBAs were deployed in a purely formative way, the trainer who is responsible for a trainee's development could use PBAs and the feedback and action points from these to guide future training (ie, as assessment for learning).

It appeared that there was an element of ‘gaming’ of WBAs by some trainees who have their eye on securing high summative scores. An example of ‘gaming’ of PBAs as described by some trainees include: selecting ‘doves’ (less critical trainers/assessors) to complete their PBAs; and selecting cases where they have performed particularly well for assessment (and not completing a PBA for cases where their performance was, in their view, less good).

In 2009, Munsch,33 the Chairman of the UK JCST, stated that WBAs are “feedback tools to guide and direct training, rather than summative competency-based assessments.”27 To clarify the formative role of WBAs as assessments for learning, it has even been suggested that WBAs be renamed to workplace-based assessment for learning.34

Trainer dependent

Eight of the 10 participants commented on obstruction from seniors, or a lack of support or engagement from trainers in the PBA process. For example, when questioned on the negatives of PBAs, one interviewee responded:The main obstacle or issue I find is from higher levels and not actually from me getting them achieved…finding consultant input to sit down and go through them with to sign them off was a difficulty…it's so difficult to get them…CT2 orthopaedic surgery

The critical role of faculty in WBAs has been highlighted in the literature, as has the need for strategies to enhance their participation and training.11 Furthermore, since the quality of training is dependent on the effectiveness of the surgeon–teachers in their roles as educators and assessors, as well as proficient professional (surgeon) service providers, the evaluation of these qualities in the surgeon–teacher forms an important component of the WBA.35

There has been a move away from the presumption that all surgeons are good trainers and that having a trainee as a surgical consultant is a right. One of the implications of this study is to highlight that additional senior support is necessary for trainees to facilitate the use of WBAs.

Strengths and benefits of PBAs

Undoubtedly, PBAs have strengths and benefits; the interviews highlighted the huge educational potential of PBAs. A positive aspect, commented on by more than half the trainees, was the value of PBAs in documenting progression:I have had consultants who have said we are doing that operation on Thursday. I want you to go and prepare for it. It's an operation that you haven't seen much and I've gone away and done my homework and got some of the best training I could because both of us had shown up that day. You've done your homework about the operation, what it involves, all the steps and they are there ready to train you because they have set that aside beforehand.ST4 ear, nose and throat (ENT) surgery

This approach could certainly be captured and encouraged if the formative role of PBAs were emphasised. Also, if the intent to complete a PBA was declared well in advance, for example, when the cases for a particular operating list are determined, this would improve the alignment of trainee development need with training opportunities. This, however, should not preclude the use of PBAs opportunistically, at short notice, if it is felt that a particular case, for example, an emergency procedure, presented an opportunity for feedback, reflection and generation of an action plan for learning.

Another benefit of PBAs identified by this study is the effect they can have on standardising the assessors, as they are required to complete a pro forma assessing a range of aspects of the trainee's performance:So it can almost bring different assessors to the same level. So if you have got an assessor who doesn't look at the whole picture, it can make them look at the whole picture because it looks at everything.CT2 orthopaedic surgery

The majority of trainees also identified that PBAs themselves can generate both training and reflection. A trainee may use a PBA to prompt a senior to instigate a training session. The majority of surgical training is ‘on-the-job’ and spontaneous. PBAs provide a method for formalising this learning and providing a record that it occurred. Further, when completed thoroughly, encouraging development over time with repeat assessments, PBAs may provide a very valuable tool. Pelgrim et al36 suggest a stepwise process to support this and obtain maximum effectiveness.

Limitations and areas for improvement

It was highlighted that, in the experience of our interviewees, PBAs are not being used as they were intended or designed. Interviewee 6 described having had virtually no experience of WBA being used as intended, and that the forms were predominantly filled in retrospectively by the trainee:It's a really difficult question. The only reason it is difficult…is because I have had little experience of them being used properly.ST4 general (vascular) surgery

Such comments are only a limited reflection on the PBAs themselves and somewhat reflect the system that they are being used in and those individuals who use them (both trainees and trainers). Indeed, a lack of awareness of the educational rationale for PBAs and, therefore, poor implementation seems to be at the heart of the challenges described.

Proponents of WBAs state that: “absence of evidence of benefit does not mean that WBAs, when undertaken correctly, do not work.”37 The trainees describe efforts to improve the way in which PBAs are used and to reverse a number of ‘bad habits’ that crept into the system when it was initially launched:They are having to undo a lot of damage that was done when workplace based assessments generally were introduced without really any particular training or fanfare. They were just dropped on trainees and trainers and I think unfortunately that has fostered a lot of resentment on both sides that is still being felt many years down the line.ST5 general (colorectal) surgery

In addition to having a potentially negative impact on individual trainees, it is pertinent to consider the impact of the introduction, use and perception of these tools on the training system as a whole:My general impression is that perhaps not everybody or every trainer is using them in the manner to which they were intended and so I think they may be actually giving some sort of a slightly false impression of standardisation in training.ST5 general (colorectal) surgery

As this interviewee suggested, rather than raising standards in postgraduate surgical training, PBAs could actually be blinding problems.

Suggestions for improvement: the trainee perspective

Suggestions for improving PBAs related to how they were used, as opposed to the design of the PBAs themselves. Recurring themes included improving the quality of the feedback and changing the ethos of the process, so that it is more educational and less stressful as detailed below.

The use of PBAs

Interestingly, some of the interviewees' proposed solutions related to PBAs to be used in the way they were intended and included providing more evidence that a formative trainee–trainer interaction had occurred, eliminating the tick box, blind sign-off:If the structuralise thing helps then you can use that as your form but to upload a recorded conversation between you and your consultant about what you learned because then there is clear evidence of that discussion, that training having taken place from both your angle and the consultants. You can't do it retrospectively particularly easily. You can't just be signed off. It would need to be something that you sat down and did.ST4 ENT surgery

Here, the trainee spoke about a genuine dialogue between the trainee and trainer. Nicol12 argues that feedback should be conceptualised as a dialogue between the giver and receiver. This feedback dialogue carried out before and after the PBA observation would significantly maximise the benefit of PBAs and other WBAs and represents one of the few examples of the combination of written and immediate verbal feedback in postgraduate medical education. This is supported by a recent systematic review highlighting that feedback given in verbal and written form appears to be more beneficial than either of these forms alone.38

Half of the participants commented that trainees often fill in their own PBA forms, and all of the participants highlighted the limited or complete lack of review of the trainees' and trainers' free-text boxes by subsequent assessors, for example, at the Annual Review of Competence Progression. However, if the PBA process is to have educational value, it is important for trainees to read their trainers' comments, use this to inform reflections on their performance and document this.

The majority of participants described PBAs as being stressful. PBAs were designed to support trainees' development and certainly not to cause them undue stress. Our findings indicate that it is the way that PBAs have been used and the pressure to accrue large numbers of them that is a cause of stress. This is compounded by the fact that they have been used for summative assessment, which is often inherently stressful.

Improving the quality of feedback: a self-assessment

Trainees suggested that using self-reflection to make the feedback more trainee-specific and using the PBA as a tool for prompting solicited and targeted trainer feedback could improve the relevancy and acceptability of the feedback the trainee receives, as the following quote describes.I think you should rate yourself and send that to the consultant and along the boxes he should tick either “agree” or “disagree” and then put comments. And I think that would be better feedback for your own strengths and weaknesses because an area that you think you have done really well, the consultant may actually think that you need to develop, you weren't slick or you weren't safe with your instruments or something.ST3 general (colorectal) surgery

Strengths and limitations

This study has a number of important strengths. A semistructured interview has many advantages over a survey approach, such as avoiding superficial responses and offering the flexibility to follow-up on and seek detail and clarification on key points and interesting comments raised by the trainees. There was a single interviewer (JS) for all of the semistructured interviews who has an understanding of the subject and the study. He was, therefore, able to pursue lines of questioning and explore areas of interest where the trainee showed a strength of feeling or opinion.

There are a number of limitations to this study. First, the study involved a single data collection method, namely transcribed and coded semistructured interviews. Second, there was selection bias with regard to the selection of the four PBAs that the trainee brought with them to the semistructured interview. This bias is somewhat deliberate and was designed to highlight extremes, both in terms of positive and negative assessment and feedback experiences, and to prompt reflection. This reflection was evident in the trainees' responses. There exists a selection bias in relation to the themes which were drawn from the transcripts, as well as those themes identified for further discussion; however, the systematic data analysis approach aimed to identify those themes that appeared most important to interviewees. A further potential source of bias is JS, a surgical trainee himself, as interviewer. Third, with regard to sample selection, the study makes no claims on the basis of representativeness or generalisability across trainees, but rather aimed to offer in-depth insight into trainees' views.

This is an evolving area of surgical training and further work is required to investigate the changing perspective of trainees over time. Future studies may also gain more insight into the value of PBAs at different stages of training and any associations between trainee understanding of purpose PBAs and their value.