Study design and procedures

The overarching objective is to direct future quality improvement activities to improve patient related outcomes, to advance clinical care and to improve communication between patients and healthcare professionals. The protocol adopts the Guidelines for Inclusion of Patient-Reported Outcomes in Clinical Trials Protocols30 (see online supplemental file).

A mixed-methods design shall be used, with three phases. To develop the final set of ‘AusPROM’ implementation recommendations, data from phases I, II and III will be combined in an iterative process with phase I extending alongside phases II and III. Data from phase I will influence phase II and III, and likewise, data from phase II and III will influence the latter stages of phase I (figure 1). Phase I will identify staff barriers and facilitators to nationwide implementation of an ePROM to elective surgery patients using the Delphi technique with health professionals and other hospital staff. During this phase, we shall also generate the AusPROM recommendations. Because phase I is an iterative process, it will allow the findings to be integrated periodically throughout phases II and III. Phase II will use a feasibility design31 to determine QoR-15 PROM acceptability from the perspective of elective surgery patients from four pilot hospitals from 29 healthscope hospitals, selected as a sample of convenience. Phase III is the national implementation (29 hospitals). Consumer feedback and co-design in embedded throughout the phases. This includes a consumer codesigning and coauthoring the project from its concept; patients completing brief acceptability surveys alongside the QoR-15 throughout phase II, in the week prior to surgery, in the week following surgery and 4 weeks postsurgery; as well as a consumer focus group at the end of phase II.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The overlapping phases of the study to develop the final set of ‘AusPROM’ implementation recommendations. AusPROM, Australian patient-reported outcome measures; PROMs, patient-reported outcome measures; QoR-15, Quality of Recovery 15 item short-form.

To provide structure to the implementation process, the research team will use the PROM-cycle framework.32 In addition, the national implementation will be shaped according to recommendations developed during the first two focus group iterations of phase I and the patient acceptability from phase II. Phase III will also examine the concurrent validity of the QoR-15 and generic EQ-5D-5L multi-attribute quality of life measure, with data collection at the four pilot hospitals.

The QoR-15 PROM is a 15-item short-form and it was based on the 40 item QoR-40.29 The QoR-15 has 15 items each rated on an 11-point scale from 0 to 10, with a maximum score of 150. It takes 2.4 min to complete and has reported good validity, reliability and responsiveness.4 29 There is evidence that the QoR-15 can be used from presurgery up to 7 days postsurgery, as a measure of change over time.33 34 The minimal clinical important difference of the QoR-15 is 8.0.35

Phase I: The primary outcome of phase I is the development of the set of national implementation recommendations (AusPROM recommendations), with the primary endpoint being conclusion of the national implementation following the conclusion of phase III. It is expected that staff and patient education will be developed and delivered based on these recommendations. Even though the AusPROM recommendations will initially be developed for the Australian context, a number of the recommendations will have international applicability. A key goal is to simplify administration, while acknowledging that compliance be assisted by hospital staff (eg, front desk staff, medical assistants, nurses, allied health professionals, medical practitioners, surgeons) encouraging patients to fill out the PROMs. Therefore, there will be two perspectives: (1) from staff implementing it centrally at corporate office and (2) staff in the hospitals who are encouraging patients to complete the ePROM as well as using findings from the ePROM survey. This will include health professionals as well as some non-clinical hospital staff from the front desk and administration teams. Several prior studies discuss the impost (cost/time) of data collection36–38 and our objective is to circumvent that by ensuring that system-wide processes are in place so that the tool can be implemented efficiently.

The Delphi technique can be used to examine complex problems through an iterative process guided by expert opinions, known as a group knowledge acquisition model.39 The Delphi technique in this study was aligned to the classical Delphi where the focus is on facts and the objective is the elicit opinion and gain consensus via a series of focus groups.39–42 The Delphi technique will involve nursing staff from each of the four pilot hospitals, as well as doctors who have involvement in the implementation. They will be asked to participate in each of the three iterative focus groups. Focus groups will occur prior to the commencement of phase I, as well as prior to, and at the conclusion of, phase III. The focus groups will be directed towards two issues of priority: (1) barriers and enablers for the national implementation of ePROMs and (2) recommendations for the implementation and integration of an ePROM into usual care.

Staff inclusion criteria include being aged 18+, employed at Healthscope hospitals and working at one of the included hospitals, and a registered nurse, doctor, allied health professional or administration staff member. There are no specific exclusion criteria. Written informed consent is required for participation.

An email will be sent from the site director of nursing to the potential staff participants across the four pilot hospitals, inviting the staff member to participate. They will be invited to contact the research team if they would like to participate in the study. Staff will be identified via the site director of nursing and the chief medical officer or general manager. It will be explained that participation includes three 1-hour focus groups spread out over a 10-month period. It is expected that there will be at least 10 staff participants in the Delphi study. Previous studies have shown that a Delphi study sample size ranging from 6 to 50 had minimal impact on 6 of 9 different consensus indices,43 indicating that the planned sample of size of up to 10 participants will be adequate for this Delphi study.

Phase II will use a feasibility design to complete survey pretesting at one pilot hospital, as well as determine the response rate and QoR-15 ePROM acceptability from an elective surgery patient perspective across four pilot hospitals. The pretesting (n=100) will investigate feasibility from a technical perspective (the rest of this phase relates to feasibility from the patient perspective). Technical feasibility testing includes the pulling of survey distribution list reports from hospital administration data, distributing the survey and testing the assumed patient email and/or mobile number capture rate for survey distribution. Patients will complete brief surveys across three time points, incorporating the QoR-15 and two acceptability questions, in the week prior to surgery (noting small QoR-15 modifications were required presurgery), in the week following surgery and 4 weeks postsurgery. Time to complete the survey is estimated at 5 min based on previous studies.29 The primary outcome of phase II is feasibility relating to the response rate and the primary endpoint will be 4 weeks postsurgery. The secondary outcome is the degree of patient ePROM survey acceptability. At the conclusion of phase II consumers will be invited to participate in a focus group to discuss in detail the patient acceptability of the PROM survey as well as recommendations for implementation. It is acknowledged that optimal time points for PROM data collection can sometimes vary according to the patients’ condition. For example, elective knee replacement patients often do not confer their full benefit until many months after surgery whereas elective hernia repairs recover within weeks. The extra complexity involved with tailoring time points to different surgeries was beyond the scope of the current study, hence we standardised the time points for PROMs data collection for elective surgeries. The optimal time points for data collection will be further investigated through the consumer and staff feedback on acceptability.

Quantitative data include the survey response rate and completion rate for patients who receive an invitation to participate, as well as acceptability of the ePROM survey on a 0–10 Likert scale (10=highly acceptable and 0=not acceptable). In addition, response scores for the QoR-15 will be reported over the three time points as a change score and as a percentage of participants who return to presurgical status at 4-week postsurgery.

Qualitative data include patient responses from an open-ended question regarding ePROM survey acceptability as well as the consumer focus group. Responses will be themed via a content analysis using the theoretical framework of acceptability (TFA).44 The TFA includes aspects of patient attitude, burden (including length of survey and the timing of the three surveys), ethicality, understanding of the intervention, opportunity costs, perceived effectiveness and self-efficacy for survey completion.44 There will also be a content analysis where the frequency of themes is reported for each of the TFA domains.

Patients aged 18+ will be recruited via email and/or text messages following hospital preadmission for elective surgery at one of the included hospitals. It is noted that, in Australia, email and text are appropriate strategies for PROM data collection as 86% of households have internet access,45 91% with household internet use mobile or smart phones45 and 94% of people who use the internet do so to access emails.46 The current patient email capture rate is around 80% for the health service and patients will be excluded if they do not provide either a valid email address or mobile phone number. Patients will also be excluded if they do not have adequate English (survey is only presented in English), if they tick the ‘opt out’ box on the hospital admission paperwork for participation in patient surveys, if they are pregnant, or if they are undergoing a hip, knee or shoulder replacement and in the case of death no further surveys will be sent. Patients undergoing a hip, knee or shoulder replacement are excluded due to a parallel project in place at the health service targeting this patient population through another PROM process. The survey invitation will include a link to the participant information sheet and there will be a tick box for consent to participate at the start of the survey. Data will be deidentified and presented in an aggregate format. For incomplete surveys, a reminder email and text will be sent up to 1 week later, to improve adherence rates. We shall include the data from all patients, whether they complete one, two or the complete set of three surveys.

For phase II, four hospitals have been recruited to participate in data collection. To be representative of the national health service involved in the study, the hospitals will have a mix of day and overnight services. They will include small and large hospitals, and will be located across three states of Australia. They were selected as samples of convenience of facilities with more than 200 beds across multiple states in Australia and staff willing to participate. It is estimated that over a 3-month period around 2000 patients will receive the ePROM survey. As current patient survey response rates are around 40% for the health service, it is estimated that around 800 patients will complete the presurgery survey over 3 months of data collection, with only 500 patients completing all three surveys due to the 5-week time horizon between surveys combined with the 3-month data collection period. As phase II is a feasibility study, a formal power calculation for the sample size has not been undertaken.47 Instead, the sample size was based on numbers needed to adequately determine the response rate at 4 weeks postsurgery (primary outcome).

Phase III focuses on the national ePROM implementation (29 hospitals), informed by the early phase I Delphi study informing the AusPROM recommendations, and the concurrent validity analysis of the QoR-15 and generic EQ-5D-5L multiattribute quality of life measure (four hospitals). The primary outcome for phase III is the national survey response rate (29 hospitals), with success achieved if the response rate for the pilot sites (4 hospitals) is equalled or exceeded. Patient recruitment and inclusion/exclusion criteria is the same as phase II.

As the objective of this study is to successfully integrate an ePROM across a national health service to direct future quality improvement activity and ultimately advance clinical care and patient–clinician communication, a whole of health service approach is required. The implementation phase, therefore, has a sample size based on national hospital representation and it is estimated that over a 3-month period around 15 000 patients will receive the ePROM survey.

Data analysis plan

Phase I: To report the staff barriers and enablers for implementation of an ePROM, the results of the three Delphi focus groups will be themed according to the National Institute of Clinical Studies barriers and enablers framework.48 This framework includes six levels of potential barriers and enablers including the innovation itself (integrating the ePROM survey into usual care), the professionals/staff, the patient, the social context, the organisation context and the economic and political context. To report the recommendations for the integration of an ePROM into usual care, consensus statements will be drafted in the initial focus group, and redrafted and refined in the subsequent focus groups.

Phases II and III: Survey response rates and completion rates will be reported as a number and percentage of the total. Response scores for the QoR-15 will be reported over the three time points as a change score and as a percentage of participants who return to presurgical status at 4 weeks postsurgery. This will include (1) a comparison between all surveys at baseline, within 1-week postsurgery and at 4 weeks postsurgery; and (2) only include patients who have completed all three surveys (captured through a unique survey identified which will link multiple surveys completed by the same patient). Missing data shall be in reference to a patient missing one or more of the three surveys. There will be no imputation of missing data. We shall also perform an analysis whereby we stratify the PROM results for different hospitals, different surgical groups and according to age. This will enable us to compare our results with global reports on surgical PROM outcomes for different groups.

Phase II: Acceptability of the ePROM survey on a 0–10 Likert Scale will be presented as a mean with IQRs. Responses from the open-ended survey question and the consumer focus group, regarding ePROM survey acceptability, will be themed via a content analysis using the TFA.44 There will also be a content analysis where the frequency of themes is reported for each of the TFA domains.

Phase III shall establish if the condition-specific QoR-15 PROM has concurrent validity with the generic EQ-5D-5L multiattribute quality of life measure, and data from the four pilot sites during the phase II patient ePROM survey. We will assess the concurrent validity between the tests, using Spearman’s correlation coefficients, as the data are not expected to be normally distributed. A correlation coefficient of less than 0.3 will be considered weak, between 0.3 and 0.5 will be considered moderate, and above 0.5 will be considered strong. It is noted that this analysis of the additional quality of life questions are pending on the acceptability of the phase II ePROM survey (which did not include quality of life). Missing data will be managed by excluding participants case wise. Statistical significance is defined as p<0.05 and analyses will be completed on IBM SPSS V.25.49 Of note, the minimally clinically important difference for the QoR-15 PROM has already been established by Myles et al as 4.6–8.0.35 The manuscript by Myles et al also shows the value of the ‘patient acceptable symptom state’ (PASS).35 For the QoR-15, it is a score or 118 or better. PASS defines what minimal threshold (score) patients would accept for their own recovery.

Patient and public involvement

We designed this protocol ensuring patient involvement in the choice of PROM, study design, data collection forms and implementation plan. Consumers (patients, health professionals, healthcare managers) will be involved in all parts of the project dissemination of study findings. Consumer representatives contributed to this document.

Development of the AusPROM recommendations will provide a novel contribution to the literature, locally and globally. Of note, the AusPROM is not another new PROM. Rather it is a set of recommendations for implementation of PROMS in hospital settings. It is anticipated that the findings will highlight the value of patient (acceptability domains) and health professional (Delphi technique) codesign to inform the implementation recommendations for patient focused outcome measures. The results of this PROM study will also illuminate the feasibility and value of using the QoR-15 to understand how patients rate their elective surgery outcomes. In addition, the findings have the potential to benefit elective surgery patients, clinicians, hospitals, researchers and policy-makers.

Once embedded into usual care, data from this e-PROM could help to improve patient experiences and outcomes for elective surgery. Information gained during the barriers and enablers phase of the study shall inform the development of e-PROM-related educational materials for patients and clinicians. The education material shall aim to ensure that patients are better prepared for postdischarge management of their condition and better able to cope with the recovery process. Potential health service benefits could include benchmarking different hospitals to see if e-PROM results are higher or lower at a particular site, or for specific surgical procedures or disciplines, allowing strategies to respond to positive or negative deviance. For policy-makers, this study has the potential to provide input into economic funding directions, as funding moves towards paying for outcomes, rather than only paying for activity.

The results will be compared and contrasted with previous nation-wide PROM implementation projects. This will be important given the challenges encountered during the implementation of some measures, such as the UK NHS PROM50 and some orthopaedic-related PROMs.51 The current project will be different and arguably more effective due to strong consumer engagement at all stages of design and implementation, as well as drawing on the learnings of hundreds of surgical outcome studies of the QoR-15 from across the globe,4 52 including large randomised trials.53–55