Article Text
PDF
Abstract
Introduction Rapid-cycle randomised testing holds high potential to enhance quality improvement practice but remains under-utilised because it requires significant resource commitment. However, infrastructure for learning networks, such as collaborative quality initiatives and large-scale quality improvement consortia, holds potential to support rapid-cycle testing at low cost and with minimal effort. For example, rapid-cycle randomised testing could be used to optimise ‘precision feedback’, which prioritises highly motivating tailored content to improve engagement with audit and feedback. We combined these concepts (rapid cycle, randomised testing and precision feedback) with a low-resource emphasis in conceiving this trial.
Methods and analysis A stepped wedge randomised controlled trial will deliver an intervention consisting of precision feedback and modifications to audit and feedback communication to 100 urologists performing ureteroscopy within the Michigan Urological Surgery Improvement Collaborative (MUSIC) and will be compared with a control consisting of standard ‘one-size-fits-all’ audit and feedback. The study’s primary endpoint is online dashboard engagement, measured as the clickthrough rate through the tracking of embedded links in emails. The stent rate following pre-stented ureteroscopy will also be measured. The primary hypothesis is that precision feedback will increase engagement with an audit and feedback dashboard and decrease rates of stenting following pre-stented ureteroscopy. Endpoints will be analysed by linear modelling accounting for repeated measures within individuals, exploring the primary hypothesis through a main effect by the study arm.
Ethics and dissemination Ethics and regulatory approval have been obtained from the Institutional Review Board of the University of Michigan (HUM#00248876). The findings will be disseminated in peer-reviewed journals and conferences.
Trial registration number ClinicalTrials.gov registration number: NCT06465667. Registered 6/20/2024. <https://clinicaltrials.gov/study/NCT06465667>
- Clinical Trial
- UROLOGY
- Implementation Science
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
A hybrid design helps to understand reasons why trial intervention does or does not affect patient outcomes.
A multi-institutional cohort across diverse practice settings improves generalisability.
A stepped wedge design with all providers eventually receiving intervention facilitates ‘uptake’ after the trial’s conclusion.
This study takes place within the existing Collaborative Quality Initiative, which may alter provider attitudes towards the intervention.
Background
Audit and feedback for healthcare performance improvement is an active area of interest. Audit and feedback is clearly feasible; however, its impact is highly dependent on context and limited by poor engagement.1 2 A body of literature has emerged around the concept of ‘precision feedback’, which prioritises feedback based on its likelihood of motivating change in performance.3 Ongoing work will evaluate the efficacy of and mechanisms behind the impact of precision feedback on care quality.4 Nonetheless, because of its potential to meaningfully impact healthcare delivery without spending on financial incentives or major infrastructural upgrades, improvements in audit and feedback processes may represent a priority in organisations with fewer discretionary resources.
Relatedly, rapid-cycle randomised testing holds high potential to enhance quality improvement practice but remains under-utilised because it requires significant resource commitment. For example, Horwitz et al describe their experience of conducting ‘rapid-cycle randomised testing’ that adds methodologic rigour to projects traditionally deemed ‘quality initiative (QI)’ but acknowledge the role of institutional financial, infrastructural and cultural commitment in launching these laudable efforts.5 Infrastructure for learning networks, such as collaborative QIs (CQIs) and large-scale quality improvement consortia, hold a potential to support rapid-cycle testing at low cost and with minimal effort. Furthermore, there exists a significant system-wide potential knowledge gain by making such projects quotidian rather than reliant on sporadic access to intra- or extra-mural funding.
Therefore, we combined the principle of rapid-cycle randomised testing with a desire to improve audit and feedback in envisioning the proposed randomised controlled trial (RCT), to take place within a statewide CQI – the Michigan Urological Surgery Improvement Collaborative (MUSIC). The trial involves a behaviour change intervention around the practice of ureteral stenting following uncomplicated pre-stented ureteroscopy for kidney stones. The placement of ureteral stents in such a setting has been found to contribute to extra emergency department visits and can be considered of low value.6 De-implementation of ureteral stenting following uncomplicated pre-stented ureteroscopy is an area of emphasis within MUSIC and was set as an internal quality metric. Accordingly, an interactive audit and feedback dashboard was conceived and piloted, and the results of the pilot directly informed the need for an RCT around behaviour change interventions within MUSIC.
Objectives
This RCT will evaluate the ability of modifications to the existing organisational audit and feedback programme to increase engagement with a quality process measure dashboard within MUSIC. The trial will deliver, through a stepped wedge design, precision feedback along with changes to email messaging and dashboard access to selected MUSIC urologists. The trial will, by extension, evaluate the efficacy of the dashboard in reducing unnecessary ureteral stenting and therefore comprises a hybrid implementation–effectiveness trial.7
The proposed trial addresses two aims. The first is to assess the effects of ‘precision feedback’ on quality measure dashboard engagement and care quality. Precision feedback prioritises information about performance based on its motivational potential for a particular recipient.3 Currently, MUSIC applies the conventional ‘one-size-fits-all’ feedback in the form of performance reports and dashboards. However, a breadth of literature along with qualitative data from MUSIC suggests heterogeneity of preferences for feedback content and delivery. The proposed trial will inform the ability of tailored motivational information to increase dashboard engagement and process measure performance (omission of ureteral stents following uncomplicated, pre-stented ureteroscopy). Dashboard engagement is prioritised as an endpoint, in alignment with Clinical Performance Feedback Intervention Theory (CP-FIT) and information value chain analysis.8 Feedback information must be received before it can be acted on; therefore, interaction with this information was emphasised as a primary endpoint, particularly given that clinical practice is subject to a myriad of preconditions and moderators. Our primary hypothesis is that precision feedback will increase dashboard engagement and deliver a modest impact on the process measure of interest.
The second aim is to iteratively design and adapt a quality measure dashboard and its dissemination to optimise its uptake. Although this trial is not explicitly ‘adaptive’, participants will provide their ‘feedback’ on the revised dashboard programme, which may inform a limited amount of redesign. We hope that the pre-trial pilot revealed the most obvious areas for improvement so that changes during the trial do not prohibit inference around the content and delivery of the behaviour change intervention. Nonetheless, this aim emphasises pragmatism and parallels real-world cyclical QI work. It also addresses modifications to feedback delivery that are not necessarily ‘precision feedback’, such as email design, distribution, timing and dashboard access.
Methods/design
Trial design
Subjects
The study will include urologists participating in MUSIC who were in the top 100 individuals in an annual case volume from 2022 for ‘pre-stented ureteroscopy’, the case for which the dashboard contents are most relevant. This number of participants was chosen based on the case volume, quarterly variability in individual case counts and distribution of total collaborative-wide volume across urologists. Urologists who are not in the top 100 individuals in the annual ‘pre-stented ureteroscopy’ case volume, urologists in the study team or those who opt out from the study at the individual or site level will be excluded (see ‘Consent’).
Patient and public involvement
Given that providers are the subjects of this study, it was decided to not be appropriate to directly involve patients or the public in the design, conduct, reporting or dissemination plans of our research. Patient advocates are a part of MUSIC and contribute to the collaborative’s overall strategic mission.
Trial structure
A stepped wedge RCT of a revised feedback programme will be conducted. Items reported herein and in a future results manuscript adhere to the Standard Protocol Items: Recommendations for Interventional Trials reporting guideline and Consolidated Standards of Reporting Trials extension to stepped wedge cluster randomised trials (figure 1).9 10 A stepped wedge design is appropriate in this setting for several reasons. The proposed means of precision feedback delivery are new to MUSIC, and logistical challenges are anticipated. A stepped wedge approach allows us to address issues as they arise (provided adjustments remain within the spirit of the study’s aims). Furthermore, as we have observed an underlying secular trend in the stent omission rate, a stepped wedge design will allow us to account for this while still delivering the intervention to the entire study population.
Standard Protocol Items: Recommendations for Interventional Trials schedule of enrolment, interventions and assessments.
One hundred urologists will be randomised via statistical software into four groups at the start of the study, stratified by case volume to ensure equal group allocation. The trial will consist of four steps; at the beginning, one group will cross over from the control to the intervention arm (figure 2). The first crossover will be preceded by a 3-month pre-rollout period, comprising a total study length of 15 months (five periods of 3 months each). The number of groups was chosen based on the step length and overall duration of the trial. The step length was chosen based on feedback from MUSIC urologists who viewed quarterly communication as appropriate. The duration of the trial was chosen based on a desire to deliver ‘rapid’ results to inform the scale-up of the intervention and to mirror the period over which a typical dashboard project may be conceived, designed and delivered. The case volume may predict dashboard engagement, stenting practices and privileged awareness of ongoing study efforts (ie, attention to announcements around kidney stone-related projects), and is therefore suitable for stratification.
Diagram of the stepped wedge trial design. Each group is comprised of 25 urologists, which cross over in the above sequence from control (dashboard, blue) to intervention (dashboard with precision feedback, orange), over five periods of 3 months each.
Intervention
The trial experimental intervention has both a content component and a delivery component. The behaviour change intervention content component is precision feedback. In this context, precision feedback is a tailored message that seeks to provide a single highly motivational piece of information for behaviour change. Precision feedback messages will be based on a previously described typology, which highlights comparisons, trends and level achievement/loss as motivating information.3 Further, general principles of optimising feedback interventions will be observed where feasible, such as consistency with established goals, timeliness, display of individualised data, linked visual display and summary message and credibility.11 Examples of precision feedback and control interventions are shown in figure 3. The content of precision feedback messages is subject to adaptation during the trial based on observations in the data.
Examples of control (left) and intervention (right) email correspondence, containing ‘one size fits all’ and precision feedback, respectively.
The behaviour change intervention delivery component will also involve modifications to the existing programme (see ‘Control’ below). Delivery changes are conceptually related to the idea of ‘precision feedback’, seeing as display preferences serve as factors modifying the motivational potential of a given message. Namely, performance feedback will be reported directly in the email as either plain text or an embedded or attached image with graphed data. The attached images will contain links directing recipients to the dashboard. In addition, individualised URLs will be monitored for clickthrough, and a follow-up email will be sent if a link is not clicked, recognising that some degree of failure to engage with data comes from missed communication.
Control
The trial control intervention consists of an email containing a link to the dashboard, along with a generic dashboard-related subject line and a short, written description of the dashboard. No performance data will be included in the email, and repeat emails will not be generated if links are unclicked. This message is consistent with typical MUSIC communication around audit and feedback programmes.
Both intervention and control emails will contain links for ‘opting out’, communicating with the study team about technical issues related to email materials, and for soliciting a recipient’s impression of the email (‘helpful’ or ‘not helpful’).
Endpoints
The primary endpoint of the study is the clickthrough rate to the online dashboard. The level of observation is a single link included in a single email. Clickthrough is defined as a link ever being clicked in the 3 months following the email send date, at which point in time a new email will be sent. Clickthrough is a binary variable with a value of ‘0’ if the link was not clicked within 3 months, and a value of ‘1’ if the link was clicked at any point within 3 months. This will be captured via the University of Michigan’s (UM) URL shortener service. The clickthrough rate will be calculated as the proportion of steps (ie, 3-month intervals) during which an embedded link was clicked.
The secondary endpoints will include the email ‘open’ rate, total number of dashboard visits, cumulative time spent on the dashboard, opt-out rate, ‘helpful/not helpful’ rates and stent rates for pre-stented ureteroscopy. These metrics will be attributed to an individual urologist and therefore intervention or control will be based on the date of the crossover. The email open rate will be captured via a UM-approved mail merge application. Dashboard visits and time spent will be captured by Google Analytics. Urologist-level opt-out will be collected via a separate link embedded in both control and experimental emails (see Consent, below).
Analysis
Endpoints will be analysed by linear modelling accounting for repeated measures within individuals. The primary hypothesis will be explored through a main effect by the study arm (precision feedback email vs control email and ‘intention to treat’). Models will also contain terms for site ‘size’ (ie, number of providers) and ‘type’ (academic vs community) and the urologist pre-study stent omission rate. The study team will remain blinded to aggregated results until the trial’s conclusion, with the exception of quality control at each time point to ensure endpoint capture. The study statistician will have access to the final dataset. A formal power calculation/effect size estimate will be performed prior to the start of the study but will not meaningfully change the study design. Final results will be disseminated via publication in a peer-reviewed journal.
Ethics and dissemination
The proposed trial was approved by the University of Michigan Medical School Institutional Review Board (HUM#00248876). Limited, de-identified data created during the current study will be available from the corresponding author on reasonable request. Findings will be disseminated in peer-reviewed journals and conferences.
Funding statement
The Blue Cross Blue Shield of Michigan provided support for the Michigan Urological Surgery Improvement Collaborative (MUSIC) as part of the BCBSM Value Partnerships Programme. PL is supported by the NCI (T32 CA180984). KS is supported by the NIDDK (K12 DK111011) and NIH NCI (P30 CA046592). ZLL is supported by NLM (R01 LM013894).
Consent
We plan to provide two ‘opt-out’ mechanisms from the trial; otherwise, we will waive written documentation of informed consent. Our rationale for avoiding documented consent is that asking urologists to ‘opt-in’ will create a subset of urologists who behave differently with respect to online dashboards, and results from their analysis will likely not generalise to the target audience (the broader MUSIC membership). Furthermore, this trial involves a very low-risk experimental intervention.
The identity of urologists participating in this study will remain blinded to other participants and the study team, except for the coordinating centre staff member assigned to this project. This person will be responsible for handling personally identifiable information necessary for delivering the appropriate email message.
Discussion
Our primary aspiration is that the proposed trial will contribute both a ‘local solution’ to increasing engagement, along with ‘generalisable knowledge’ around the motivating potential of precision feedback messages. Specifically, we hypothesise that precision feedback will increase dashboard engagement and deliver a modest impact on the process measure of interest.
This work builds on an emerging trend of breaking down silos around ‘QI’ versus ‘research’. Rapid-cycle randomised testing addresses projects that fall into the grey area between these two camps through the promotion of a more rigorous evaluation methodology around QI programmes. The intention is that increased rigour will both strengthen internal convictions around a programme’s efficacy and facilitate reporting to a broader audience. Dissemination of results in the literature ultimately increases the downstream impact of these efforts, bringing ‘local solutions’ into a space where they can be more readily assembled into ‘generalisable knowledge’. Furthermore, we hope to contribute to a growing literature of trials that emphasise embeddedness within an existing clinical or organisational context.12 Pragmatic clinical trials that are embedded in the electronic medical record draw an analogy to our work, where measurement and randomisation applied to otherwise routine CQI conduct contribute to knowledge.
With these considerations in mind, improvement in audit and feedback is a particularly suitable space for rapid-cycle randomised testing. An analogy to A/B testing in marketing and web design spaces is apparent; changing feedback content is akin to testing alternative headlines for online journalism.13 A large, cluster randomised trial, currently recruiting at the time of writing, will explore more broadly the impact of precision feedback-enhanced messages and provide valuable information regarding the effects of specific feedback types on behaviour change.4 The trial described herein emphasises engagement and clickthrough, supported by CP-FIT, which suggests that interaction with feedback precedes acceptance and intention to change behaviour.8 Nonetheless, linking increases in clickthrough to changes in target behaviour will be critical and should not be assumed. The stent rate following pre-stented ureteroscopy is therefore an important secondary endpoint in this study, and successes in increased engagement must be interpreted within the context of behaviour change. Relatedly, we hope to compare the results of our feedback intervention to others from diverse medical and surgical specialties, hypothesising that while our results may generalise to other surgical and procedural specialties, they could differ from interventions around prescribing or longitudinal patient management, where optimal feedback may take distinct forms.
Finally, our work seeks to emphasise the relatively low cost of evaluation programmes, which should encourage others to engage in similar work. Existing infrastructure, including a URL shortener service for clickthrough tracking and mail merge for distributing personalised emails in bulk, will facilitate the execution of this and future similar studies. Small-scale add-ons such as programming code scripts to constitute a precision feedback ‘knowledge base’ for ureteral stenting are low-cost and can be readily repurposed across other programs in our collaborative. This project was conceived and will be executed without significant dedicated extramural funding. We believe similar studies could be executed by organisations and CQIs with an interest in bolstering inference around the efficacy of QI projects.
Trial status
Opt-out opened on 6/24/24 and closed on 7/24/24. The study will run through 10/24/25. There is no formal recruitment/enrollment process; eligible providers were ‘enrolled’ with the ability to opt out before or during the study. The trial intervention has not been delivered to any participants at the time of writing.
Ethics statements
Patient consent for publication
Acknowledgments
Although the Blue Cross Blue Shield of Michigan and MUSIC work collaboratively, the opinions, beliefs and viewpoints expressed by the authors do not necessarily reflect the opinions, beliefs and viewpoints of BCBSM or any of its employees. The authors acknowledge the significant contributions of the clinical champions, urologists, administrators and data abstractors in each participating MUSIC practice (details can be found at www.musicurology.com), as well as members of the MUSIC Coordinating Center at the University of Michigan.
Footnotes
Contributors PL, AK, ZLL and KS conceived the study. PL, SDN, JR, KG, CD and KS contributed to the design the study. PL and BS created the software used in the study. All the authors have contributed to the manuscript and approved the final version. These findings have not been previously published and are not under consideration elsewhere. PL is the guarantor.
Funding The Blue Cross Blue Shield of Michigan provided support for the Michigan Urological Surgery Improvement Collaborative (MUSIC) as part of the BCBSM Value Partnerships Program. PL is supported by the NCI (T32 CA180984). KS is supported by the NIDDK (K12 DK111011) and NIH NCI (P30 CA046592). ZLL is supported by NLM (R01 LM013894).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.