Article Text
PDF
Abstract
Objectives Authentic patient and family engagement in child health research is defined as researchers working in partnership with patients and families on all aspects of the research process, including refining the research question, tailoring the intervention, devising study procedures and disseminating study findings. While there is good evidence of a positive impact of patient engagement on the research process, on research teams and on patient partners, there are few empirical data on the impact of patient and family engagement on research quality and dissemination. We conducted a systematic review to compare research quality and dissemination metrics for paediatric randomised controlled trials (RCTs) that engaged patients and families in the research process with trials that did not.
Design Systematic review using the Cochrane Highly Sensitive Search to identify RCTs.
Data sources Ovid MEDLINE from 1 January 2011 through to 31 December 2020.
Eligibility criteria We included RCTs involving children and youth (<18 years of age) published in The BMJ (a peer-reviewed general medical journal).
Data extraction and synthesis Trials were categorised as those engaging patients and families (PE+) and those that did not (PE−). A standardised review form was used to confirm trial eligibility and extract data on study characteristics. Two reviewers independently screened and sorted RCTs into PE+ and PE− groups, extracted data and assessed research quality using the modified Cochrane Risk of Bias Tool (based on seven methodological criteria). The dissemination of RCT findings was determined using measures of academic and non-academic citation collected from Web of Science and Scopus.
Results From 2011 to 2020, The BMJ published 45 RCTs involving children and youth. Only 10/45 RCTs (22%) reported engaging patients and families in the research process. Research quality for PE+ and PE− paediatric RCTs was similar; 4/10 (40%) of PE+ trials and 13/35 (37%) of PE− trials were rated as ‘fair’ or ‘good’ (p=1.00). Academic citation frequency per year was similar for PE+ trials and PE− trials: Web of Science (median 6.6 vs 7.1, respectively; p=0.84). Non-academic dissemination measures were generally higher among PE+ trials; for example, median PlumX Social Media score per year for PE+ trials was 46.6, compared with a median score of 7.6 for PE− trials (p=0.02).
Conclusions Despite increasing interest in patient and family engagement in child health research, this review showed that few paediatric RCTs report patient engagement activity. Research quality was similar for trials engaging patients and families compared with those that did not. Patient and family engagement in the trial, however, was associated with higher metrics for social media attention, compared with trials with no engagement.
- PAEDIATRICS
- Child
- Social Media
- Randomized Controlled Trial
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
We conducted a systematic review of paediatric randomised controlled trials (RCTs) published in The BMJ to compare research quality and dissemination metrics for trials that engaged patients and families in the research process with trials that did not.
We assessed research quality using the modified Cochrane Risk of Bias Tool for RCTs (based on seven methodological quality criteria) and dissemination metrics using measures of academic and non-academic citation collected from Web of Science and Scopus.
We only sampled paediatric RCTs published in one journal, but The BMJ was selected deliberately, given the requirement of submitting authors to report patient and public involvement in their research.
There may have been misclassification of RCTs that engaged patients and families (PE+ trials) as trials that did not (PE−), particularly for trials published prior to The BMJ policy in 2015 that mandated reporting of patient engagement.
Introduction
Patient-oriented research has been defined as a continuum of research that engages patients as partners, focuses on patient-identified priorities and outcomes and integrates the knowledge generated into policy and practice to improve healthcare outcomes.1 The term ‘patient’ is overarching and includes those with lived experience of a health issue, as well as informal caregivers, such as family and friends. Patient-oriented research is hypothesised to improve the quality, relevance and uptake of health research.2
Authentic patient engagement in health research involves a partnership between patients and researchers working together on any or all aspects of the research process, including choosing the research question, selecting the study design, tailoring the intervention, devising study procedures and dissemination of study findings. Barriers to patient engagement, however, such as parenting commitments, work schedules and long research timelines, can make patient and family engagement in child health research challenging.3
Aubin et al4 proposed a framework for measuring impact related to patient-oriented research. The framework identifies potential impact metrics across four domains: improvements to the research process, impact on policies, impact on health outcomes and contribution to social change. In the context of child health research, there is qualitative evidence of a positive impact of patient and family engagement on the research process (refinement of the research question, intervention and materials), on research teams (enhanced knowledge and cultural competency) and on patient partners (empowerment and skills development).5–10 There are, however, few quantitative data on the impact of patient-oriented research, and to our knowledge, the impact of patient and family engagement on research quality and dissemination has not been examined.
Therefore, the objective of this study was to conduct a systematic review of paediatric RCTs published in The BMJ (a peer-reviewed general medical journal with a high impact factor), examine measures of research quality and dissemination (academic and non-academic) and compare RCTs that reported engaging patients and families in the research process with RCTs that did not.
Methods
This study followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses reporting guidelines.11 We did not register a protocol prior to conducting the review.
Search strategy and information sources
An information science specialist (QM) developed an Ovid MEDLINE electronic search strategy to identify RCTs involving children and youth (<18 years of age) published in The BMJ over the 10-year period from 1 January 2011 to 31 December 2020. The search strategy was optimised for sensitivity and specificity using The Cochrane Highly Sensitive Search to identify RCTs.12 A search start date of 2011 was selected to align with national initiatives in patient and public involvement (PPI) in health research (Canada’s Strategy for Patient-Oriented Research and the Patient-Centred Outcomes Research Institute in the USA), as well as the 2015 The BMJ mandatory PPI reporting requirement. A search end date of 2020 was chosen, given the widespread impact of the COVID-19 pandemic on health research. Online supplemental table 1 describes the search strategy.
Supplemental material
Inclusion and exclusion criteria
RCTs published in The BMJ over the 10-year period that assessed a specific intervention were eligible. Only primary reports of trial results related to the trial’s primary outcome were included; that is, publications reporting a secondary analysis of RCT data were excluded. Trials were included if the study population was limited to children and youth from birth to less than 18 years of age. Excluded were clinical trial study protocols; non-randomised comparative trials; cross-sectional studies; non-comparative studies; systematic, scoping and narrative reviews; conference abstracts and editorials/commentaries.
Study selection and data extraction
A standardised review form was developed to confirm trial eligibility and extract data on study characteristics. Two reviewers (two of NH, KMN, TK and MB) independently performed an eligibility assessment for each article using the inclusion and exclusion criteria, first screening titles and abstracts and then full texts of potentially relevant articles. Any discrepancies were resolved through discussion and adjudication with a third reviewer (CMB).
For this review, patient and family engagement in research (a continuum of research that engages patients as partners, focuses on patient-identified priorities and aims to improve patient outcomes)1 was considered the ‘exposure of interest’ and research quality and dissemination were the ‘outcomes of interest’. Therefore, trials were categorised into two groups: the PE+ group (trials that reported engaging patients and families in the research process) and the PE− group (trials that did not report engaging patients and families in the research process). Recognising that information on patient and family engagement might not necessarily be reported in The BMJ publication, we also reviewed the information in published trial protocols (when available) and clinical trial registries such as ClinicalTrials.gov (when available) for all studies.
Two reviewers independently extracted data on RCT characteristics and methods, including author name, year of publication, trial setting, trial type, multicentre (yes/no), multinational (yes/no), participant’s age; primary outcome, sample size, number lost to follow-up and patient and/or family engagement (yes/no). For the PE+ trials, additional data were collected: number of patient/family/caregivers engaged, youth engagement (yes/no) and area of engagement in the research process.
Outcomes
‘Research quality’ was assessed using the modified Cochrane Risk of Bias Tool for RCTs.13 The Cochrane tool appraises seven methodological quality criteria: (1) random sequence generation (selection bias), (2) allocation concealment (selection bias), (3) selective reporting (reporting bias), (4) other bias, (5) blinding of participants and researchers (performance bias), (6) blinding of outcome assessment (detection bias) and (7) incomplete outcome data (attrition bias). Two reviewers independently evaluated the research quality of eligible trials, and any discrepancies were resolved through discussion with a third reviewer. An overall quality rating for each trial based on a summation of the seven criteria was also determined; trials were categorised as ‘good’, ‘fair’ or ‘poor’ based on the risk of bias tool guide.
Dissemination of trial results was determined using measures of academic and non-academic citation. Data on citation numbers and citation frequencies from Web of Science and Scopus were collected up to 14 February 2024. Web of Science has a narrower coverage of biomedical journals than Scopus and may therefore give a more conservative citation count.14 The primary academic dissemination metric captured was citation frequency per year (citation count divided by the number of years since publication).
PlumX data from Scopus (https://plumanalytics.com/learn/about-metrics/) and altmetric data from Web of Science (https://www.altmetric.com/research-access/) were used to determine non-academic citations.15 PlumX data capture interactions with research output in the online environment across five domains: citations (indexes, clinical or policy citations), usage (URL clicks, downloads and views), captures (bookmarks, favourites and follows); mentions (news media, blog posts and Wikipedia references) and social media (shares, likes and comments). The Altmetric Attention Score is a weighted count of the public attention a research article has received based on a variety of sources (citations, news media, social media mentions, blogs, etc). PlumX and Altmetric Attention Scores were described by year (total interactions divided by the number of years since publication of the trial).
Statistical analysis
Descriptive statistics were used to describe the frequency of paediatric RCTs published in The BMJ (by calendar year and patient engagement) and other RCT characteristics. Medians and IQRs were used to describe continuous variables that were non-normally distributed. PE+ and PE− trials were compared on research quality and dissemination metrics using Fisher’s exact test for categorical outcomes and the Mann-Whitney test for continuous variables. Statistical significance was defined as p<0.05; all statistical tests were two-sided.
Patient and public involvement
This systematic review was conducted in partnership with a patient partner (FB, co-author) who provided input on framing the research question, research methods, interpreting the research findings, writing the manuscript and preparing dissemination plans.
Results
The literature search strategy (see online supplemental table 1) identified 29 944 citations in The BMJ up to 2020, with 818 (2.7%) classified as potential RCTs. Of these, 132 (16%) were indexed as including children and were assessed for eligibility. After review, 87 records were excluded, leaving 45 paediatric trials included in the review (the search flowchart and reasons for exclusion are described in online supplemental figure 1).
Over the period 2011–2020, The BMJ published 45 RCTs involving children and youth.16–60 As shown in table 1, only 10 of the 45 RCTs (22%) reported engaging patients and/or families in the trial (additional information on the 10 PE+ trials is provided in online supplemental table 2). In alignment with the 2015 mandate from The BMJ in Instructions to Authors, all seven of the PE+ trials published from 2015 onwards included a PPI paragraph in the Methods section of the manuscript. Of the PE− paediatric trials, seven were published in 2015 or later; however, only three included a PPI paragraph; the four trials without a PPI paragraph were all published in 2015. Full implementation of the new reporting policy may have been slower than anticipated.
Number of paediatric RCTs published in The BMJ by patient engagement status and year of publication
As described in figure 1 and online supplemental table 2, among the 10 PE+ trials, studies described a spectrum of engagement, including engagement in the research preparation phase (9 (90%)), execution phase (8 (80%)) and translation phase (5 (50%)) and often across more than one aspect of the study. For example, eight PE+ trials had patient and/or family input on the development of the intervention,18 32 41 48 53 55 56 60 and five PE+ trials had input on the dissemination of trial results.41 48 55 58 59 Of the 10 PE+ RCTs, three trials included study participants 12–17 years of age specifically engaged youth partners32 48 59; the remaining seven trials engaged only parents or caregivers. Only one PE+ trial engaged families in developing the research question, and only this trial provided information on the numbers engaged; parents and caregivers of 40 children contributed to the refinement of the research question, selection of outcome measures and feedback on the intervention.53
Areas of patient and family engagement in paediatric trials that reported engaging patients and families in the research process (PE+) published in The BMJ.
The characteristics of PE+ RCTs (n=10) and PE− RCTs (n=35) are described in table 2. Across the 45 trials, the most common clinical categories were mental health (7 (16%)), endocrinology (4 (9%)), neonatology (4 (9%)), public health/preventative medicine (3 (7%)), infectious diseases and vaccines (3 (7%)), neurology (3 (7%)) and respirology (3 (7%)). Given the small sample size, formal statistical testing was not conducted. The descriptive data, however, suggest that PE+ RCTs were somewhat more likely to use a patient-reported outcome measure (PROM) as the primary outcome. Of the seven trials using a PROM, three trials used a parent proxy measure (two PE+ and one PE−) as participating children were as young as 1–3 years of age. Likewise, based on descriptive data, PE+ RCTs were also more likely to have more than one primary outcome, a larger sample size and a lower percentage of loss to follow-up, compared with PE− RCTs.
Characteristics of paediatric randomised controlled trials that reported engaging patients and families in the research process (PE+) and trials that did not (PE-) published in The BMJ, 2011-2020*
Research quality—assessed using the modified Cochrane Risk of Bias Tool—was similar for PE+ and PE− RCTs (see figure 2). Only 4/10 (40%) of PE+ trials and 13/35 (37%) of PE− trials were rated as ‘fair’ or ‘good’ (p=1.00). Additional data on the quality scores for PE+ and PE− trials is shown in online supplemental table 3.
Cochrane risk of bias scores for trials that reported engaging patients and families in the research process (PE+) and trials that did not (PE-).
Academic and non-academic measures of dissemination for the PE+ and PE− trials are described in table 3. With respect to academic measures of dissemination, citation frequency per year was similar for PE+ trials and PE− trials: Web of Science (median 6.6 vs 7.1, respectively) and Scopus (median 9.3 vs 9.5, respectively). Non-academic measures of dissemination, however, tended to be higher for PE+ trials, compared with PE− trials. For example, the median Altmetric Attention Score per year was 23 for PE+ trials compared with 5.4 for PE− trials. Likewise, the median PlumX Social Media score per year for PE+ trials was 46.6, compared with a median score of 7.6 for PE− trials. Last, while median PlumX Captures scores per year were higher for PE+ trials (41.5), compared with PE− trials (29), PE+ trials had a lower median PlumX Usage score per year (3.9), compared with PE− trials (41.9). For all dissemination metrics, the data showed a positively skewed (or right skewed) distribution with most values clustering at smaller values; there were large IQRs for non-academic measures of dissemination indicating greater variability.
Measures of academic and non-academic citation for paediatric randomised controlled trials that reported engaging patients and families in the research process (PE+) and trials that did not (PE-) published in The BMJ, 2011-2020.
Discussion
This systematic review collected and compared research quality and dissemination metrics for paediatric RCTs that engaged patients and families in the research process with trials that did not. Over a 10-year period, 10 of 45 RCTs published in The BMJ reported engaging patients and families in the research process. Descriptive data suggested that PE+ trials were somewhat more likely to use a PROM as the primary outcome and were more likely to have more than one primary outcome, a larger sample size and fewer participants lost to follow-up, compared with PE− trials. Research quality and academic dissemination metrics were similar for PE+ and PE− trials. PE+ trials, however, had higher social media scores compared with PE− trials, suggesting that patient and family engagement in a paediatric RCT may increase the likelihood of dissemination of trial research findings to the public via social media networks.
In total, 22% of paediatric trials in this review reported patient and family engagement. This compares favourably with data on patient engagement in adult RCTs. For example, a systematic review by Benizri et al61 that examined 50 RCTs published in 2021 in three leading medical journals (not including The BMJ) noted that only 5% of the RCTs reported patient engagement. A previous systematic review of RCTs published between 2011 and 2016 suggested that less than 1% of trials reported any patient engagement.62
Paediatric care is child- and family-centred; therefore, patient and family engagement in the research process may be more likely in this clinical milieu. Second, the increasing frequency of patient and family engagement may be related to the fact that several national research funding agencies mandate patient engagement in research proposal submissions. Last, an important factor, was the introduction of a new policy in 2015 by The BMJ that required authors to provide a PPI paragraph in the Methods section of submitted manuscripts.63 This requirement provides authors with a mandate and corollary word count to report patient and family engagement in their research. In addition, the policy may encourage researchers to engage patients and families in their research if The BMJ is the target journal.
It is unsurprising that PE+ paediatric trials in the review were somewhat more likely to use a PROM as the primary outcome, compared with PE− trials. In addition, the data suggested that PE+ trials had larger sample sizes and lower loss to follow-up, compared with PE− trials. Crocker et al64 have shown that PPI in clinical trials significantly improves participant enrolment and may improve retention.
PPI in health research has long been hypothesised to improve research quality and dissemination of findings; however, there are few empirical data on the topic.4 65 66 Barriers to precise measurement of engagement impact include inconsistent terminology for engagement, unpredictable reporting of engagement in the published literature and the difficulty in accurately measuring the direct impact of engagement, given the role of other factors such as context, policy and culture. This systematic review showed no material difference in the research quality of PE+ and PE− paediatric RCTs. Of note, relatively few trials in the sample were considered ‘fair’ or ‘good’ quality. The measure of research quality, however, was of the trial itself; we did not measure the ‘quality’ of patient engagement. In this context, none of the RCTs in the sample (neither PE+ nor PE−) used a reporting tool, such as the Guidance for Reporting Involvement of Patients and Public checklist, to describe PPI.67 This lack of standardised reporting of patient engagement limits the analysis of the impact of patient and family engagement in paediatric RCTs.
Measures of academic dissemination—based on citation frequency per year—also showed no difference between PE+ and PE− trials. Non-academic dissemination measures, however, were generally higher among PE+ paediatric trials, compared with PE− trials. PlumX Usage scores were higher for PE− RCTs compared with PE+ RCTs; however, usage scores (URL clicks, downloads and views) may reflect academic dissemination. PlumX Social Media scores (shares, likes and comments) more likely represent non-academic dissemination. To our knowledge, this is the first empirical study to examine the impact of patient and family engagement in paediatric RCTs on research quality and dissemination.
Strengths and limitations
Strengths of this study included a comprehensive and rigorous search strategy and eligibility review process. In addition, published trial protocols and clinical trial registries such as ClinicalTrials.gov were also searched for information on engagement for eligible trials.
There were limitations to our study. First, only one database (MEDLINE) and one journal (The BMJ) were searched, limiting generalisability. The BMJ was selected deliberately, given the mandatory reporting requirement of submitting authors to report PPI in their research. In addition, the a priori sampling strategy was to examine high-quality child health research, and The BMJ is a high-impact journal. Second, a 10-year sampling period with a 2011 start date was selected to align with national initiatives in PPI in health research (Canada’s Strategy for Patient-Oriented Research and the Patient-Centred Outcomes Research Institute in the USA), as well as the 2015 The BMJ mandatory PPI reporting requirement. The small sample size of paediatric trials, however, limited a formal assessment of the impact of these initiatives. Third, there may have been a misclassification of PE+ trials as PE− trials if PPI had not been reported by the manuscript authors. This issue is particularly relevant for those trials published prior to The BMJ policy in 2015 that mandated reporting of patient engagement. To mitigate this risk of misclassification, in addition to reviewing information reported in the published article, we also reviewed the information in published trial protocols (when available) and clinical trial registries (when available) for all studies when categorising trials as PE+ or PE−. Other authors have noted that patient and family engagement is underreported in the published literature.68 69 Given the a priori hypothesis and the finding of no difference in research quality between PE+ and PE− trials in our review, any bias because of misclassification, that is, PE+ trials misclassified as PE− would likely have been towards the null. Likewise, the academic citation frequency for any publication accumulates as the length of time from publication increases. Proportionately more PE− trials were published at the beginning of the study period, compared with PE+ trials. Therefore, given that academic citation frequencies per year were similar between PE+ and PE− trials, any bias would again have likely been towards the null.
Conclusion
Despite widespread recommendations for patient and family engagement in child health research, this systematic review showed that only one in five paediatric RCTs published in The BMJ over a 10-year period reported patient and family engagement. Research quality was similar for trials engaging patients and families compared with trials that did not. Patient and family engagement, however, was associated with higher non-academic dissemination metrics, in particular, social media attention, compared with trials with no patient and family engagement. The next steps include the development and application of standardised tools and methods to better measure the quantity, quality and impact of patient engagement in paediatric RCTs.
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
Ethics approval for the study was not required as no data were collected from human subjects and only peer-reviewed published studies in the public domain were reviewed.
Footnotes
Contributors CMB conceptualized and designed the study, designed the data collection instruments, coordinated and supervised the data extraction, performed the statistical analysis, interpreted the data, drafted the manuscript, critically reviewed and revised the manuscript. CM and FB conceptualized and designed the study, designed the data collection instruments, interpreted the data, drafted the manuscript, critically reviewed and revised the manuscript. NH performed the data extraction, performed the statistical analysis, interpreted the data, critically reviewed the manuscript for important intellectual content. KMN, TK, MB performed the data extraction, interpreted the data, critically reviewed the manuscript for important intellectual content. QM developed the search strategy, performed data curation, interpreted the data, critically reviewed the manuscript for important intellectual content. MT, EC, PCP interpreted the data, critically reviewed the manuscript for important intellectual content. All authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work. The corresponding author, as guarantor, accepts full responsibility for the finished article and attests that all listed authors meet the authorship criteria and that no others meeting the criteria have been omitted.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.