Article Text
PDF
Abstract
Objectives To evaluate the extent and quality of patient involvement reporting in examples of current practice in health research.
Design Mixed-methods study. We used a targeted search strategy across three cohorts to identify health research publications that reported patient involvement: original research articles published in 2019 in the British Medical Journal (BMJ), articles listed in the Patient-Centered Outcomes Research Institute (PCORI) database (2019), and articles citing the GRIPP2 (Guidance for Reporting Involvement of Patients and Public) reporting checklist for patient involvement or a critical appraisal guideline for user involvement. Publications were coded according to three coding schemes: ‘phase of involvement’, the GRIPP2-Short Form (GRIPP2-SF) reporting checklist and the critical appraisal guideline.
Outcome measures The phase of the study in which patients were actively involved. For the BMJ sample, the proportion of publications that reported patient involvement. The quality of reporting based on the GRIPP2-SF reporting guideline. The quality of patient involvement based on the critical appraisal guideline. Quantitative and qualitative results are reported.
Results We included 86 publications that reported patient involvement. Patients were most frequently involved in study design (90% of publications, n=77), followed by study conduct (71%, n=61) and dissemination (42%, n=36). Reporting of patient involvement was often incomplete, for example, only 40% of publications (n=34) reported the aim of patient involvement. While the methods (57%, n=49) and results (59%, n=51) of involvement were reported more frequently, reporting was often unspecific and the influence of patients’ input remained vague. Therefore, a systematic assessment of the quality and impact of patient involvement according to the critical appraisal guideline was not feasible across samples.
Conclusions As patient involvement is increasingly seen as an integral part of the research process and requested by funding bodies, it is essential that researchers receive specific guidance on how to report patient involvement activities. Complete reporting builds the foundation for assessing the quality of patient involvement and its impact on research.
- QUALITATIVE RESEARCH
- STATISTICS & RESEARCH METHODS
- MEDICAL ETHICS
Data availability statement
Data are available in a public, open access repository. The data generated in this study are openly available under the Creative Commons Attribution 4.0 International license in Zenodo (https://doi.org/10.5281/zenodo.6245750).
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
A targeted search strategy was used to identify examples of patient involvement reporting in a variety of publication types and study designs in health research.
A mixed-methods approach allowed for an analysis of both the completeness and quality of patient involvement reporting.
In this study, we coded statements reporting on patient involvement in 86 health research publications that may be adapted for further use.
Qualitative studies were excluded from the analysis: in many cases, it was difficult to distinguish between patients’ contributions as research partners or as subjects of the qualitative research.
Reporting of patient involvement was insufficiently detailed to allow for a systematic assessment of the quality of patient involvement.
Introduction
Patients’ viewpoints should be included in clinical research as they are the most affected by it.1 Different approaches can be used to make the outcomes of clinical research more relevant to patients. One option is to actively involve patients or patient representatives in study design, study conduct and dissemination. Different terms are used to describe this active involvement, for example, ‘patient and public involvement’ (PPI) or ‘patient engagement’. Patient involvement in health research varies widely and can be categorised, for example, according to the level or continuity of involvement, involvement in different phases of the research or the methods applied for involvement.1 2 Standards and principles for patient involvement focus mainly on the management of the relationship between patients and researchers.3 These principles are important for a good collaboration, but only reflect one quality aspect of patient involvement or its impact on outcomes of clinical research. Given the efforts from patients, there is also an ethical imperative to reflect about their input in the publication, and the results and the impact of patient involvement should be evaluated and published.
Quality in clinical research is assessed with critical appraisal tools, such as the widely used risk of bias tool for randomised controlled trials (RCTs).4 5 A critical appraisal tool to assess the quality of patient involvement was developed in 2010.6 High-quality reporting is needed to allow for critical appraisal and quality assessment. A reporting guideline GRIPP (Guidance for Reporting Involvement of Patients and Public) was developed in 20117 and updated in 2017 to GRIPP2.8 9 GRIPP2 comes in two different formats: a long form (LF) for studies with patient involvement as primary focus and a short form (SF) for studies with patient involvement as secondary or tertiary focus, such as, for example, clinical studies being informed by an active involvement of patients.
In 2014, the British Medical Journal (BMJ) group endorsed a policy which made it a requirement to report on PPI in BMJ journals and recommends GRIPP2 as a reporting standard.10 Price et al11 compared reporting of PPI before and after the introduction of the BMJ policy. They found that while 86% of research articles included a PPI statement about 1 year after the introduction of the policy, only 11% actually reported PPI activities. Funding organisations are also likely to play an important role in improving the quality and reporting of patient involvement, especially as they increasingly require patient involvement in clinical research.12
The objective of this study was to analyse the extent and quality of patient involvement reporting and the quality of patient involvement in examples of current practice in health research. We are aware of several studies that investigated the rate and quality of reporting13 or critically appraised patient involvement in specific domains of clinical research.12 14–16 Overall, these previous assessments identified very few publications that reported patient involvement and/or engagement and reporting quality was suboptimal (eg, 0.4% of the sample described the active involvement of patients in orthopaedic research16). Jones et al15 also included studies which had patient involvement as primary focus, such as prioritising research topics. Our scope was different: we focused on studies that actively involved patients to inform the study methodology (including dissemination) but did not have patient involvement as primary focus. We did not limit our analysis to a specific research area or experimental design (eg, RCTs), but included three purposively selected cohorts of publications in which we expected reporting of patient involvement. We considered patients as people affected by the disease or topic, their family members or representatives of those affected.
Methods
A protocol detailing the methods of this study was pre-registered in the Open Science Framework (28 April 2020) and is openly available17 (see also online supplemental material 1).
Supplemental material
Samples
In order to identify publications that report on patient involvement, we used a targeted search strategy in the following three samples, restricted to English-language publications:
Publications in the journal the BMJ, which requires reporting on patient involvement in research articles. We performed a Web of Science search (Web of Science Core Collection, 4 March 2020) to obtain all publications published in 2019 in the BMJ (document types: ‘Article’ or ‘Review’).
Publications listed in the Patient-Centered Outcomes Research Institute (PCORI) database.18 PCORI is a US-based organisation funding patient-centred research, which continuously screens Medline via PubMed, relevant journals and PCORI staff recommendations for publications on patient engagement in health research. We filtered for topic: example of engagement in health research; stakeholder involvement: patients; year: 2019.
Publications indexed in Dimensions19 citing one of the two GRIPP2 publications8 9 or the critical appraisal publication.6
Details of the search strategies can be found in online supplemental material 2. All included publications across these samples were checked for additional links or references, which described patient involvement in more detail. If additional relevant documents were found (eg, online supplemental materials, previously published protocols), they were included in the sample.
Inclusion and exclusion criteria
The inclusion and exclusion criteria outlined in the protocol were refined during the screening process to accommodate for the wide variety of studies included in the sample. As a result, the following inclusion and exclusion criteria were applied across all samples (see flow chart in online supplemental figure 1).
Study type
Quantitative studies (RCTs, observational studies, etc) were included; qualitative research studies were excluded. We excluded qualitative research publications, as in many cases it was difficult to distinguish between the active involvement of patients in the study and their involvement as subjects of the qualitative research.
Systematic reviews and scoping reviews were included; narrative reviews were excluded.
Mixed-method studies and those which used qualitative and quantitative methods were included if the methods were mainly quantitative.
Protocols were excluded. If an original publication in our sample cited a protocol which provided more detailed information on patient involvement, this protocol was included as an additional document.
Studies in which a tool was developed and tested were included. Studies in which tools/interventions/outcomes were developed but not applied were excluded.
Comments, editorials, guidelines, consensus papers and other publications, which did not aim to answer a research question, were excluded.
Patient involvement
Publications were included if a patient involvement activity was described for at least one phase of the study in the main text (only in the acknowledgements was not considered as sufficient), that is, patients or patient representatives were actively involved in designing or running the study, were engaged as co-researchers, supported the dissemination of results or had an advisory function; participating in a study as a ‘subject’ or ‘participant’ was not considered as sufficient to qualify as patient involvement.
Publications were included if patient involvement was used to inform the study, but it was not the primary focus of the study. The authors’ decision whether to complete the SF or LF of the GRIPP2 reporting checklist was used as an indicator of the focus of the study (if applicable).
Publications were included if patient involvement activities had already been conducted (not only planned). The only exception was for patient involvement in dissemination activities. Given that dissemination activities often take place after a study is published, studies with planned patient involvement in dissemination activities were included.
All identified studies were screened by at least two members of the team. Discrepancies were discussed in the team until a consensus was reached.
Coding
We coded all publications and additional documents using three coding schemes (online supplemental tables 1–3):
Phase of involvement: included publications had to report patient involvement in at least one of the three study phases: study design (subcodes ‘research question’ and ‘outcome measures’), study conduct or dissemination (subcode ‘coauthoring the manuscript’).
GRIPP2-SF8 to assess the reporting of patient involvement.
Critical appraisal tool6 to assess the quality of patient involvement.
We used an inclusive and pragmatic approach in the coding. For example, we accepted a statement such as ‘patients were included to inform the study design’ as sufficient to describe the aim of patient involvement according to GRIPP2. A mere description of tasks was considered as sufficient to code ‘have the researchers discussed the nature of tasks’ according to the critical appraisal tool. We also coded acknowledgement and contribution statements if these mentioned phases or activities of patient involvement. With the aim of assessing the quality and impact of patient involvement based on included publications, we coded statements that addressed criteria in the critical appraisal tool. However, the varying amount of detail reported across publications and samples did not allow for a systematic appraisal of the quality of patient involvement.
Two raters coded all included publications according to the coding schemes. Discrepancies were discussed until a consensus was reached. If this was not possible, a third person assessed the respective passage and the team decided by majority vote.
Analysis
Coded segments were exported from MAXQDA20 and analysed further in Microsoft Excel. Codes from additional documents were merged with that of the original study. We quantified how many publications in each sample reported one of the codes at least once. For the BMJ sample, we additionally report the frequency of patient involvement across all research articles that were published in 2019, given the journal’s requirement to report whether PPI has taken place.10 Of the n=200 search results for the BMJ sample, n=155 were articles that reported (original) research (see online supplemental figure 1). These had to be identified to compare our results with Price et al.11
Additionally, we conducted a qualitative content analysis based on the extracted GRIPP2 codes. Similar codes within the same GRIPP2 category were grouped into overarching themes. All coded segments were also reviewed for illustrative examples.
Patient and public involvement
Patients or the public were not involved in the planning or conduct of this meta-research study. The analyses were not restricted to studies on specific diseases or patient populations; therefore, it would not have been adequate to include a specific patient group since this research is not specifically relevant for them. The main target audience includes researchers and other stakeholders in health research (eg, journal editors, funders). The results have been discussed in workshops with health researchers and patients and/or patient representatives and other stakeholder (eg, funders) to raise awareness of this topic and to describe the progress of integrating patient involvement in health research.
Results
Inclusion and exclusion
A total of 86 research publications were included in the analysis after applying our inclusion and exclusion criteria (see online supplemental figure 1). From the BMJ sample, 32 of 155 research articles (21%) were included because they reported PPI activities and qualified as quantitative study. We included a further 41 publications from the PCORI sample and 13 from the Citation sample (12 citing GRIPP2 and 1 citing the critical appraisal tool). Most frequently applied exclusion criteria were ‘no patient involvement’ and ‘no research publication’. We included 35 additional documents, which provided further information on patient involvement described in the publications.21
Phase of involvement
Patients were most frequently involved in the study design (90% of included publications, n=77), followed by study conduct (71%, n=61) and dissemination (42%, n=36) (table 1 and 21). In 17% (n=15) of the publications, patients were involved in formulating the research question and in 31% (n=27) in defining outcome measures.
Phase of involvement
GRIPP2 (short form)
Between 13% (n=4, BMJ sample) and 77% (n=10, Citation sample) of the publications reported the aim of patient involvement (table 2 and 21). The predominant code for aim identified in our sample was ensuring that patients’ perspectives were taken into account. Coded segments ranged from vague statements (table 2, Example (E) 1) to more elaborated accounts (table 2, E2). Other examples included support with recruitment or the dissemination of the results and ensuring the accessibility or acceptability of the study.
Reporting of patient involvement according to GRIPP2-SF: quantitative and qualitative results
More than half of the publications (57%, n=49) provided some information about the methods used for patient involvement in the study. However, these accounts were often not very detailed. The most predominant code was consultation or giving feedback, indicated by describing the group involved: patient representative, advisory group, patient group, adviser to the steering committee, patient engagement group or lay representative. In many cases, even very basic information such as the number of involved patients, the frequency of meetings or explanations on how discussions took place was lacking (table 2, E3). Some publications reported on approaches they used for the consultation, such as working on an online platform or group meetings. Others used additional methods such as focus and discussion groups or interviews with patients to get further input on specific questions. A detailed, informative example was the development of a ‘roadmap’ prior to the study (table 2, E4), which served to identify how stakeholders could influence the study.
Of all publications, 59% (n=51) reported the results of patient involvement in the study. Examples of the reported outcomes included making materials easily understandable, supporting with or sharing ideas on recruitment, raising awareness about the study and identifying patient-centred outcomes. The level of reporting varied from broad statements with only few or no concrete examples (table 2, E5) to more detailed information on the outcomes of the patient involvement and its influence on the study (table 2, E6; see also Minneci et al22 for detailed information in a supplement).
Of all publications, 42% (n=36) provided information on the influence of patient involvement on the study (‘discussion and conclusions’). Examples include its influence on the intervention, recruitment, retention, usability of study findings (table 2, E7) and outcomes (table 2, E8).
A relatively small number of publications (22%, n=19) reported reflections and critical perspectives on patient involvement. Some reflections related to the research context and how its structure and settings may not always be welcoming for patient involvement (table 2, E9). Others discussed a possible lack of representativeness or diversity in the sample of PPI contributors (table 2, E10).
Critical appraisal tool
The critical appraisal tool6 focuses on the quality and impact of user involvement in research (table 3 and 21). Specific appraisal criteria were rarely reported, such as discussing the level of involvement (7%, n=6), considering whether findings were disseminated appropriately to recipients (7%, n=6) or conducting a formal evaluation (6%, n=5). More general appraisal criteria were reported more frequently, such as the nature of tasks patients were asked to perform (45%, n=39), how findings were disseminated (not requiring an active part of patients) (47%, n=40) or a general evaluation of the added value of involving patients in the research process (48%, n=41).
Reporting of patient involvement according to Wright et al6 critical appraisal tool: quantitative results
Further appraisal criteria that were addressed in very few publications were the nature of training of patients (8%, n=7) and researchers (1%, n=1), as well as ethical (3%, n=3) or methodological (6%, n=5) considerations and how these were managed (3%, n=3 and 5%, n=4, respectively).
Discussion
We analysed a sample of 86 publications in health research that reported on patient involvement. While many publications provided information on general aspects relating to patient involvement, even very basic details were often lacking. For example, nearly all publications reported generally that patient involvement took place during study design. However, more specific information about whether this involvement in study design included defining the research question or prioritising outcome measures was reported to a much lower extent. Similarly, 40% and 57% of publications reported on GRIPP2 aims and methods, respectively, but the reporting was often suboptimal and statements rather vague. Despite authors alluding to many aspects of patient involvement included in the GRIPP2-SF and critical appraisal guidelines, we identified a need to improve completeness and details of reporting. This corroborates findings from previous studies on reporting of patient involvement and/or engagement.12 13 16
Due to this incomplete reporting, coding according to the GRIPP2-SF categories was a challenge: the sparseness of reporting in many of the publications meant that these categories were relatively broad and overlapping. For example, a statement such as ‘patients helped with the identification of meaningful outcomes’ could describe the method (ie, focusing on the process) or the results (ie, identifying outcomes) of patient involvement. While the GRIPP2-SF reporting checklist is certainly useful to guide reporting in studies not having patient involvement as primary focus, our findings suggest that complementary measures could further bolster its impact on the quality and consistency of the patient involvement evidence base. Such measures could, for example, include broader requirements to include a statement on patient involvement in publications, more specific guidance for authors and peer reviewers, and standardised formats without word count restrictions to support more complete and consistent reporting. High-quality reporting is the basis for assessing the quality of patient involvement.
We observed considerable differences between our three samples regarding reporting, with both the Citation and PCORI sample providing more information on patient involvement than the BMJ sample. In these samples, patients were also more often coauthors of the manuscripts, reflecting their active roles in the whole research process. This is not surprising given the expected emphasis on patient involvement in the former samples compared with providing information in a mandatory section.
For the BMJ sample, we assessed the percentage of publications that reported patient involvement in the mandatory PPI section. Of all research articles published in 2019 (n=155), 21% reported patient involvement activities. In the sample of Price et al,11 which included research articles published between June 2015 and May 2016 in the BMJ, patient involvement was reported in only 11% of the articles. Thus, the proportion of research articles reporting on patient involvement doubled in only a few years, demonstrating the impact of this journal policy to enhance visibility and to raise awareness for patient involvement. While this trend is encouraging, descriptions of patient involvement in the BMJ sample were generally very short and did not elaborate on the results of patient involvement or provide a thorough description of the process. For example, several publications reported on the inclusion of patient discussion groups without describing the composition, specific tasks or influence of this group.
In contrast, in the PCORI and Citation samples, we often found very detailed descriptions of patient involvement, including the nature of performed tasks, concrete examples of its influence on the study and critical reflections (see Weschke et al21 and table 2). In many cases, these descriptions were provided in additional documents. This suggests that the word count limit imposed by journals likely contributes to the limited detail in patient involvement reporting. Additional documents or structured tables for reporting of patient involvement may be helpful. However, this approach may come with the risk that patient involvement is seen as an add-on rather than as an integral part of the conducted research.
Strengths and limitations
One of the strengths of our study was the use of a targeted search strategy to identify examples of patient involvement reporting in current practice across a variety of publication types and study designs in health research. Moreover, coding according to three distinct schemes allowed us to capture different aspects of relevance, including the phase of patient involvement, the use of and adherence to reporting guidelines (GRIPP2-SF), and the quality and impact of patient involvement (critical appraisal tool). All statements reporting patient involvement analysed in this study are openly available21 for further use. For example, coded statements may inform the development of automated tools to detect reporting of patient involvement in publications.
We could not systematically assess the quality of patient involvement according to the critical appraisal criteria as originally planned. Quality assessment highly depends on reporting completeness and detail, which was inconsistent across publications. Such an analysis in our diverse sample might have favoured long versus short reports, or participatory health research approaches versus PPI activities informing a clinical trial.
Initially, we did not plan to exclude publications applying qualitative research methods. However, we did not find a clear definition to differentiate between the active involvement of stakeholders and their involvement as participants in qualitative research, for example, in focus groups or interview studies.23–26 In some cases, both were reported in the same publication.23 This particular challenge has previously been noted in the context of assessing reporting of PPI.11 Despite attempts to delineate these approaches by differentiating between producing data to answer research questions versus informing decision-making processes of a research project,27 this is not common practice yet. More generally, excluded qualitative research studies that reported patient involvement often had patient involvement as primary focus. An analysis of these studies was beyond our scope. While the targeted search used in this study facilitated the identification of patient involvement reporting in practice, it was challenging to develop appropriate inclusion and exclusion criteria and apply them consistently across the wide variety of included studies. In some cases, a different categorisation may have been possible. Moreover, since reporting of patient involvement is sometimes limited to very brief statements within the main text of a publication, it is possible that some publications reporting patient involvement were missed. Finally, we used GRIPP2 as a reporting guideline to assess the completeness of reporting of the included publications. The use of reporting guidelines without modification to serve as evaluation tools has been questioned by Logullo et al28 as their purpose is to guide writing. However, the authors of GRIPP2 explicitly stated that it can also be used for planning patient involvement or for quality assurance.8
Conclusion
Despite important developments in the last years, patient involvement is still not a well-established approach in clinical or health research.14 29 Therefore, we would encourage journals to request an obligatory patient involvement statement from their authors, and to give guidance on detailed reporting in a structured table or additional document. We would also encourage journals and funding organisations to support the reporting of patient involvement by requiring the use of GRIPP2-SF as a reporting tool. Finally, we encourage researchers to include sufficient detail on patient involvement in their study to allow others to derive and apply lessons learnt in their own studies.
We expect that patient involvement will become more important in the next years to increase the relevance of research, in line with increasing demand from funders, publishers and society. Broader implementation of policies and more specific guidance are needed to leverage the impact of existing reporting guidelines, and thereby improve the quality of the patient involvement evidence base. Complete reporting builds the foundation of assessing the quality and appropriateness of patient involvement and is essential towards increasing its impact on research.
Data availability statement
Data are available in a public, open access repository. The data generated in this study are openly available under the Creative Commons Attribution 4.0 International license in Zenodo (https://doi.org/10.5281/zenodo.6245750).
Ethics statements
Patient consent for publication
Ethics approval
This study does not involve animal subjects, human participants or patient data; thus, ethics approval was not required.
Acknowledgments
We thank Lauren Fayish from PCORI for providing the dataset from the Engagement in Health Research Literature Explorer.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
SW and DLF contributed equally.
Contributors SW—conceptualisation, investigation, methodology, project administration, validation, writing (original draft), writing (review and editing). DLF—conceptualisation, data curation, investigation, methodology, project administration, validation, visualisation, writing (review and editing). AKS—conceptualisation, investigation, methodology, validation, visualisation, writing (review and editing). L-SB—investigation, writing (review and editing). DS—conceptualisation, methodology, writing (review and editing). SGS—conceptualisation, formal analysis, investigation, methodology, project administration, supervision, validation, visualisation, writing (review and editing). Guarantor for the overall content: SW.
Funding This research was funded by intramural funds of the QUEST Center for Responsible Research, Berlin Institute of Health, Charité–Universitätsmedizin Berlin (award/grant number N/A).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.