Fostering practice-based evidence through routine outcome monitoring in a university psychotherapy service for common mental health problems: a protocol for a naturalistic, observational study
================================================================================================================================================================================================

* Jorge Valdiviezo-Oña
* Adrián Montesano
* Chris Evans
* Clara Paz

## Abstract

**Introduction** Data-informed psychotherapy and routine outcome monitoring are growing as referents in psychotherapy research and practice. In Ecuador, standardised web-based routine outcome monitoring systems have not been used yet, precluding data-driven clinical decisions and service management. Hence, this project aims at fostering and disseminating practice-based evidence in psychotherapy in Ecuador by implementing a web-based routine outcome monitoring system in a university psychotherapy service.

**Methods and analyses** This is a protocol for an observational naturalistic longitudinal study. Progress and outcomes of treatment in the Centro de Psicología Aplicada of the Universidad de Las Américas in Quito, Ecuador will be examined. Participants will be adolescents and adults (≥11 years) seeking treatment, as well as therapists and trainees working at the centre between October 2022 and September 2025. Clients’ progress will be monitored by a range of key variables: psychological distress, ambivalence to change, family functioning, therapeutic alliance and life satisfaction. Sociodemographic information and satisfaction with treatment data will be collected before and at the end of treatment, respectively. Also, semi-structured interviews to explore therapists’ and trainees’ perceptions, expectations and experiences will be conducted. We will analyse first contact data, psychometrics of the measures, reliable and clinically significant change, outcome predictors as well as trajectories of changes. Moreover, we will conduct a framework analysis for the interviews.

**Ethics and dissemination** The protocol for this study was approved by the Human Research Ethics Committee of the Pontificia Universidad Católica del Ecuador (#PV-10–2022). The results will be disseminated in peer-reviewed scientific articles, at conferences and in workshops.

**Trial registration number** [NCT05343741](http://bmjopen.bmj.com/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT05343741&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom).

*   Change management
*   Clinical audit
*   Protocols & guidelines
*   Quality in health care
*   Latin America
*   EDUCATION & TRAINING (see Medical Education & Training)

### STRENGTHS AND LIMITATIONS OF THIS STUDY

*   This project targets adults and adolescents, allowing for intragroup and intergroup comparisons of trajectories of change and outcomes.

*   Links between changes in psychological distress and three transdiagnostic variables: therapeutic alliance, family functioning and ambivalence to change will be explored.

*   Data loss will be prevented by constant supervision of data collection procedures.

*   Though causal inferences are not possible from observational studies, ecological validity and generalisability will be strong.

*   This is a protocol for a study that will be conducted at a single service with a limited number of therapists; however, it offers a template for other interested services.

## Introduction

Certain characteristics of randomised controlled trials (RCTs) conducted from the evidence-based practice paradigm, such as strict inclusion criteria and control conditions, limit external validity of the findings. To counteract some of these limitations the practice-based evidence (PBE) paradigm has emerged.1

PBE studies share, at least, three common features: (1) data come from clinical non-controlled routine practice with inclusion criteria that are minimally restrictive and no researcher-imposed exclusion criteria; (2) the assessed variables should capture routine practice and (3) the research plan and its implementation are not defined by the researcher hierarchically, rather it is devised by close collaboration with clinicians.2 3 Routine Outcome Monitoring (ROM) is one key approach within PBE. ROM is the common use of measures to explore changes and outcomes during and ideally after treatment in everyday practice. ROM can be used at different levels (within and/or between clients; intraservice or multiservice) and with different purposes such as, monitoring change, service management, clinical decision-making, feedback for therapists, clients or both, etc.4 When using ROM, global outcome measures are typically used rather than disorder-specific measures, because they allow assessment of treatment outcomes from a transdiagnostic and transtheoretical perspective and across as many clients as possible.5–7 However, if a clinic provides highly specialised therapy services (eg, for eating disorders), disorder-specific measures may also be used.8

Previous PBE and psychotherapy research studies have been conducted mainly in Europe and North America. Available literature has addressed some core issues in psychotherapy such as effectiveness, therapist effects, therapeutic alliance, participants’ characteristics, contextual factors, among others that contribute to change in routine clinical practice.2 In this protocol, based on the literature review and in collaboration with practitioners, we have included several clinically relevant client variables to assess treatment progress and outcomes. Furthermore, we will explore therapists’ effects on outcomes since they have been observed in previous studies.9–14

Regarding clients’ variables, we will be monitoring outcomes by measuring psychological distress with a set of global outcome measures with two different age groups. Findings in previous studies are mixed: some studies suggest that greater initial symptom severity correlates with lower rates of reliable change, while others state that clients with high distress levels at baseline show greater change.15–18 There is also diversity of findings about age in studies with adult clients: some have found older adult clients show lower levels of improvement,19 while others have found the reverse.20 Broadening the age coverage, Cuijpers *et al*21 compared the impact of psychotherapies in different life stages and found that children and adolescents showed, on average, significantly lower effect sizes than young adults and middle-aged adults.

Another focus will be on monitoring therapeutic alliance and family functioning, since they are key factors affecting change. Regarding therapeutic alliance, many studies have pointed toward therapeutic alliance and clients’ improvement influencing each other on a continuous loop throughout therapy.19 22–27 Concerning family functioning in adults, previous findings suggest that worse interpersonal functioning in general is related to worse outcomes.28 Regarding family functioning in adolescents Sawyer and Borduin29 showed that improvements in family functioning mediated symptom reductions.

Besides those well-studied variables, we will study ambivalence to change given its clinical relevance and under-researched status in PBE studies.30 31 Ambivalence to change is a significant variable influencing outcomes in both improvement and deterioration cases.32

Only a handful of psychotherapy research studies have been conducted in Latin America (LA). Paz *et al*33 conducted the first review in LA to identify all published articles regarding psychotherapy outcome and change measures and found 207. When conducting their searches, they found 53 096 papers but only 3834 from LA: 7%. Globally, the first paper was published in 1947 but the first from LA was not published until 1990. Their results underline the shortage of research on outcome measures in LA and they showed that there was a particular deficit of LA papers using global rather than diagnosis specific outcome measures. However, their review showed that PBE is an emerging topic in LA.

In Ecuador, only two PBE studies have been published; Zúñiga-Salazar *et al*34 assessed changes in the psychological distress of LGBTIQA+ people who attended affirmative psychological services and Valdiviezo-Oña *et al*35 explored the psychological distress outcomes of clients who completed the Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) at least twice at the Centro de Psicología Aplicada (Centre for Applied Psychology, CPA) between 2018 and 2020. To date, no study in Ecuador has assessed ambivalence to change in psychotherapy for any age group and none has explored adolescents’ life satisfaction and change in therapy. Finally, no previous study has implemented a web-based ROM system in psychotherapy services in Ecuador.

## Study objectives

### General objective

To foster and disseminate PBE in psychotherapy in Ecuador by implementing a pathfinder web-based ROM system in a university psychotherapy service.

### Specific objectives

1.  To implement a standardised ROM system in a university psychotherapy service in Ecuador.

2.  To describe the trajectories of change of clients attending the service.

3.  To explore the predictors of change (e.g., sociodemographics, therapeutic alliance, family functioning and ambivalence to change) of clients attending the service.

4.  To delve into therapists’ perceptions and expectations regarding the implementation of the ROM system in the service.

5.  To describe the main challenges and opportunities of implementing a ROM system in psychotherapeutic practice in a university psychotherapy service in Ecuador.

## Methods and analysis

### Study design

This is a non-experimental, longitudinal, exploratory and descriptive mixed methods study, with a mainly quantitative focus. This study will be conducted in a naturalistic setting, at the CPA of the Universidad de Las Américas (UDLA) in Quito, Ecuador.

### Participants

#### Clients

This is a study of routine practice including both adolescent (11–17 years old) and adult clients, and it adopts the inclusion and exclusion criteria of the centre where it is conducted. All clients who approach the service until September 2025 seeking psychotherapy and are 11 years old or older will be eligible for participation. The CPA offers psychological services for a wide variety of psychological problems; however, severe mental health problems such as psychosis, substance abuse, physical and sexual violence are not admitted for treatment in the centre. Most clients attend one session per week, but some attend two sessions weekly, and others attend a session every 2 weeks. Duration of the therapies varies, ranging from 1 to 20 sessions with a median number of 5 and a mean of 6.52 (*SD* = 4.77; for more details see Valdiviezo-Oña *et al.*35). Considering the actual client flow (approximately 350 per year), taking into account possible refusals (5%–10%), we expect to collect information from at least 300 adults and 40 adolescents. Based on a meta-analysis,36 including more than 83 000 adult psychotherapy clients from 669 studies that concludes that almost 20% of clients opt out of psychotherapy, we expect our rate to be alike or lower.

#### Psychotherapists, trainees and supervisors

This centre employs a co-therapy model, whereby a professional psychotherapist and a clinical psychology trainee work together from different psychotherapeutic models (e.g., cognitive-behavioural, systemic and integrative; for more details, see Valdiviezo-Oña *et al.*35). There is also a team of supervisors monitoring the progress of all cases. All therapists, trainees and supervisors will be invited to participate. We estimate that we will include around 40–60 psychotherapists and trainees.

### Measures

All measures will be offered to the participants in Spanish. Spanish is the official language and the most widely used by the country’s inhabitants (93%)37 with a 94% literacy rate.38

### Primary outcome measures

#### Clinical Outcomes in Routine Evaluation System measures

##### Clinical Outcomes in Routine Evaluation-Outcome Measure

The Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM)39 is a 34-item self-report instrument designed to be used in heterogeneous services, based on a pan-theoretic core of psychological distress in adults, including: well-being, problems, risk and functioning. Items are scored on a scale from 0 (never) to 4 (always or almost always). Higher scores indicate greater psychological distress. The Spanish version of this measure was translated by Feixas *et al*40 and has shown good psychometric properties in Spain41 and Ecuador.42

##### Clinical Outcomes in Routine Evaluation-10

The Clinical Outcomes in Routine Evaluation-10 (CORE-10)43 is a short psychological distress assessment measure for adults, whose items were drawn from the CORE-OM. It has shown good psychometric properties in its original version in the UK43 and in its Spanish version in Ecuador, showing a good internal consistency score as well as convergent and discriminant validity (Valdiviezo-Oña *et al.*, submitted for publication).

##### Young Person’s-Clinical Outcomes in Routine Evaluation

The Young Person’s-Clinical Outcomes in Routine Evaluation (YP-CORE) is a brief self-report instrument that assesses psychological distress in adolescents.44 It has 10 items, which are answered on a 5-point scale (0–4). The YP-CORE was translated to Spanish by Feixas *et al*45 and showed adequate psychometric properties. In the present study, we will use a version of this measure adapted for Ecuador and LA approved by the CORE System Trust (CST; [https://www.coresystemtrust.org.uk/](https://www.coresystemtrust.org.uk/)), whose psychometric properties are being explored.

### Secondary measures

#### Systemic Clinical Outcome and Routine Evaluation-15

This is a 15-item questionnaire with a scale of 1 to 5, where one means *strongly disagree* and five means *strongly agree*. It was designed to be completed by family members, aged 12 years or older.46 It has three dimensions: *Strengths*, *Difficulties*, and *Communication*. Lower scores correspond to better family functioning. In addition, the Systemic Clinical Outcome and Routine Evaluation-15 (SCORE-15) has two scales ranging from 1 to 10 where participants rate perceived effectiveness of the intervention and perceived severity of the difficulty leading to treatment. The Spanish version of this measure has shown adequate reliability and validity in Spain.47 This measure will be administered to all clients individually. In couples and family therapy, every attending member will be asked to complete it.

#### Life Satisfaction Scale

Participants are asked to rate their overall degree of satisfaction with life in the present with the following question: Considerando todos los aspectos, ¿qué grado de satisfacción tiene con su vida? [All aspects considered, how satisfied are you with your life?]. This measure is rated on a 10-point Likert scale, in which 1 represents Nada satisfecho (not at all satisfied) and 10 represents Totalmente satisfecho (totally satisfied). This single item has shown an acceptable reliability, and it is theoretically consistent.48 This measure will be administered to all clients.

#### Ambivalence in Psychotherapy Questionnaire

It is a 9-item questionnaire that measures the clients’ level of ambivalence toward change in psychotherapy. It includes two subscales, demoralisation and wavering, and an overall score of ambivalence toward change.30 Both the original version30 and the Spanish version (Montesano *et al.*, submitted for publication) have demonstrated good psychometric properties, including good convergent and divergent validity.

#### Therapeutic Alliance Scale

It is a one-item measure created ad hoc for this study that evaluates therapeutic alliance on a 10-point scale, in which 1 indicates: En general, no me siento conectado/a con mi terapeuta, no hemos hablado sobre lo que me interesaba trabajar [In general, I don't feel connected with my therapist, we haven't talked about what I was interested in working on], while 10 indicates: En general, me siento conectado/a con mi terapeuta, hemos hablado sobre lo que me interesaba trabajar [In general, I feel connected with my therapist, we have talked about what I was interested in working on].

#### Consumer Reports Effectiveness Scale

The Consumer Reports Effectiveness Scale (CRES-4) has four items: one question on satisfaction, one on the level of resolution of the main problem, one on the emotional state before the start of treatment and one question on the emotional state when completing the questionnaire.49 These questions allow interpretations to be made about clients' perceived change in their emotional state, satisfaction and resolution of the problem.

### Instruments for collecting sociodemographic and clinical information

#### CORE Therapy Assessment Form

The CORE Therapy Assessment Form (CORE-A-TAF) is a pragmatic form designed to be filled out by therapists at baseline. The CORE-A-TAF includes referral information, sociodemographic data on the client and data on the nature, severity and duration of the client’s problems.50 51 The CORE-A-TAF includes questions regarding risk which can be later contrasted with the clients’ self-reported data in the CORE-OM, CORE-10 and YP-CORE items that measure risk. We will use the Spanish version of this measure, which has already been translated for its use in Ecuador and approved by the CST.

#### CORE End of Therapy

The CORE end of therapy (CORE-A-EoT) is a pragmatic form designed to be filled out by therapists immediately after treatment completion. The CORE-A-EoT reports information regarding the treatment, including the number of sessions, type of therapy, length and frequency of sessions, whether the ending was planned or unplanned, and the potential benefits of therapy.50 51 We will use its Spanish version, which has already been translated for its use in Ecuador and approved by the CST.

### Qualitative techniques

#### Semi-structured interview

Semi-structured interviews will be conducted with the centre’s therapists, trainees and supervisors using some predefined but open-ended questions.52 The interview questions (see online supplemental material S1 for the interview guides in the original Spanish version and their translation into English) are related to this study’s fourth specific objective; therefore, they will address topics such as: knowledge and expectations regarding ROM, perception of therapists’ and trainees’ agency, as well as perception of usability, risks and benefits of ROM, overall experience and perception of the implementation of the ROM system in the service. All interviews will be recorded and stored securely until this study is completed in an institutional account’s shared online folder with two-step verification. Only the principal investigator (JV-O) and research assistants will have access to this folder.

### Supplementary data

[[bmjopen-2023-071875supp001.pdf]](pending:yes)

### Procedure

The centre has already approved the implementation of the project, data collection started in October 2022 and is planned to end in September 2025. All adolescent and adult clients initiating psychotherapy at the CPA will be invited to participate. Once enrolled, the clients will be asked to complete the measures throughout the therapeutic process. Figure 1 shows an example of the assessment timeline for both an adult and an adolescent client. For clients’ data collection, we will use MarBar, a free online software available at [https://www.marbarsystem.com/](https://www.marbarsystem.com/) for Spanish-speaking therapists and centres.

![Figure 1](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/13/5/e071875/F1.medium.gif)

[Figure 1](http://bmjopen.bmj.com/content/13/5/e071875/F1)

Figure 1 
Example of assessment schedule for clients attending nine sessions. APQ, Ambivalence in Psychotherapy Questionnaire; CORE-A-EOT, CORE End of Therapy; CORE-A-TAF, CORE Therapy Assessment Form; CORE-OM, Clinical Outcomes in Routine Evaluation-Outcome Measure; CORE-10, Clinical Outcomes in Routine Evaluation-10; CRES-4, Consumer Reports Effectiveness Scale; LSS, Life Satisfaction Scale; SCORE-15, Systemic Clinical Outcome and Routine Evaluation-15; TAS, Therapeutic Alliance Scale; YP-CORE, Young Person’s-Clinical Outcomes in Routine Evaluation. Note. The CORE-OM and LSS are completed before the first session and after the last session. The CORE-10, SCORE-15, APQ and YP-CORE are completed before the specified sessions. The CORE-A-TAF is completed by the therapists between the initial session and the third session. The TAS after each session. CRES-4 and CORE-A-EoT after the last available session. This flowchart represents an example of the assessment structure for patients who attend at least nine sessions. From that point onwards CORE-10, YP-CORE and TAS data continue to be collected routinely every session. APQ is administered to adult patients every four sessions (W1, W4, W8, etc.) until treatment completion.

In addition, individual semi-structured interviews will be conducted with psychotherapists and new trainees at the beginning and at the end of each term with each new cohort of trainees. The first will address expectations and perceptions regarding ROM, whereas the latter will address general experiences and perceptions regarding the implementation of the ROM system.

### Data analysis

The units of analysis will be clients and psychotherapists undergoing and providing therapy at the CPA, respectively. In this section, we will briefly describe how data will be analysed. For further details, please refer to online supplemental table S2. Here we present the pre-planned analyses; however, post-hoc analyses may arise. We will clearly state post-hoc findings as such in future publications.

### Supplementary data

[[bmjopen-2023-071875supp002.pdf]](pending:yes)

### Quantitative analyses

The analyses in this project are framed in the Problem, Plan, Data, Analysis, Conclusions and Communication approach.53 54 The problem is to study clients’ progress and outcomes of treatment regarding psychological distress, life satisfaction, ambivalence in psychotherapy, family functioning and therapeutic alliance in a university psychotherapy service in Ecuador. When addressing this problem, the first specific objective is accomplished. The plan is to analyse the routinely collected item-level data from the aforementioned measures.

All quantitative analyses will be performed with R in Rstudio55 56 and they will contribute to accomplishing the second and third specific objectives: to describe the trajectories of change of clients attending the service and to explore the predictors of change (e.g., sociodemographics, therapeutic alliance, family functioning nd ambivalence to change) of clients attending the service.

### First contact analyses

Descriptive statistics of the sociodemographic and clinical characteristics of the participants, as well as of the scores of the measures will be reported. In all analyses, whenever possible, we will report the information with 95% bootstrapped CIs. Additionally, we will evaluate the acceptability and the internal consistency of the CORE-OM, YP-CORE, CORE-10, SCORE-15 and Ambivalence in Psychotherapy Questionnaire (APQ). Correlations between questionnaires will also be explored.

We will evaluate the associations among gender, age and education and the total scores in the first completion of the measures.

### Trajectories and predictors of change

Analyses of the trajectories of change in the measures—CORE-10, YP-CORE, APQ and SCORE-15—that are completed on three or more occasions will be performed, at the individual and group level using trajectory plots with two axes (ie, Cat’s cradle plots for individual and group data and Violin plots for group data, see figures 2 and 3). One axis representing the scores on the measure, and the second representing the different assessment points in time. Group trajectories of change will be plotted with violin plots, showing mean scores with 95% CI for each occasion the measures were completed. Predictor/subgroup analyses will be conducted addressing gender, educational level, age and risk levels.

![Figure 2](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/13/5/e071875/F2.medium.gif)

[Figure 2](http://bmjopen.bmj.com/content/13/5/e071875/F2)

Figure 2 
Cat’s cradle plot example. Note. This cat’s cradle plot example has been created with the data published in the Zúñiga-Salazar *et al*34 study, however this figure was not included in that article. Besides trajectories of change, this plot shows a nonlinear fit (LOESS regression) — ignoring nesting within patients— in blue, a linear fit in brown and a quadratic fit in orange.

![Figure 3](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/13/5/e071875/F3.medium.gif)

[Figure 3](http://bmjopen.bmj.com/content/13/5/e071875/F3)

Figure 3 
Violin plot example. Note. This Violin plot example summarizes the distributions of scores and means for participants across sessions. The horizontal blue line is the mean score at the first completion. The blue line with interconnected dots shows the mean scores with confidence intervals across sessions.

### Intrapersonal, intragroup and intergroup differences

This study will have several analytical levels that will produce nested data. To model the dependencies between these levels, this study will use multilevel analyses, and intends to use growth-mixture models (GMM) and latent class growth analyses (LCGA). Multilevel analyses (see the levels of the data in figure 4) will allow to evaluate differences at the individual client level, and at the therapist level both intragroup (clients of the same therapist) and intergroup (comparing clients from different therapists). In light of the intended nested analyses, to get a power level of 0.80 and assuming five measurements per each participant, we would need to have at least 22 different therapists with at least eight clients each (a minimum recruitment goal of 176 clients).57

![Figure 4](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/13/5/e071875/F4.medium.gif)

[Figure 4](http://bmjopen.bmj.com/content/13/5/e071875/F4)

Figure 4 
Example of data nesting levels showing two psychotherapists and three participants. PT = Psychotherapist; C = Client; O = Occasions each patient filled out each questionnaire.

Successful implementation of GMM and LCGA requires a sufficiently large sample size (at least 300 clients to get acceptable results).58 The decision to proceed with these analyses will be contingent on sample size but also on the data completeness (missing scores), number of time points, high within-group initial score variability and any marked violation of distributional assumptions. We will consider these criteria on the basis of: (1) a small sample size (<250) limits class identification and can generate model convergence problems59 60; (2) at least 200 participants are needed with complete or not more than 20% missing data, high class separation and few classes61; (3) few time points (<4) can reduce class enumeration performance and generate bias when estimating random effect variances and covariances62; (4) high within-group initial score variability in individuals with similar growth patterns can result in not identifying or differentiating classes with precision63 and (5) breaking distributional assumptions such as within-class normality in GMM can lead to detecting more groups within the data than there actually are.64

### Reliable and Clinically Significant Change and Reliable Trend Index

In addition to these group aggregate analyses, we will assess pre–post clinically significant change in the CORE-10, YP-CORE, APQ and SCORE-15 scores using Jacobson and Truax’s65 Reliable Change Index and the Reliable Trend Index to assess change across several time points.66

### Handling missing data

To handle missing item-level data, we will use prorating whenever possible. Prorating is a technique that takes the mean of the item-level scores at disposal from each individual and imputes the scale scores for each individual if a cut-off criterion—which varies among measures, but usually goes from 50% to 80% complete items—is met. Using this technique only the incomplete cases with fewer missing items than the cut-off criterion will be used for the analyses.67 68 To minimise the impact of clients opting out of therapy leading to incomplete data we are collecting data for all measures at multiple time points, and specifically for primary outcomes measures, we are collecting data session by session to ensure we always have information on psychological distress from the last available session for each client.

### Benchmarking

Benchmarking refers to ‘the establishment of reference points to aid the interpretation of data’ (p 358).69 Benchmarking of the following aspects related to the CORE measures will be performed to contrast local data with that from other countries and regions regarding outcomes; baseline score; problems/concerns identified; outcome measure completion rates (acceptability, valid and invalid/partially completed questionnaires); number of sessions; percentage of planned and unplanned treatment ends; attended and unattended sessions and rates of reliable and clinically significant change.

### Qualitative analyses

The semi-structured interviews will be analysed using framework analysis. This method was originally developed for use in large-scale policy research, as it permits identifying commonalities and differences in data before focusing on the relationships between different parts of the data.70 The framework analysis of the interviews will contribute to accomplishing the study’s fourth and fifth specific objectives.

## Discussion

In Ecuador and in LA, ROM has historically not been a common practice in psychotherapeutic contexts.33 This project is the first to propose the use of a web-based ROM system assessing several transdiagnostic variables in Ecuador and one of the few ROM systems existing in LA.71–73

We have designed this protocol considering many characteristics of high-quality practice-oriented studies, defined by Castonguay *et al*,2 which include factors related to (1) clinical helpfulness; (2) feasibility and (3) methodological and statistical sophistication. For more details on how these characteristics are addressed in this project, please refer to table 1.

View this table:
[Table 1](http://bmjopen.bmj.com/content/13/5/e071875/T1)

Table 1 
Characteristics of high-quality practice-oriented studies defined by Castonguay *et al*2 present in this protocol

The relevance of the study resulting from this protocol can be discussed considering its short-term, medium-term and long-term benefits as well as considering specific benefits for the university psychotherapy service and benefits for psychotherapy research and practice in Ecuador and LA.

With respect to the benefits for the clinic, in the short-term therapists will be trained and will learn about PBE and ROM in psychotherapy and its potential uses and advantages. In the medium term, the results of this project could contribute to the development of a positive feedback loop between practice, training and research at the CPA. For instance, new information about change trajectories leading to changes in clinical practice and additional data collection following the implementation of changes. This knowledge can also be transferred to the syllabi of UDLA clinical psychology undergraduate and postgraduate programmes.

In the long term, this project aims to contribute to establishing outcome monitoring as a routine practice to improve the quality of psychological interventions in individual, couple and family therapy from an integrative data-informed perspective at the CPA. Eventually, clients will benefit the most by receiving a treatment that is in a constant process of refinement.

Regarding benefits for psychotherapy research and practice in Ecuador and LA that transcend this specific centre, in the short term, a culturally contextualised data collection system will be developed, implemented and made available for free to Spanish-speaking clinicians and researchers. This, in the medium term and long term, will allow sharing an open access common strategy for data collection and collaboration toward a global data sharing frame in LA and beyond. It is expected that the standardisation of ROM will yield solid culturally sensitive evidence for patterns in trajectories of change and predictors, serving as a reference for practitioners and researchers alike at the national and regional levels.

In sum, the study derived from this protocol seeks to promote the consistent and widespread use of ROM measures, which could help not only in establishing a culture of PBE in Ecuador and in LA, but also in the creation of practice-based research networks in the field of psychological interventions. These networks are much needed to boost researchers and clinicians’ collaboration and sharing of goals, methods and language.74 The results of this project will constitute a critical first step toward the creation of a big database and the subsequent training of culturally sensitive algorithms to possibly develop normative trajectories of change and expected recovery curves. This kind of advancements has proven to be of great support for psychotherapy practice optimisation, as it allows to enhance treatment outcomes for clients at risk of treatment failure.75 76

Given the non-experimental nature of the study proposed in this protocol, its main limitation is related to the difficulty of proving causality, which contrasts with RCTs. In other words, when conducting non-experimental studies, causal relationships cannot strictly be attributed, notwithstanding this type of study can help identify associations between variables as well as relevant areas for psychotherapeutic and research exploration and improvement.

## Ethics and dissemination

The protocol for this study was approved by the Human Research Ethics Committee of the Pontificia Universidad Católica del Ecuador (#PV-10–2022). The results of this project are socially relevant; thus, we will socialise them in several venues and formats. They will be disseminated at the CPA and in other places to reach a wider audience. A formal report will be delivered to the CPA’s director and supervisors, and we plan to organise talks/workshops to present the main results to all the people working at the centre. To reach a wider audience, the results will be disseminated in peer-reviewed scientific articles, in conferences and meetings (eg, Society for Psychotherapy Research and Red Latinoamericana Psicoterapia y Cambio meetings) and plots and infographics will be shared through social media accompanied by short descriptions or explanations.

## Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study. In accordance with the guidelines of the research ethics committee that granted approval for this study, individuals seeking access to the data that will be generated from the implementation of this protocol must submit a written request to the corresponding author. Suitably-qualified researchers may obtain a de-identified dataset in encrypted form, as well as a data dictionary. To have access to the data, researchers must (a) obtain ethical approval for their proposed analysis, (b) pre-register their statistical analysis plan, and (c) sign a data-sharing agreement that permits data storage and analysis for a limited period.

## Ethics statements

### Patient consent for publication

Not applicable.

## Acknowledgments

We thank the Centro de Psicología Aplicada for their support in the implementation of this project. We are grateful to all therapists, trainees, supervisors and clients who will participate in this study. Furthermore, we thank the Society for Psychotherapy Research and specially the family of Marna S Barret, PhD, for creating the Marna Barrett Award for Excellence in Psychotherapy to support excellence in the study of psychotherapy, supervision, training and mentoring practitioners. The system we are using to collect data in this study was created thanks to this prize awarded to CP in 2021.

## Footnotes

*   Twitter @jorgevaldio, @clarapaz

*   Contributors The study concept and design were conceived by JV-O, AM, CE and CP. Data analysis plan was designed by JV-O, AM, CE and CP. JV-O prepared the first draft of the manuscript. All authors read, edited and approved the final version.

*   Funding This work is supported by the Dirección de Investigación y Vinculación, Universidad de Las Américas, Ecuador, grant number [PSI.CPE.22.03].

*   Disclaimer This funding source had no role in the design of this study and will not have any role during analyses, interpretation of the data or decision to submit results.

*   Competing interests None declared.

*   Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  Margison FR, Barkham M, Evans C, et al. Measurement and psychotherapy. evidence-based practice and practice-based evidence. Br J Psychiatry 2000;177:123–30. [doi:10.1192/bjp.177.2.123](http://dx.doi.org/10.1192/bjp.177.2.123)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImJqcHJjcHN5Y2giO3M6NToicmVzaWQiO3M6OToiMTc3LzIvMTIzIjtzOjQ6ImF0b20iO3M6MjY6Ii9ibWpvcGVuLzEzLzUvZTA3MTg3NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

2.  1.  Barkham M, 
    2.  Lutz W, 
    3.  Castonguay L
    
    Castonguay L, Barkham M, Jeong Youn S, et al. Chapter 6: practice-based Evidence- findings from routine clinical settings. In: Barkham M, Lutz W, Castonguay L, eds. Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change. Wiley, 2021. [doi:https://www.wiley.com/en-us/Bergin+and+Garfield%27s+Handbook+of+Psychotherapy+and+Behavior+Change%2C+7th+Edition-p-9781119536567](http://dx.doi.org/https://www.wiley.com/en-us/Bergin+and+Garfield%27s+Handbook+of+Psychotherapy+and+Behavior+Change%2C+7th+Edition-p-9781119536567)
    
    

3.  Horn SD, Gassaway J. Practice-based evidence study design for comparative effectiveness research. Med Care 2007;45:S50–7. [doi:10.1097/MLR.0b013e318070c07b](http://dx.doi.org/10.1097/MLR.0b013e318070c07b)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1097/MLR.0b013e318070c07b&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=17909384&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000250333800011&link_type=ISI) 

4.  Evans C, Carlyle J-A. Outcome measures and evaluation in counselling and psychotherapy1st ed. Thousand Oaks: SAGE Publishing, 2021.
    
    

5.  Boswell JF. Monitoring processes and outcomes in routine clinical practice: A promising approach to plugging the holes of the practice-based evidence Colander. Psychother Res 2020;30:829–42. [doi:10.1080/10503307.2019.1686192](http://dx.doi.org/10.1080/10503307.2019.1686192)
    
    

6.  Tarescavage AM, Ben-Porath YS. Psychotherapeutic outcomes measures: A critical review for practitioners. J Clin Psychol 2014;70:808–30. [doi:10.1002/jclp.22080](http://dx.doi.org/10.1002/jclp.22080)
    
    

7.  Wampold BE. Routine outcome monitoring: coming of age-with the usual developmental challenges. Psychotherapy (Chic) 2015;52:458–62. [doi:10.1037/pst0000037](http://dx.doi.org/10.1037/pst0000037)
    
    

8.  Grau Touriño A, Feixas G, Medina JC, et al. Effectiveness of integrated treatment for eating disorders in Spain: protocol for a Multicentre, naturalistic, observational study. BMJ Open 2021;11:e043152. [doi:10.1136/bmjopen-2020-043152](http://dx.doi.org/10.1136/bmjopen-2020-043152)
    
    

9.  Ali S, Littlewood E, McMillan D, et al. Heterogeneity in patient-reported outcomes following low-intensity mental health interventions: A Multilevel analysis. PLoS ONE 2014;9:e99658. [doi:10.1371/journal.pone.0099658](http://dx.doi.org/10.1371/journal.pone.0099658)
    
    

10. Firth N, Saxon D, Stiles WB, et al. Therapist effects vary significantly across psychological treatment care sectors. Clin Psychol Psychother 2020;27:770–8. [doi:10.1002/cpp.2461](http://dx.doi.org/10.1002/cpp.2461)
    
    

11. Green H, Barkham M, Kellett S, et al. Therapist effects and IAPT psychological wellbeing practitioners (Pwps): a Multilevel Modelling and mixed methods analysis. Behav Res Ther 2014;63:43–54. [doi:10.1016/j.brat.2014.08.009](http://dx.doi.org/10.1016/j.brat.2014.08.009)
    
    

12. Hayes JA, Owen J, Bieschke KJ. Therapist differences in symptom change with racial/ethnic minority clients. Psychotherapy 2015;52:308–14. [doi:10.1037/a0037957](http://dx.doi.org/10.1037/a0037957)
    
    

13. Kivlighan DM Hill CE, Ross K, et al. Testing a mediation model of psychotherapy process and outcome in Psychodynamic psychotherapy: previous client distress, Psychodynamic techniques, Dyadic working alliance, and current client distress. Psychotherapy Research 2019;29:581–93. [doi:10.1080/10503307.2017.1420923](http://dx.doi.org/10.1080/10503307.2017.1420923)
    
    

14. Larrison CR, Schoppelrey SL, Hack-Ritzo S, et al. Clinician factors related to outcome differences between black and white patients at Cmhcs. PS 2011;62:525–31. [doi:10.1176/ps.62.5.pss6205\_0525](http://dx.doi.org/10.1176/ps.62.5.pss6205_0525)
    
    

15. Gyani A, Shafran R, Layard R, et al. Enhancing recovery rates: lessons from year one of IAPT. Behav Res Ther 2013;51:597–606. [doi:10.1016/j.brat.2013.06.004](http://dx.doi.org/10.1016/j.brat.2013.06.004)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.brat.2013.06.004&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=23872702&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

16. McAleavey AA, Youn SJ, Xiao H, et al. Effectiveness of routine psychotherapy: method matters. Psychother Res 2019;29:139–56. [doi:10.1080/10503307.2017.1395921](http://dx.doi.org/10.1080/10503307.2017.1395921)
    
    

17. Saxon D, Barkham M, Foster A, et al. The contribution of therapist effects to patient dropout and deterioration in the psychological therapies. Clin Psychol Psychother 2017;24:575–88. [doi:10.1002/cpp.2028](http://dx.doi.org/10.1002/cpp.2028)
    
    

18. Saxon D, Firth N, Barkham M. The relationship between therapist effects and therapy delivery factors: therapy modality, dosage, and non-completion. Adm Policy Ment Health 2017;44:705–15. [doi:10.1007/s10488-016-0750-5](http://dx.doi.org/10.1007/s10488-016-0750-5)
    
    

19. Kumar M, Kuria MW, Othieno CJ, et al. Improving Psychotherapies offered in public hospitals in Nairobi, Kenya: extending practice-based research model for Lmics. Int J Ment Health Syst 2018;12:76. [doi:10.1186/s13033-018-0254-7](http://dx.doi.org/10.1186/s13033-018-0254-7)
    
    

20. Roseborough DJ, McLeod JT, Bradshaw WH. Psychodynamic psychotherapy: A quantitative, longitudinal perspective. Res Soc Work Pract 2012;22:54–67. [doi:10.1177/1049731511412790](http://dx.doi.org/10.1177/1049731511412790)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1177/1049731511412790&link_type=DOI) 

21. Cuijpers P, Karyotaki E, Eckshtain D, et al. Psychotherapy for depression across different age groups: A systematic review and meta-analysis. JAMA Psychiatry 2020;77:694–702. [doi:10.1001/jamapsychiatry.2020.0164](http://dx.doi.org/10.1001/jamapsychiatry.2020.0164)
    
    

22. Falkenström F, Granström F, Holmqvist R. Therapeutic alliance predicts symptomatic improvement session by session. Journal of Counseling Psychology 2013;60:317–28. [doi:10.1037/a0032258](http://dx.doi.org/10.1037/a0032258)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1037/a0032258&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=23506511&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

23. Falkenström F, Kuria M, Othieno C, et al. Working alliance predicts symptomatic improvement in public Hospital’Delivered psychotherapy in Nairobi, Kenya. Journal of Consulting and Clinical Psychology 2019;87:46–55. [doi:10.1037/ccp0000363](http://dx.doi.org/10.1037/ccp0000363)
    
    

24. Fisher H, Atzil-Slonim D, Bar-Kalifa E, et al. Emotional experience and alliance contribute to therapeutic change in Psychodynamic therapy. Psychotherapy (Chic) 2016;53:105–16. [doi:10.1037/pst0000041](http://dx.doi.org/10.1037/pst0000041)
    
    

25. Gómez Penedo JM, Zilcha-Mano S, Roussos A. Interpersonal profiles in emotional disorders predict the importance of alliance negotiation for early treatment outcome. Journal of Consulting and Clinical Psychology 2019;87:617–28. [doi:10.1037/ccp0000417](http://dx.doi.org/10.1037/ccp0000417)
    
    

26. Rubel JA, Rosenbaum D, Lutz W. Patients’ in-session experiences and symptom change: session-to-session effects on a Within- and between-patient level. Behaviour Research and Therapy 2017;90:58–66. [doi:10.1016/j.brat.2016.12.007](http://dx.doi.org/10.1016/j.brat.2016.12.007)
    
    

27. Zilcha-Mano S, Errázuriz P. One size does not fit all: examining heterogeneity and identifying Moderators of the alliance–outcome Association. J Couns Psychol 2015;62:579–91. [doi:10.1037/cou0000103](http://dx.doi.org/10.1037/cou0000103)
    
    

28. 1.  Barkham M, 
    2.  Lutz W, 
    3.  Castonguay L
    
    Constantino MJ, Boswell JF, Coyne AE. Chapter 7: patient, therapist, and relational factors. In: Barkham M, Lutz W, Castonguay L, eds. Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change. Wiley, 2021. [https://www.wiley.com/en-us/Bergin+and+Garfield%27s+Handbook+of+Psychotherapy+and+Behavior+Change%2C+7th+Edition-p-9781119536567](https://www.wiley.com/en-us/Bergin+and+Garfield%27s+Handbook+of+Psychotherapy+and+Behavior+Change%2C+7th+Edition-p-9781119536567)
    
    

29. Sawyer AM, Borduin CM. Effects of Multisystemic therapy through Midlife: A 21.9-year follow-up to a randomized clinical trial with serious and violent juvenile offenders. J Consult Clin Psychol 2011;79:643–52. [doi:10.1037/a0024862](http://dx.doi.org/10.1037/a0024862)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1037/a0024862&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21787045&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

30. Oliveira JT, Ribeiro AP, Gonçalves MM. Ambivalence in psychotherapy questionnaire: development and validation studies. Clin Psychol Psychother 2020;27:727–35. [doi:10.1002/cpp.2457](http://dx.doi.org/10.1002/cpp.2457)
    
    

31. Urmanche AA, Oliveira JT, Gonçalves MM, et al. Ambivalence, resistance, and alliance Ruptures in psychotherapy: it’s complicated. Psychoanalytic Psychology 2019;36:139–47. [doi:10.1037/pap0000237](http://dx.doi.org/10.1037/pap0000237)
    
    

32. Engle DE, Arkowitz H. Ambivalence in psychotherapy: Facilitating readiness to change. Guilford Press, 2006.
    
    

33. Paz C, Mascialino G, Proaño K, et al. Psychological intervention change measurement in Latin America: where from? where to Psychotherapy Research 2021;31:132–41. [doi:10.1080/10503307.2020.1750728](http://dx.doi.org/10.1080/10503307.2020.1750728)
    
    

34. Zúñiga-Salazar E, Valdiviezo-Oña J, Ruiz-Cordoba O, et al. Servicios Psicológicos Afirmativos para Personas LGTBIQA+ en Ecuador: Cambios en El Malestar Psicológico. Ter Psicol 2021;39:353–74. [doi:10.4067/S0718-48082021000300353](http://dx.doi.org/10.4067/S0718-48082021000300353)
    
    

35. Valdiviezo-Oña J, Granja E, Cuadros-López A, et al. Practice-based research with psychologists-in-training: presentation of a supervision model and use of routine outcome monitoring. Stud Psychol 2022.
    
    

36. Swift JK, Greenberg RP. Premature discontinuation in adult psychotherapy: A meta-analysis. J Consult Clin Psychol 2012;80:547–59. [doi:10.1037/a0028226](http://dx.doi.org/10.1037/a0028226)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1037/a0028226&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=22506792&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

37. Instituto Nacional de Estadística y Censos. VII Censo de Población y VI de Vivienda. 2010.
    
    

38. Instituto Nacional de Estadística y Censos. Encuesta Nacional de Empleo, Desempleo y Subempleo - Anual. 2021.
    
    

39. Evans C, Connell J, Barkham M, et al. Towards a standardised brief outcome measure: Psychometric properties and utility of the CORE-OM. Br J Psychiatry 2002;180:51–60. [doi:10.1192/bjp.180.1.51](http://dx.doi.org/10.1192/bjp.180.1.51)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImJqcHJjcHN5Y2giO3M6NToicmVzaWQiO3M6ODoiMTgwLzEvNTEiO3M6NDoiYXRvbSI7czoyNjoiL2Jtam9wZW4vMTMvNS9lMDcxODc1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

40. Feixas G, García-Grau E, Bados A, et al. La Versión Española del CORE-OM: clinical outcomes in routine evaluation - outcome measure. Rdp 2012;23:109–35. [doi:10.33898/rdp.v23i89.641](http://dx.doi.org/10.33898/rdp.v23i89.641)
    
    

41. Trujillo A, Feixas G, Bados A, et al. Psychometric properties of the Spanish version of the clinical outcomes in routine evaluation - outcome measure. Neuropsychiatr Dis Treat 2016;12:1457–66. [doi:10.2147/NDT.S103079](http://dx.doi.org/10.2147/NDT.S103079)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

42. Paz C, Mascialino G, Evans C. Exploration of the Psychometric properties of the clinical outcomes in routine evaluation-outcome measure in Ecuador. BMC Psychol 2020;8:94. [doi:10.1186/s40359-020-00443-z](http://dx.doi.org/10.1186/s40359-020-00443-z)
    
    

43. Barkham M, Bewick B, Mullin T, et al. The CORE-10: A short measure of psychological distress for routine use in the psychological therapies. Counselling and Psychotherapy Research 2013;13:3–13. [doi:10.1080/14733145.2012.729069](http://dx.doi.org/10.1080/14733145.2012.729069)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1080/14733145.2012.729069&link_type=DOI) 

44. Twigg E, Barkham M, Bewick BM, et al. The young person’s CORE: development of a brief outcome measure for young people. Counselling and Psychotherapy Research 2009;9:160–8. [doi:10.1080/14733140902979722](http://dx.doi.org/10.1080/14733140902979722)
    
    

45. Feixas G, Badia E, Bados A, et al. Adaptation and Psychometric properties of the Spanish version of the YP-CORE (young person’s clinical outcomes in routine evaluation). Actas Esp Psiquiatr 2018;46:75–82.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 

46. Stratton P, Bland J, Janes E, et al. Developing an indicator of family function and a practicable outcome measure for systemic family and couple therapy: the SCORE: systemic family and couple therapy. J Fam Ther 2010;32:232–58. [doi:10.1111/j.1467-6427.2010.00507.x](http://dx.doi.org/10.1111/j.1467-6427.2010.00507.x)
    
    

47. Rivas G, Pereira R. Validación de Una Escala de Evaluación familiar: Adaptación del SCORE-15 con Normas en Español. 2016.
    
    

48. Jovanović V. The validity of the satisfaction with life scale in adolescents and a comparison with single-item life satisfaction measures: a preliminary study. Qual Life Res 2016;25:3173–80. [doi:10.1007/s11136-016-1331-5](http://dx.doi.org/10.1007/s11136-016-1331-5)
    
    

49. Feixas G, Pucurull O, Roca C, et al. Escala de Satisfacción con El Tratamiento Recibido (CRES-4): La Versión en Español. Rdp 2012;23:51–8. [doi:10.33898/rdp.v23i89.639](http://dx.doi.org/10.33898/rdp.v23i89.639) Available: [https://revistas.uned.es/index.php/rdp/issue/view/1659](https://revistas.uned.es/index.php/rdp/issue/view/1659)
    
    

50. Barkham M, Mellor-Clark J, Stiles WB. A CORE approach to progress monitoring and feedback: enhancing evidence and improving practice. Psychotherapy (Chic) 2015;52:402–11. [doi:10.1037/pst0000030](http://dx.doi.org/10.1037/pst0000030)
    
    

51. Evans C, Connell J, Barkham M, et al. Practice-based evidence: Benchmarking NHS primary care counselling services at national and local levels. Clin Psychol Psychother 2003;10:374–88. [doi:10.1002/cpp.384](http://dx.doi.org/10.1002/cpp.384)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1002/cpp.384&link_type=DOI) 

52. 1.  Given LM
    
    Ayres L. Semi-structured interview. In: Given LM, ed. The SAGE Encyclopedia of Qualitative Research Methods. Thousand Oaks. SAGE Publications, 2008: 810–1.
    
    

53. MacKay RJ, Oldford RW. Scientific method, statistical method and the speed of light. Statist Sci 2000;15. [doi:10.1214/ss/1009212817](http://dx.doi.org/10.1214/ss/1009212817)
    
    

54. Spiegelhalter DJ. The art of statistics: learning from data. Published in paperback. UK USA Canada Ireland Australia India New Zealand South Africa. Pelican, an imprint of Penguin Books, 2020.
    
    

55. Posit team. Rstudio: integrated development environment for R. 2022. Available: [http://www.posit.co/](http://www.posit.co/)
    
    

56. Team RC. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing, 2021. Available: [https://www.R-project.org/](https://www.R-project.org/)
    
    

57. de Jong K, Moerbeek M, van der Leeden R. A Priori power analysis in longitudinal three-level Multilevel models: an example with therapist effects. Psychother Res 2010;20:273–84. [doi:10.1080/10503300903376320](http://dx.doi.org/10.1080/10503300903376320)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1080/10503300903376320&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19946814&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000277656900003&link_type=ISI) 

58. Muthén B. Latent variable analysis: growth mixture modeling and related techniques for longitudinal data. In: In: The SAGE Handbook of Quantitative Methodology for the Social Sciences. 2455 Teller Road, Thousand Oaks California 91320 United States of America. SAGE Publications, Inc, 2004: 346–69. [doi:10.4135/9781412986311](http://dx.doi.org/10.4135/9781412986311)
    
    

59. Diallo TMO, Morin AJS, Lu H. The impact of total and partial inclusion or exclusion of active and inactive time invariant covariates in growth mixture models. Psychol Methods 2017;22:166–90. [doi:10.1037/met0000084](http://dx.doi.org/10.1037/met0000084)
    
    

60. Berlin KS, Parra GR, Williams NA. An introduction to latent variable mixture modeling (part 2): longitudinal latent class growth analysis and growth mixture models. J Pediatr Psychol 2014;39:188–203. [doi:10.1093/jpepsy/jst085](http://dx.doi.org/10.1093/jpepsy/jst085)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1093/jpepsy/jst085&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=24277770&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000332344000007&link_type=ISI) 

61. Kim S-Y. Sample size requirements in Single- and Multiphase growth mixture models: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal 2012;19:457–76. [doi:10.1080/10705511.2012.687672](http://dx.doi.org/10.1080/10705511.2012.687672)
    
    

62. Diallo TMO, Morin AJS, Lu H. Performance of growth mixture models in the presence of time-varying covariates. Behav Res Methods 2017;49:1951–65. [doi:10.3758/s13428-016-0823-0](http://dx.doi.org/10.3758/s13428-016-0823-0)
    
    

63. Heggeseth BC, Jewell NP. How Gaussian mixture models might Miss detecting factors that impact growth patterns. Ann Appl Stat 2018;12. [doi:10.1214/17-AOAS1066](http://dx.doi.org/10.1214/17-AOAS1066)
    
    

64. Bauer DJ, Curran PJ. Distributional assumptions of growth mixture models: implications for Overextraction of latent trajectory classes. Psychol Methods 2003;8:338–63. [doi:10.1037/1082-989X.8.3.338](http://dx.doi.org/10.1037/1082-989X.8.3.338)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1037/1082-989X.8.3.338&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=14596495&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000185747000007&link_type=ISI) 

65. Jacobson NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology 1991;59:12–9. [doi:10.1037/0022-006X.59.1.12](http://dx.doi.org/10.1037/0022-006X.59.1.12)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1037/0022-006X.59.1.12&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=2002127&link_type=MED&atom=%2Fbmjopen%2F13%2F5%2Fe071875.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1991EV98700002&link_type=ISI) 

66. McAleavey AA. When (not) to rely on the reliable change index. Open Science Framework [Preprint] 2021. [doi:10.31219/osf.io/3kthg](http://dx.doi.org/10.31219/osf.io/3kthg)
    
    

67. Mazza GL, Enders CK, Ruehlman LS. Addressing item-level missing data: A comparison of proration and full information maximum likelihood estimation. Multivariate Behavioral Research 2015;50:504–19. [doi:10.1080/00273171.2015.1068157](http://dx.doi.org/10.1080/00273171.2015.1068157)
    
    

68. Wu W, Gu F, Fukui S. Combining proration and full information maximum likelihood in handling missing data in Likert scale items: A hybrid approach. Behav Res Methods 2022;54:922–40. [doi:10.3758/s13428-021-01671-w](http://dx.doi.org/10.3758/s13428-021-01671-w)
    
    

69. Mellor-clark J, Barkham M, Connell J, et al. Practice-based evidence and standardized evaluation: informing the design of the CORE system. European Journal of Psychotherapy & Counselling 1999;2:357–74. [doi:10.1080/13642539908400818](http://dx.doi.org/10.1080/13642539908400818)
    
    

70. Gale NK, Heath G, Cameron E, et al. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117. [doi:10.1186/1471-2288-13-117](http://dx.doi.org/10.1186/1471-2288-13-117)
    
    

71. Dogmanas D, Arias MA, Gago F, et al. Implementation of a psychological treatment outcomes evaluation system for young people at Uruguay’s national health system: using CORE‐OM. Couns and Psychother Res 2022;22:946–57. [doi:10.1002/capr.12579](http://dx.doi.org/10.1002/capr.12579)
    
    

72. Fernández-Alvarez H, Gómez B, García F. Bridging the gap between research and practice in a clinical and training network: Aigle’s program. Psychotherapy Research 2015;25:84–94. [doi:10.1080/10503307.2013.856047](http://dx.doi.org/10.1080/10503307.2013.856047)
    
    

73. Gómez B, Fernández-Álvarez J, García F. Implementing routine outcome monitoring in the treatment of a patient with Obsessive‐Compulsive disorder. J Clin Psychol 2022;78:2002–15. [doi:10.1002/jclp.23418](http://dx.doi.org/10.1002/jclp.23418)
    
    

74. Trujillo A, Paz C. Evidencia Basada en La Práctica en Psicoterapia: El Reto en Latinoamérica. CES Psico 2020;13:1–14. [doi:10.21615/cesp.13.3.1](http://dx.doi.org/10.21615/cesp.13.3.1)
    
    

75. Lutz W, Rubel JA, Schwartz B, et al. Towards integrating personalized feedback research into clinical practice: development of the Trier treatment navigator (TTN). Behav Res Ther 2019;120:103438. [doi:10.1016/j.brat.2019.103438](http://dx.doi.org/10.1016/j.brat.2019.103438)
    
    

76. Lutz W, Deisenhofer A-K, Rubel J, et al. Prospective evaluation of a clinical decision support system in psychological therapy. J Consult Clin Psychol 2022;90:90–106. [doi:10.1037/ccp0000642](http://dx.doi.org/10.1037/ccp0000642)