You are here

  • Home
  • Archive
  • Volume 2, Issue 5
  • Applying the RE-AIM framework to the Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions

Article Text

Article menu
Diabetes and endocrinology
Protocol
Applying the RE-AIM framework to the Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions
Free
  1. Lisa Wozniak1,
  2. Sandra Rees1,2,
  3. Allison Soprovich1,
  4. Fatima Al Sayah1,2,
  5. Steven T Johnson1,3,
  6. Sumit R Majumdar1,4,
  7. Jeffrey A Johnson1,2
  1. 1ACHORD, University of Alberta, Edmonton, Alberta, Canada
  2. 2Department of Public Health Sciences, University of Alberta, Edmonton, Alberta, Canada
  3. 3Center for Nursing and Health Studies, Athabasca University, Edmonton, Alberta, Canada
  4. 4Department of Medicine, University of Alberta, Edmonton, Alberta, Canada
  1. Correspondence to Jeffrey A Johnson; jeff.johnson{at}ualberta.ca

Abstract

Introduction Diabetes represents a major public health and health system burden. As part of the Alberta's Caring for Diabetes (ABCD) Project, two quality-improvement interventions are being piloted in four Primary Care Networks in Alberta. Gaps between health research, policy and practice have been documented and the need to evaluate the impact of public health interventions in real-world settings to inform decision-making and clinical practice is paramount. In this article, we describe the application of the RE-AIM framework to evaluate the interventions beyond effectiveness.

Methods and analysis Two quality-improvement interventions were implemented, based on previously proven effective models of care and are directed at improving the physical and mental health of patients with type-2 diabetes. Our goal is to adapt and apply the RE-AIM framework, using a mixed-methods approach, to understand the impact of the interventions to inform policy and clinical decision-making. We present the proposed measures, data sources and data management and analysis strategies used to evaluate the interventions by RE-AIM dimension.

Ethics and dissemination Ethics approval for the ABCD Project has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. The RE-AIM framework will be used to structure our dissemination activities by dimension.

Results It will be presented at relevant conferences and prepared for publication in peer-reviewed journals. Various products, such as presentations, briefing reports and webinars, will be developed to inform key stakeholders of the findings. Presentation of findings by RE-AIM dimension will facilitate discussion regarding the public health impact of the two interventions within the primary care context of Alberta and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. It will also promote the application of evaluation models to better assess the impact of community-based primary healthcare interventions through our dissemination activities.

  • Primary Care

https://doi.org/10.1136/bmjopen-2012-002099

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Article focus

    • Diabetes represents a public health and health system burden; primary care requires collaborative team interventions and self-management support.

    • Study protocol using a mixed-method approach for a comprehensive evaluation.

    Key messages

    • Evidence on effectiveness of public health interventions in real-world settings is needed to better inform decisions about practice and policy.

    • RE-AIM model provides a framework to elicit contextual information to better interpret the effectiveness of interventions.

    • Dissemination activities can be structured using the RE-AIM dimensions to identify target audiences, key messages and appropriate knowledge products.

    Strengths and limitations of this study

    • Application of the RE-AIM framework and a mixed-methods approach allows for a comprehensive evaluation of the ABCD Project interventions beyond clinical effectiveness.

    • This represents the evaluation of a pilot study in four Primary Care Networks (PCNs), which may not be representative of other primary care models.

    Introduction

    Diabetes represents a major public health and health system burden. The Canadian National Diabetes Surveillance System has estimated that 6.2% of the population has diabetes.1 In Alberta, 206 000 people were living with diabetes in 2009, representing over 5.5% of the population.2 This signifies a doubling of affected individuals within the past decade. The majority (ie, >90%) of these individuals have type 2 diabetes. As the number of people with diabetes increases, the number of resulting complications and co-morbidities increases, creating a greater demand on healthcare resources.2 ,3

    The Alberta's Caring for Diabetes (ABCD) Project, funded by the Alberta Health ministry as part of the provincial diabetes strategy, was developed to improve the quality and efficiency of care for diabetes in Alberta, Canada, with a focus on supporting Primary Care Networks (PCNs) in non-metro areas of Alberta. PCNs consist of a voluntary network of family physicians (hereby referred to as ‘member physicians’) and allied health professionals, who identify priorities and coordinate health services for patient populations.4 ,5 The PCN model is akin to the ‘patient-centered medical home’ model emerging in the USA.6 ,7

    The ABCD team has worked with participating PCNs to implement a number of quality improvement interventions. This includes an ongoing, survey-based cohort study that seeks to understand why some people with type 2 diabetes develop complications while others do not. This study involves an annual survey of individuals with type 2 diabetes over 5 years, to collect data on lifestyle behaviours, self-management and patient-reported outcomes and linkage with administrative databases to assess healthcare utilisation and longer-term clinical outcomes. In addition, participating PCNs will implement pilot interventions including: (1) Healthy Eating and Active Living in Diabetes (HEALD-PCN), a pedometer-based walking programme;8 and (2) TeamCare-PCN, a collaborative team-based, depression case management intervention.9 Key features of HEALD-PCN include the provision of information in a group setting by an exercise specialist on increasing the amount and intensity of physical activity (ie, walking), the glycemic index and individual goal setting. The HEALD-PCN programme also provides opportunities for participants to implement lessons learned (ie, walking group sessions) through partnerships with community recreational facilities.10 Key features of TeamCare-PCN include coordinated care by a nurse care manager to direct active patient follow-up, treat-to-target principles and specialist (ie, psychiatrists and internists/endocrinologists) consultation.11

    The efficacy of both pilot interventions has been proven in other settings,10–12 and the study protocols to determine the effectiveness of HEALD-PCN and TeamCare-PCN in the PCN environment in Alberta have been published.8 ,9 Our goal is to also assess the impact of the entire ABCD project activities, including how these different interventions were simultaneously implemented, in Alberta's PCN environment. The purpose of this paper is to describe the design of the evaluation for the different elements of the ABCD project, using the RE-AIM framework.13

    Evaluating the ABCD pilot interventions using RE-AIM

    The gaps between health research, policy and practice have been well documented.13–16 Evaluations of health interventions are often limited to efficacy studies rather than assessment of potential public health impact.17 Efficacy studies tend to focus on the internal validity of high-intensity health interventions with motivated and homogeneous populations in controlled settings.13 This narrow focus hinders the translation of research into practice and reduces the ability to generalise findings to similar settings.13 Evidence on the external validity of less-intensive interventions in real-world settings is needed to better inform decisions about practice.13

    In this context, assessment of clinical effectiveness alone is not enough to inform decisions about a programme's broader public health impact. The RE-AIM evaluation framework was designed to assess health interventions beyond effectiveness to include multiple criteria to better identify effect and transferability.18 The framework consists of five dimensions: Reach into the target population; Effectiveness of the intervention; Adoption by target settings, institutions and staff; Implementation, including consistency and cost of delivery; and Maintenance of intervention effects over time.18

    The RE-AIM model addresses two levels of assessment: individual (Reach, Effectiveness); organisation (Adoption and Implementation) or both (Maintenance).13 To fit our evaluation goals, we expanded the assessment level of ‘Reach’13 beyond the individual assessment level (ie, absolute number, proportion and representativeness of individuals willing to participate in an intervention) to include an organisation assessment level (ie, an organisation's ability to identify the entire target population) (table 1). An example of an organisational strategy to identify a population is the development and use of a patient registry.

    View this table:
    Table 1

    RE-AIM dimensions, definitions and assessment levels for evaluation of the ABCD pilot interventions

    In table 1, we provide the original definitions for each RE-AIM dimension. Italicised words or phrases indicate modifications made by the ABCD Project team to the original ‘Reach’ definition and assessment level.13 This table was compiled and adapted from several sources.13 ,17 ,18

    While there are other evaluation frameworks, such as Procede-Proceed19 and Health Impact Assessment,20 we assert the RE-AIM model is well suited to evaluate the ABCD pilot interventions for two reasons. First, RE-AIM is considered more appropriate for evaluation of behavioural change interventions21 than other models. Second, the dimensions of the RE-AIM model are well matched to inform the specific needs of our audiences and interested parties including healthcare providers, PCN management, policy makers and funders.

    Methods and analysis

    We will employ a mixed-methods approach22 for our comprehensive evaluation of the ABCD pilot interventions. Using the RE-AIM model, our research team developed logic models and data matrices for both interventions in consultation with advisory committees (see online supplementary Appendix 1; web only file). The overarching questions guiding the evaluation for each intervention are: (1) Is the service delivery model effective in the context of Alberta's primary care setting and (2) What factors contribute to the effectiveness (or ineffectiveness) of the intervention? The more specific evaluation questions related to the RE-AIM framework that will direct the collection and analysis of data for both interventions include:

    1. Reach: Is the intervention reaching the intended target population?

    2. Adoption: Has the intervention been adopted by the PCNs and staff?

    3. Implementation: Is the intervention being implemented as intended? Is it cost-effective?

    4. Effectiveness: What are the immediate, intermediate and long-term impacts of the intervention?

    5. Maintenance: Is the intervention sustainable in a cost-effective way?

    Measurement by RE-AIM dimensions

    In the following section, we outline the measures proposed for each dimension of RE-AIM to evaluate the ABCD project interventions. A detailed summary is provided in table 2.

    View this table:
    Table 2

    Measures, data sources and data collection timeline by RE-AIM dimension and assessment level

    Reach

    Evaluation of reach will be done at the individual (patient) and organisation (PCN) assessment levels to determine if the ABCD pilot interventions are reaching those in most need. At the individual assessment level, we will examine total recruitment into the interventions and usual care groups and compare their characteristics with respect to eligibility criteria, demographic information and other measures. As possible, we will compare characteristics between participants (ie, intervention and usual care groups) and non-participants using aggregate demographic information accessed through PCN patient registries and Alberta Diabetes Surveillance System data.23 Facilitators and barriers to individual patient recruitment and suggestions for improvement will be identified through interviews with PCN staff.

    At the organisation assessment level, we will document usual care in the PCNs, including the ability to estimate and identify target patient populations in the focus areas (ie, type 2 diabetes management, depression management and lifestyle counseling) through completion of a standardised checklist. We will examine processes related to registry development and identify facilitators and barriers related to development, use and maintenance through interviews with PCN staff. In addition, we will elicit recommendations related to the PCNs’ ability to identify patient populations to actively offer targeted health services.

    Effectiveness

    Evaluation of effectiveness will be conducted at the individual assessment level to determine impact of the pilot interventions on important outcomes. The design and rationale for controlled evaluations of the effectiveness of the two ABCD pilot interventions have been described elsewhere.8 ,9 The primary outcome of HEALD-PCN is improvement in physical activity (ie, brisk walking), determined by step pedometers and self-report.8 For TeamCare-PCN, the primary outcome is improvement of depressive symptoms as measured by the Patient Health Questionnaire-9 (PHQ-9) items.9 We will also use a variety of measures to determine the effectiveness of both interventions on important outcomes at the individual assessment level including clinical measures (eg, improvements in glycemic control, blood pressure, lipid measurements and body mass index), self-reported health-related quality of life,24–26 self-efficacy,27 satisfaction with care,28 ,29 and process indicators. In addition, we will document unanticipated consequences (positive or negative), such as improved patient linkages with community health resources, to provide a richer understanding of effectiveness. Additional measures and data sources to assess effectiveness are provided in table 2.

    Adoption

    We will assess the adoption of the ABCD pilot interventions at the organisation level, including documentation of the criteria for PCN selection and participation in the ABCD Project and PCN Board approval. Also, we will document and compare the characteristics of the participating PCNs (eg, number of family physicians, number of patients served and governance structure) as well as usual care in the focus areas. Dependent on availability of secondary data, we will consider the representativeness of participating PCNs compared with non-participating PCNs. This will be accomplished through document review (eg, ABCD project documents, PCN websites, business plans), use of a standardised usual care checklist and interviews with PCN staff. In addition, perceptions related to the extent to which the ABCD pilot interventions have been adopted by the PCNs and modified to suit their contexts will be elicited through interviews with PCN staff. Identified facilitators and barriers to adoption of the interventions along with creative solutions or modifications will also be documented.

    Implementation

    Evaluation of implementation of the ABCD pilot interventions will be done at the individual and organisation assessment levels to determine patient adherence, consistency of implementation and costs of delivering the pilot interventions. To address implementation at an individual assessment level, participant adherence to the intervention models will be determined for both interventions. For HEALD-PCN, attendance at group sessions, participant step logs (ie, recording the number of steps over 3 days) and self-reported physical activity will be assessed. For TeamCare-PCN, adherence to treatment plans, including medication and behavioural modifications (eg, engaging in planned pleasant activities), will be assessed. These types of data will be derived from patient outcome tracking systems employed in each PCN and/or survey items.

    At the organisation assessment level, consistency of implementation and the cost of delivering the ABCD pilot interventions will be evaluated to determine the practicality of the interventions. Actual versus intended implementation will be assessed through extensive documentation including development of project materials (eg, training and resource materials), presence of systems and processes (eg, patient registries), intervention staff recruited or hired by PCNs and provision and quality of training in the intervention models. Additional measures and data sources to assess consistent implementation are provided in table 2. Our implementation assessment will also include economic evaluations of the ABCD pilot interventions, which have been described in detail elsewhere.8 ,9

    Maintenance

    For both ABCD pilot interventions, maintenance will be evaluated at the individual and organisation assessment levels to measure continuation of intervention effects over time. We will use a previously developed conceptual framework that defines sustainability outcomes of health interventions.30 At the individual level, maintenance will be evaluated based on patient-reported health behaviours and self-care collected annually through the ABCD cohort study survey and interviews with a subsample of HEALD-PCN intervention group participants at 6-months post-intervention.

    At the organisation assessment level, interviews with PCN staff will be conducted post-interventions to assess integration of intervention model components into practice (eg, continued use of patient registries or screening tools), enhanced organisational capacity (eg, maintaining partnerships) and continued focus on the interplay between diabetes, depression and lifestyle (eg, incorporation of the intervention models into future business plans). In addition, interviews with specialists participating in TeamCare-PCN will be conducted with a focus on sustainability of the model in the current primary care environment, including appropriate compensation and funding approaches and potential medicolegal liability issues.

    Data management

    Our comprehensive evaluation will involve the collection and management of a wide range and large volume of data. Primary data sources for the evaluation of the ABCD pilot interventions include: (1) clinical outcome measures; (2) patient-reported outcomes; (3) interviews (eg, with PCN staff, HEALD-PCN intervention group participants and specialists for TeamCare-PCN); (4) document review (eg, usual care checklists, project documents and field notes) and (5) administrative healthcare datasets.

    Clinical outcomes and survey data captured in the patient outcome tracking systems or standardised case forms used in each PCN will be entered into centralised, web-accessible databases. These study databases will be housed on secure servers in the research offices at the University of Alberta. Once the pilot interventions are completed, all data will be exported and merged, based on individually assigned study ID numbers, to form an analysable dataset. Investigators, research assistants and analysts will be masked to allocation status at all times.

    Semistructured interviews will take place at the PCN offices of the interviewees. Interviews with HEALD-PCN intervention group participants and TeamCare-PCN specialists will be conducted via telephone. Interviews will be facilitated through the use of interview guides. Interviews will be digitally recorded for subsequent analysis, transcribed verbatim by an independent transcriptionist and verified for accuracy.

    Regarding document review, we will develop a standardised usual care checklist by adapting themes from the Change Process Capability Questionnaire31 and the Organisational Readiness to Change Scale32 to be validated by staff of the participating PCNs. Topic areas include: usual care for people with type-2 diabetes; existing PCN diabetes, depression and lifestyle programming and organisational factors and strategies related to PCN patient care. Also, we will document how the ABCD pilot interventions unfolded in each PCN through field notes, communications and meeting minutes. All qualitative data sources, including interview transcripts and documents, will be compiled and managed using Nvivo V. 9.0 software.

    Patients enrolled in the pilot interventions and the ABCD Cohort study will be asked for permission to access their medical records by providing their personal health number, thus allowing linkage to provincial healthcare administrative data from Alberta Health for physician, hospital, and emergency department billing and pharmaceutical data (for patients 65 years and older). This linkage will allow healthcare utilisation and healthcare costs to be included in the evaluation.

    Data analysis

    We are undertaking a broad mixed-methods approach to analysis. In terms of quantitative data, the approach to power, sample size calculations, assessment and statistical modelling of clinical effectiveness have been previously detailed.8 ,9 In terms of qualitative data, we will take a general inductive approach33 with the evaluation questions related to the RE-AIM framework directing the analysis of data. Findings will be derived directly through a content analysis34 of the raw data without preconceived notions about specific findings.

    Ethics and dissemination

    Ethical considerations

    Ethics approval for the entire ABCD Project and its associated interventions has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. However, the Board deemed this component of the ABCD Project as evaluation and not research; therefore, it did not require ethics review and approval. Regardless,  the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans35 will be followed.

    Discussion and dissemination

    The ABCD Project was developed to improve the quality and efficiency of diabetes care in non-metro Alberta. In order to address the gap between research, policy and practice, we have adapted and expanded the RE-AIM model to conduct a comprehensive evaluation of the ABCD pilot interventions. This will contribute to our knowledge of the broader impact of the two interventions within the evolving primary care context of Alberta beyond effectiveness, as outlined in the study trial designs.8 ,9 The purpose of this article was to present the proposed measures and data sources to be used to evaluate the interventions by RE-AIM dimension. Using the RE-AIM evaluation framework will allow us to systematically identify facilitators, challenges, opportunities and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. In addition, our application of the RE-AIM evaluation framework may encourage others to use similar models to determine the impact of community-based primary-healthcare interventions. The RE-AIM model will also be used to structure our dissemination activities. For example, each RE-AIM dimension will inform the development of products (such as academic manuscripts for peer-review publication, presentations at relevant conferences and workshops, and briefing reports) and identification of relevant target audiences.

    Acknowledgments

    JAJ is a Senior Scholar with Alberta Innovates-Health Solutions and a Centennial Professor at the University of Alberta. SRM is a Health Scholar funded by Alberta Innovates-Health Solutions and holds the Endowed Chair in Patient Health Management funded by the Faculties of Medicine and Dentistry and Pharmacy and Pharmaceutical Sciences of the University of Alberta. This work was supported in part by a contract from Alberta Health, and a CIHR Team Grant to the Alliance for Canadian Health Outcomes Research in Diabetes (reference #: OTG- 88588), sponsored by the CIHR Institute of Nutrition, Metabolism and Diabetes (INMD).

    References

    1. ↵
      Public Health Agency of Canada. Report from the National Diabetes Surveillance System: diabetes in Canada, 2009. (Ottawa): Public Health Agency of Canada, 2009.
    2. ↵
      1. Johnson JA,
      2. Vermeulen SU
      . Diabetes trends in Alberta. ADSS Newsl 2011;5:1.
    3. ↵
      1. Johnson JA,
      2. Balko SU
      . Chapter 2: Epidemiological Trends of Diabetes in Alberta. In Johnson JA, editor; Alberta Diabetes Atlas 2009. Edmonton: Institute of Health Economics, 2009:11–31.
    4. ↵
      Primary Care Initiative. About PCI. http://www.albertapci.ca/AboutPCI/Pages/default.aspx (accessed 19 Oct 2012).
    5. ↵
      R.A. Malatest & Associates Ltd. Primary Care Initiative Evaluation: Final Formative Evaluation Report. Edmonton, AB, 2010.
    6. ↵
      1. Rittenhouse DR,
      2. Shortell SM
      . The patient-centered medical home will it stand the test of health reform? JAMA 2009;301:2038–40.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Reid RJ,
      2. Coleman K,
      3. Johnson EA,
      4. et al
      . The group health medical home at year two: cost savings, higher patient satisfaction, and less burnout for providers. Health Aff (Millwood) 2010;29:835–43. doi:10.1377/Hlthaff.2010.0158.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Johnson ST,
      2. Mundt C,
      3. Soprovich A,
      4. et al
      . Healthy eating and active living for diabetes in primary care networks (HEALD-PCN): rationale, design, and evaluation of a pragmatic controlled trial for adults with type 2 diabetes. BMC Public Health 2012;12. doi: 10.1186/1471-2458-12455 (published Online First: Epub Date)
    9. ↵
      1. Johnson JA,
      2. Al Sayah F,
      3. Wozniak L,
      4. et al
      . Controlled trial of a collaborative primary care team model for patients with diabetes and depression: rationale and design for a comprehensive evaluation. BMC Health Serv Res 2012;12. doi:10.1186/1472-6963-12-258 (published Online First: Epub Date).
    10. ↵
      1. Johnson ST,
      2. Bell GJ,
      3. McCargar LJ,
      4. et al
      . Improved cardiovascular health following a progressive walking and dietary intervention for type 2 diabetes. Diabetes, Obes Metab 2009;11:836–43. doi:10.1111/j.1463-1326.2009.01050.x
      OpenUrlPubMedWeb of Science
    11. ↵
      1. Katon WJ,
      2. Lin EHB,
      3. Von Korff M,
      4. et al
      . Collaborative care for patients with depression and chronic illnesses. N Engl J Med 2010;363:2611–20. doi:10.1056/NEJMoa1003955
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. McGregor M,
      2. Lin EHB,
      3. Katon WJ
      . TEAMcare: an integrated multicondition collaborative care program for chronic illnesses and depression. J Ambul Care Manage 2011;34:152–62. doi:10.1097/JAC.0b013e31820ef6a4
      OpenUrlPubMed
    13. ↵
      1. Glasgow RE,
      2. Vogt TM,
      3. Boles SM
      . Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:1322. doi:10.2105/AJPH.89.9.1322
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Glasgow RE,
      2. Lichtenstein E,
      3. Marcus AC
      . Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003;93:1261. doi:10.2105/AJPH.93.8.1261
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Tunis SR,
      2. Stryer DB,
      3. Clancy CM
      . Practical clinical trials—increasing the value of clinical research for decision making in clinical and health policy. Jama-J Am Med Assoc 2003;290:1624–32.
      OpenUrl
    16. ↵
      1. Treweek S,
      2. Zwarenstein M
      . Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials 2009;10:37. doi:10.1186/1745–6215–10–37
      OpenUrlCrossRefPubMed
    17. ↵
      1. Glasgow R
      . The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 2001;44:119–27. doi:10.1016/S0738-3991(00)00186-5
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      National Cancer Institute. Implementation Science: Integrating Science, Practice, and Policy—What is RE-AIM? Secondary Implementation Science: Integrating Science, Practice, and Policy—What is RE-AIM? July 31 2011. http://cancercontrol.cancer.gov/IS/reaim/whatisre-aim.html
    19. ↵
      1. Green LW,
      2. Kreuter MW
      . Health promotion planning: an educational and ecological approach. 3rd edn. Mountain View, CA: Mayfield Pub. Co., 1999.
    20. ↵
      1. Scott-Samuel A,
      2. Birley M,
      3. Ardern K
      . The Meyerside guidelines for health impact assessment. 2nd edn. Liverpool, England: The International Health IMPACT Assessment Consortium, 2001.
    21. ↵
      1. Farris RP,
      2. Will JC,
      3. Khavjou O,
      4. et al
      . Beyond effectiveness: evaluating the public health impact of the WISEWOMAN program. Am J Public Health 2007;97:641–47. doi:10.2105/Ajph.2005.072264
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Johnson RB,
      2. Onwuegbuzie AJ
      . Mixed methods research: a research paradigm whose time has come. Educ Res 2004;33:14–26.
      OpenUrl
    23. ↵
      Institute of Health Economics, Alliance for Canadian Health Outcomes Research in Diabetes. ADSS Interactive Website. Secondary ADSS Interactive Website 2011. http://www.interactiveadss.ualberta.ca/
    24. ↵
      1. Ware JE,
      2. Kosinski M,
      3. Keller SD
      . A 12 item short form health survey: construction of scales and preliminary tests of reliability and validity. Med Care 1996;34:220–33.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Kind P
      . The EuroQoL instrument: an index of health-related quality of life. In: Spilker B, ed. Quality of life and pharmacoeconomics in clinical trials. 2nd edn. Philadelphia: Lippincoot-Raven Publishers, 1996:191–201.
    26. ↵
      1. McGuire BE,
      2. Morrison TG,
      3. Hermanns N,
      4. et al
      . Short-form measures of diabetes-related emotional distress: the problem areas in diabetes scale (PAID)-5 and PAID-1. Diabetologia 2010;53:66–9. doi:10.1007/s00125-009-1559-5
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Lorig KR,
      2. Ritter P,
      3. Stewart AL,
      4. et al
      . Chronic disease self-management program: 2-year health status and health care utilization outcomes. Med Care 2001;39:1217–23.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Gugiu PC,
      2. Coryn C,
      3. Clark R,
      4. et al
      . Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument. Chronic Illn 2009;5:268–76. doi:10.1177/1742395309348072
      OpenUrlAbstract/FREE Full Text
    29. ↵
      Agency for Healthcare Research and Quality, United States Department of Health and Human Services.Adult commerical questionnaire. AHRQ pub. Rockville, MD: AHRQ, Office of Health Care Information 2011:v.
    30. ↵
      1. Scheirer MA,
      2. Hartling G,
      3. Hagerman D
      . Defining sustainability outcomes of health programs: illustrations from an on-line survey. Eval Program Plann 2008;31:335–46. doi:10.1016/j.evalprogplan.2008.08.004
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Solberg LI,
      2. Asche SE,
      3. Margolis KL,
      4. et al
      . Measuring an organization's ability to manage change: the change process capability questionnaire and its use for improving depression care. Am J Med Qual 2008;23:193–200. doi:10.1177/1062860608314942
      OpenUrlAbstract/FREE Full Text
    32. ↵
      1. Helfrich CD,
      2. Li YF,
      3. Sharp ND,
      4. et al
      . Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci 2009;4:38–1. doi:10.1186/1748–5908-4-38
      OpenUrlCrossRefPubMed
    33. ↵
      1. Thomas DR
      . A general inductive approach for analyzing qualitative evaluation data. Am J Eval 2006;27:237–46. doi:10.1177/1098214005283748
      OpenUrlCrossRefWeb of Science
    34. ↵
      1. Mayan MJ
      . Essentials of qualitative inquiry. Walnut Creek, CA: Left Coast Press, 2009.
    35. ↵
      Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans.
    36. ↵
      Alliance for Canadian Health Outcomes Research in Diabetes. ACHORD: Alliance for Canadian Health Outcomes Research in Diabetes. Secondary ACHORD: Alliance for Canadian Health Outcomes Research in Diabetes 2012. http://www.achord.ca

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors LW developed the evaluation framework for the Alberta's Caring for Diabetes (ABCD) Project and drafted the manuscript. SR, AS, STJ actively contributed to the development of the evaluation framework and critically reviewed and revised the manuscript. FA provided feedback on the manuscript. SRM provided expert feedback on the study design and critically reviewed the manuscript. JAJ conceived of the study, participated in its design and helped in drafting the manuscript. All authors read and approved the final manuscript.

    • Funding This work was supported in part by a contract from the provincial government (ie, Alberta Health), a grant from the Lawson Foundation (reference #: GRT 2010-028) (www.lawson.ca) and a Canadian Institutes for Health Research (CIHR) Team Grant (reference #: OTG- 88588) to the Alliance for Canadian Health Outcomes Research in Diabetes36 sponsored by the CIHR Institute of Nutrition, Metabolism and Diabetes. The funding sources had no role in the design of the studies or evaluation and will have no role in the conduct, analysis or reporting of the studies or evaluation, nor access to the data.

    • Competing interests None.

    • Ethics approval Ethics approval for the entire ABCD Project and its associated interventions has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. However, the Board deemed this component of the ABCD Project as evaluation and not research; therefore, it did not require ethics review and approval. Regardless, the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans will be followed.

    • Provenance and peer review Not commissioned; internally peer reviewed.