Article Text
PDF
Abstract
Introduction Dementia is one of the most relevant widespread diseases, with a prevalence of currently 55 million people with dementia worldwide. However, about 60–75% of people with dementia have not yet received a formal diagnosis. Asymptomatic screening of cognitive impairments using neuropsychiatric tests has been proven to efficiently enhance diagnosis rates. Digital screening tools, in particular, provide the advantage of being accessible without spatial or time restrictions. The study aims to validate a digital cognitive screening test (digiDEM-SCREEN) as an app in the German language.
Methods and analysis This is a multicentre study in Bavaria. Participants are people with mild cognitive impairment, people with dementia in an early stage and cognitively healthy people. Recruitment will take place in specialised diagnostic facilities (memory outpatient clinics). 135 participants are aimed based on a power analysis. Sociodemographic data, diagnosis and results of neuropsychiatric tests (Consortium to Establish a Registry for Alzheimer’s Disease, Montreal Cognitive Assessment, digiDEM-SCREEN) will be collected at one point per person via electronic data capturing. The sensitivity, specificity and corresponding cut-off values will be determined based on receiver-operating-characteristic curves. The correlation of the digiDEM-SCREEN test with existing cognitive screening/testing procedures will be analysed.
Ethics and dissemination The study obtained ethical approval from the Ethics Committee of the Julius-Maximilians-Universität of Würzburg (JMU) (application number: 177/23-sc). The test will give feedback about the current cognitive status and possible cognitive impairments that should lead to the users seeking further diagnostic measures by medical professionals. It will be accessible free of charge in established app stores. The results of the validation study will be published in peer-reviewed journals.
- Chronic Disease
- Cognition
- Dementia
- eHealth
- GERIATRIC MEDICINE
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
Participants are recruited based on diagnoses made by specialists in memory outpatient clinics. Diagnostic procedures are based on the actual national dementia guideline.
Including the different degrees of cognitive impairment severity (mild cognitive impairment, mild dementia) allows for a more sensitive analysis of psychometric values.
Potential end users were included in the development process of the digital screening test as an app in order to improve usability and avoid technical or linguistic barriers.
This is a multicentre study; however, there is no random sampling, so a selection bias of people with specific care backgrounds cannot be ruled out.
There is a residual risk that the test cannot be carried out properly. If the test cannot be carried out due to technical or language barriers, the user is advised to visit a specialised clinician.
Introduction
There are currently more than 55 million people with dementia worldwide, with forecasts predicting an increase to 150 million people with dementia by 2050.1 In Bavaria alone, there are currently more than 270 000 people with dementia, a number that is expected to rise to 380 000 by 2040.2 The annual healthcare costs for dementia amount to US$ 1.3 trillion worldwide.1 In Germany, the total cost of people with dementia for payers in 2016 was €34 billion;3 more recent calculations assume total annual costs of €32.6 billion.4 The future care and support of these people represent one of the most significant challenges not only for the healthcare system but also for society. This applies in particular to the care of people with dementia in rural areas.
When experiencing symptoms of dementia, a timely diagnosis is crucial in order to prepare for organisational challenges and initiate therapeutic measures that can slow down the disease progression. However, particular care challenges are already evident in the stage of the diagnostic process. The rate of undiagnosed cases of dementia is high. A study by Eichler et al found that 60% of people with dementia in Germany had no formal diagnosis,5 and worldwide it is estimated that as many as 75% of dementia cases are undiagnosed.6 There are vast differences in dementia detection between high-, middle- and low-income countries.6 7 Another problem lies in the long diagnosis periods. In the prospective, multicentre longitudinal study, Bavarian Dementia Survey, Wolff et al found that the median time between the first perceived symptoms and diagnosis in Bavaria was 16 months.8 In international studies, this discrepancy is even more pronounced, which is in particular true for people with young-onset dementia.9–12 As Barth et al were able to show, rural areas are also particularly affected here due to more difficult access to prompt assessment and diagnosis for people with dementia.9
Asymptomatic screening of cognitive impairments using neuropsychiatric tests is essential for improving the diagnosis rate. A study with 146 participants in Mecklenburg-Western Pomerania has shown that diagnoses could be increased by almost 50% through upstream cognitive screening.5 Digital screening tools offer the advantage that they can be used at a low threshold, regardless of time and place. That is particularly true for mobile applications, which can be accessed via a smartphone or tablet.
A recent systematic review of the diagnostic performance of digital cognitive tests for identifying MCI and dementia identified 46 different digital cognitive tests.13 22 of those digital cognitive tests showed a good diagnostic performance for dementia, with a sensitivity and a specificity of over 0.80, proving that digital cognitive screening can provide valuable results in detecting cognitive impairments. Among others, the Self-Administered Tasks Uncovering Risk of Neurodegeneration (SATURN)14 proved to be a test with particular promising diagnostic values (sensitivity: 0.92; specificity: 0.88 in dementia cases vs controls).13 The test is usable via a tablet. Administration time is about 10 min, which can be especially beneficial for older adults as shorter tests might induce less fatigue and therefore be more suitable for repeat administration compared with lengthier instruments.15 Thus, SATURN provides the foundation for the development and validation of a German adaption of the test usable as an app via a smartphone and tablet.
The present development and validation of the digital screening tool for the identification of cognitive impairments (digiDEM-SCREEN) are part of the research project Digital Dementia Register Bavaria—digiDEM Bayern, which is funded by the Bavarian State Ministry of Health, Care and Prevention (funding reference: G42d-G8300-2017/1606-83). digiDEM Bayern aims to improve the living conditions of people with mild cognitive impairments (MCIs) and dementia and their family caregivers in Bavaria, especially in rural areas. A detailed description of the aims, methodology and the technical infrastructure is provided in a study protocol published in BMJ Open in 2021.16
Objectives
The aim of the study is the validation of a digital cognitive screening test as an app. The digiDEM-SCREEN app was developed in 2023 in cooperation with the Institute for Medical Data Sciences at the University Hospital of Würzburg and the Institute for Clinical Epidemiology and Biometry at the University of Würzburg as an app for smartphones or tablets. As mentioned before, the development of the screening app is based on an already existing and scientifically analysed screening test, SATURN. SATURN is a 30-item test that includes orientation, word recall and maths tasks adapted from the Saint-Louis University Mental Status Test, as well as modified versions of the Stroop and Trail Making tests and other assessments of visuospatial function and memory. The test was designed as a tablet application and validated in a scientific study with 75 participants.14
The existing test was to be further developed for a German-speaking population. There is currently no German-language version of the SATURN test. The translation of the English version into German was carried out independently by two researchers from the digiDEM Bayern project using the translate-retranslate method and then discussed in the research group. As the next step, the German version will be validated in a German-speaking study population. To this end, the test will be administered to patients from various memory outpatient clinics and then evaluated for its sensitivity and specificity against the background of existing diagnoses and other non-digital cognitive tests. Cut-off values for categorising current cognitive abilities will also be determined as part of the validation.
The finished screening test is to be made available to all citizens free of charge. It thus represents a low-threshold alternative to existing commercial digital screening tools or those currently under development. End users will ultimately perform the screening test in an unsupervised, remote setting, like the home environment.
The app can thus contribute to improving the diagnosis rate on two levels. On the one hand, it creates a low-threshold opportunity to detect possible memory impairments, which can be the impetus for a subsequent diagnostic process. By doing so, the app can provide an initial assessment of possible cognitive impairments, particularly for people in structurally weaker rural regions. Given the high number of undiagnosed cases of dementia, the app is intended to reach this target group in particular. In addition, the app’s low-threshold access is also intended to help raise awareness by sensitising people who do not receive a conspicuous test result.
The primary aim of the validation study is to analyse the sensitivity and specificity of the screening test for identifying cognitive impairments and the corresponding cut-off values. The correlation with existing cognitive test procedures will be examined as a secondary objective.
Methods
Study design
This is a multicentre study with several memory outpatient clinics in Bavaria, in which the results of all test procedures are measured at one point in time. The memory outpatient clinics planned for this study are facilities already contractually involved as recruitment centres in the digiDEM Bayern project. digiDEM Bayern is a multicentre, prospective, longitudinal register study that will be conducted in all administrative regions of Bavaria. The methodology is described elsewhere.16 The recruitment period for the study participants is scheduled for 6 months. After checking the inclusion and exclusion criteria and obtaining written consent, the participant’s data will be collected at one point per participant. Before the start of the study, the employees of the memory outpatient clinics are introduced to the use of the app and the data collection and documentation systems. Participants will perform the screening test without supervision. However, the employees of the memory outpatient clinics will be allowed to aid participants with starting the tablet and opening the app. Each participant completes the Montreal Cognitive Assessment (MoCA) test as well as the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD) test battery and the digiDEM-SCREEN test in a single session. In order to minimise possible biases due to similarities in the cognitive test procedures, the sequence in which the individual tests are conducted is varied for the participants. The randomisation process is independent of the test results and target group membership. This minimises a possible learning effect that could influence the test results due to a fixed sequence of test procedures (see figure 1).
Crossover sequence testing. CERAD, Consortium to Establish a Registry for Alzheimer’s Disease; MoCA, Montreal Cognitive Assessment.
The start of the study was in August 2023, with the conceptualisation of the project and the ethics application. Participant recruitment started in April 2024. The planned end of the data collection and analysis is by the end of July 2025.
Hypothesis
Primary hypothesis
The screening test has a specificity of at least 0.90 with a sensitivity of at least 0.5.
CERAD test battery,17 which is part of the diagnostic process of memory outpatient clinics, is considered the gold standard for cognitive test procedures in dementia diagnostics in Germany.18
Secondary hypothesis
MoCA19 is considered the gold standard in the national and international clinical and research context to screen cognitive impairment.
digiDEM-SCREEN shows a high correlation (Pearson r at least 0.7) with the existing cognitive test procedure MoCA. This test for the hypothesis r>0.3 (r=0.3 as an irrelevant parameter) can be performed with a sample size of n=42 with a statistical power of 95%. A cut-off of r>0.3 reflects the effect size interpretation of John Hattie, who defined every value >0.3 as being in the zone of desired effects.20
Study population/participants
The following target groups are intended to participate in the study:
Cognitively healthy people
People with MCI
People with dementia
The identification of the target group in the study is based on the diagnoses in the memory outpatient clinics. Diagnoses are based on the German national guideline for dementia.18 Due to the high level of expertise and the availability of modern diagnostic procedures (medical history, cognitive testing, education, laboratory diagnostics), the memory outpatient clinics are able to differentiate between the individual clinical pictures as best as possible.
The inclusion and exclusion criteria are shown in table 1.
Inclusion and exclusion criteria
Power analysis
The power analysis resulted in 135 participants required to investigate the study’s hypothesis.
Primary endpoints are the sensitivity and specificity of the digiDEM-SCREEN, with the test’s potential for error being weighted differently: a false-positive result puts the user under stress, even if they know that the test result merely suggests further investigations. A false-negative result, on the other hand, lulls the user into a sense of security that may prevent the early utilisation of therapeutic measures and thus contribute to unchecked progression. For these reasons, particular emphasis is placed on the case number estimate on a high specificity (0.9), which can be reliably achieved by the test development study. The sensitivity as an indication for further diagnostics, which can be reliably achieved by the specified case numbers, is set at a lower level of 0.5. This does not mean that the app only achieves a sensitivity of 0.5, but that it operates with at least this sensitivity when validated on 135 people. Repeating the test using the app is possible at any time with a low threshold, which increases the sensitivity for monitoring screening.
The predictive values of the test are particularly relevant from an ethical point of view and the patient’s perspective. Notably, these also depend on the actual prevalence of the disease in the sample analysed. Various scenarios were therefore illustrated in the appendix to show how the required 90% specificity with minimum sensitivities of 50% and 70% affect the predictive values in a high-prevalence sample, a low-prevalence sample and in a 50-50 comparison of 1:1 age-matched ill/healthy people. Particularly in the low-prevalence scenario, the negative predictive value (ie, the certainty of a diagnosis of exclusion) is very good at 87% and 91% respectively.
The range of 95% CIs around the calculated specificity requires a total of 188 persons in the sample with a precision of 0.10. This case number requirement drops to n=88 if a precision of 0.20 (width of the CI) is tolerated. These refer to the tabulation in Bujang and Adnan.21 With a sample size of 135 test subjects, which can be recruited as quickly as expected, satisfactory results on the usefulness of the app are therefore possible from a practical point of view. Further information concerning the clarification and justification regarding sample size and prevalence can be found in the online supplemental file 1 and online supplemental file 2.
Supplemental material
Supplemental material
Power analysis was calculated using the specific software PASS (https://www.ncss.com/)
Study recruitment
The choice of memory outpatient clinics as recruitment centres is based on their structure and role in diagnosing dementia. The service description, quality requirements, data protection and other framework conditions of the research cooperation with the memory outpatient clinics will be contractually fixed in the form of a cooperation contract before the start of data collection. The memory outpatient clinics planned for the present study are facilities that are already contractually involved as recruitment centres in the overall digiDEM Bayern project. Participants are recruited based on patients who come to the relevant memory outpatient clinic for diagnostic clarification. As it is expected that not every person who comes to the outpatient memory clinic for diagnostic clarification has an underlying illness, cognitively healthy people should also be recruited in this way. The data are predominantly already collected as part of the primary diagnostics. Only the digiDEM-SCREEN screening instrument to be validated would have to be carried out in addition to the standard diagnostics.
Patient and public involvement
The digiDEM-SCREEN for recording the current cognitive status of users was developed in-house as part of the project. In order to make the app as user-friendly as possible, feedback from potential future end users was essential during this development process. Therefore, a user-centred iterative development approach was chosen.
To this end, three focus groups with user tests were conducted at various locations in 2023. The participants were senior citizens with and without cognitive impairments. In an iterative process, the feedback was successively incorporated into the app prototype before the next round of testing took place. User feedback was collected both qualitatively and quantitatively with the help of standardised questionnaires and systematically evaluated. The group discussion was recorded and transcribed afterwards. After that, each participant was asked to name the most considerable problems associated with the app and if they wanted to change something. The results were then categorised and analysed based on Mayrings’s qualitative content analysis.22 The system usability scale was used as an easy and understandable instrument to measure end-user usability.23 The main focus here was on minimising potential sources of error. For example, the components were increasingly customised to the needs mentioned in order to reduce operating problems (such as inconsistent gestures) and other technical hurdles (such as unclear menu navigation). Nevertheless, there is still a residual risk that the test cannot be carried out properly. If the first three simple test tasks are not answered correctly, an end screen appears with the message that the test cannot be carried out due to technical or language barriers. In this case, the user is advised to visit a specialised clinician.
Data protection
The digiDEM-SCREEN study will comply with high standards of data protection. The project team developed a data protection concept according to the European General Data Protection Regulation. This concept was integrated into the general data protection concept, which the local data protection supervisor of the Friedrich-Alexander-Universität Erlangen-Nürnberg approved. The medical data are stored electronically and centrally and are only accessible to the study management and the responsible employees. The data that allow conclusions to be drawn about the person’s identity are stored in paper form in a locked cabinet in the office.
Outcome measures
In order to analyse the aforementioned hypothesis, sociodemographic data, diagnosis and results of neuropsychiatric tests (CERAD, MoCA, digiDEM-SCREEN) will be collected at one point in time per person. Diagnosis based on the CERAD scores will be used for calculating the sensitivity and specificity of the digiDEM-SCREEN. MoCA results will be used for analysing the correlation with the digiDEM-SCREEN.
Data collection
The data are collected at one point in time per person. The research partners will be specifically trained by the digiDEM Bayern researchers and will conduct the interviews afterwards. The interviewer enters all collected data online in the software (Research Electronic Data Capture (REDCap)) during the interview or immediately after the tests. The declaration of consent is collected on paper and serves exclusively as proof of the legality of the data collection. It, therefore, has no connection to the data collected online as the declaration of consent is not linked to the data set. The employees of the research partners will be responsible for the initial screening, explanation of the project, obtainment of informed consent and data collection.
Statistical analysis
The sensitivity, specificity and corresponding cut-off values are determined based on receiver-operating-characteristic (ROC) curves using the clinical diagnoses. The correlation of the total values of the screening tests developed by the applicants (digiDEM-SCREEN) with the total values of existing cognitive screening/testing procedures is determined by comparing ROC curves and calculating Pearson or Spearman correlation coefficients and effect sizes. Descriptive statistics are tabular and include mean values, SD and 95% CI. Missing data, if occurring, will be calculated by multiple imputations. As estimations become more precise with a higher number of iterations (due to a decrease in the SEs of the estimated parameters), 20 iterations are aimed to be conducted. The number of iterations might vary depending on the extent of missing data. Each imputed data set will then be analysed individually. Afterwards, parameter estimates from the 20 data sets will be pooled into a single result. In the case of a few data missing completely at random, listwise deletion resulting in a complete case analysis might be a more suitable way of dealing with the missing data.24 25
Data will be analysed using SPSS 28 (IBM Corp. Released 2021. IBM SPSS Statistics for Windows, V.28.0. Armonk, NY).
Technical aspects
The project will implement a digital web-based platform for data collection using REDCap electronic data capture tools.26 27 REDCap is a secure, web-based software platform to support data capture for research studies. REDCap enables the interviewers to capture data online via laptop, tablet or smartphone. REDCap is available to institutional partners at no charge.
Ethics and dissemination
digiDEM Bayern will be conducted in accordance with the provisions of the Declaration of Helsinki. The study obtained ethical approval from the Ethics Committee of the Julius-Maximilians-Universität of Würzburg (JMU) (application number: 177/23-sc). Informed consent will be obtained from the participants prior to study inclusion. Participation is voluntary.
Study results will be published in national and international peer-reviewed journals and will be presented at national and international conferences. The journal articles will be available after publication on the homepage of the project digiDEM Bayern.
Registration
The validation study is registered in the German study registry Deutsches Register Klinischer Studien (registration number: DRKS00033764).
Discussion
Digital screening tools can recognise early cognitive changes, speed up the diagnostic process and support monitoring the disease’s progression.
Digital screening tools primarily enable early detection of cognitive abnormalities. These tools can identify changes in cognitive status through regular, standardised cognitive tests. This early detection offers the possibility of prompt therapeutic planning. The instruments can also facilitate the continuous monitoring of the progression of the disease. The assessment enables a more precise evaluation of the course of the disease, more effective treatment planning and targeted adaptation of therapeutic measures.28 In addition to improving clinical care, digital screening tools offer improved access to remote monitoring of patients with dementia. Relatives and carers can use these tools to monitor the patient’s health remotely and intervene if necessary, increasing care safety and efficiency. This includes, in particular, people from rural communities or those with limited mobility. In addition, the data collected by digital screening tools could also be used for research and development of new treatment approaches. Analysing large data sets can also help identify patterns and trends during the disease.29
Besides the potential benefits of a (digital) screening, ethical considerations need to be considered when developing and implementing screening tools, particularly in the case of (digital) tools that will be used in an unsupervised environment. Becoming aware of possible pathological cognitive developments might lead to emotional distress. However, a randomised controlled trial from the USA found no harm from screening in terms of depressive and anxiety symptoms.30
Despite validation measures, false-positive and false-negative results are expected, providing a false perception of safety or leading to unnecessary further diagnostic actions.31 Thus, it is essential to provide a transparent and realistic interpretation of the test results.
Since the emergence of genetic diagnostic measures, a debate about the provision and withholding of information arose, culminating in the question of whether a ‘right not to know’ exists. Supporters argue that patients should be granted the autonomy to make their own choices, which include not being forced to receive information. That is, in particular, true for diseases without a cure, where medical information can result in hopelessness, distress or even stigma and discrimination. On the other hand, some argue that autonomy and autonomous decision-making necessarily require information. Furthermore, as disease might have an impact on the social environment, the refusal to receive information might harm others.32
In the case of providing information, clarification and truthfulness are central to communication with the affected people as different aspects of pathological cognitive impairments might be difficult to understand.33 In this regard, it is important to emphasise that the planned screening test digiDEM-SCREEN is not supposed to provide a diagnosis itself. In fact, the test will give feedback about the current cognitive status and possible cognitive impairments that should lead to the users seeking further diagnostic measures by medical professionals. Therefore, based on the result, there might be a recommendation to consult medical professionals. Interpretations of the results and limitations of the screening test will be communicated in a clear and comprehensible manner.
Digital screening tools for dementia can provide low-threshold access to cognitive testing and, therefore, make important contributions to increasing diagnostic rates. However, there are also structural obstacles that need to be overcome in order to increase diagnostic rates. In Germany, only specialised diagnostic facilities can provide comprehensive diagnostic measures. Those facilities are usually part of clinics that are predominantly accessible in metropolitan areas. A recent study by Rühl et al demonstrated the limited access to diagnostic facilities in the rural areas of Bavaria by illustrating the spatial distances between the facility and the place of residency. The majority of the people with dementia in the Bavarian study population (40%; n=93 950) lived in municipalities with an average journey time of 20 to 40 min to the nearest specialised facility. More than 20% (n=50 877) have to travel longer than 40 min. Thus, the authors resumed that in view of demographic developments there is an urgent need for good accessibility to memory clinics. They recommend the targeted expansion of specialised facilities in areas with long journey times or the provision of mobile diagnostic services.34
For future research, a comparison with digitally administered screening tools (eg, MoCA Duo35 or ‘XpressO’ by MoCA36) would give meaningful insights. When conceptualising the study, the MoCA as it is at the moment still more common in the scientific literature and among our research partners as the digital version. The digiDEM-SCREEN will be administered after the diagnostic work-up. By not carrying out too many further tests, the additional burden on the participants should be kept to a minimum and possible learning effects should be avoided.
Ethics statements
Patient consent for publication
Acknowledgments
The present work was performed by Michael Zeiler in (partial) fulfilment of the requirements for obtaining the degree 'Dr. rer. biol. hum.' at the Medical Faculty of the Friedrich-Alexander-Universität Erlangen-Nürnberg.
References
Footnotes
MZ and ND contributed equally.
Contributors MZ, ND, H-UP, EG and PK-R initiated and planned the project. MZ, ND and PK-R designed and conceptualised the study protocol. MZ and ND wrote the study protocol. PK-R supervised the protocol. MZ, ND, PH and PK-R developed the methodology. Technical implementation was done by KK and RP. RP, PH and PK-R performed quality control within the project. MZ, ND, KK and RP contributed to ethics and data protection. Patient and public involvement was carried out by MZ and ND. Biometry was done by UF. Critical revision of the protocol was done by RP, PH and PK-R. The guarantor of the study is PK-R. PK-R accepts full responsibility for the finished work and the conduct of the study, had access to the data and controlled the decision to publish.
Funding The project is funded by the Bavarian State Ministry of Health, Care and Prevention as part of the funding initiative ‘BAYERN DIGITAL II’ (funding code: G42d-G8300-2017/1606-83). The funder was not involved in the study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication.
Competing interests RP is a shareholder in Lenox uG, which has set itself the goal of translating scientific findings into digital health applications. Lenox uG holds shares in HealthStudyClub GmbH. For its part, HealthStudyClub GmbH is responsible for developing the technical app for this project. KK is also a shareholder in HealthStudyClub GmbH and has joined the company as chief technical developer. digiDEM Bayern that is funded by the Bavarian State Ministry of Health, Care and Prevention is owner of the data.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.