Jump to comment:

• A call for transparent reporting of methodological limitations when conducting RWD transportability analysis
  Carolin Lennartz, Melanie Frank and Nina Haug
  Published on: 28 April 2025

• Published on: 28 April 2025

  A call for transparent reporting of methodological limitations when conducting RWD transportability analysis

  • Carolin Lennartz, Manager Biostatistics iOMEDICO AG
  • Other Contributors:
    • Melanie Frank, Head of Statistics
    • Nina Haug, Senior Data Scientist - Statistics

  The recently published study investigates transportability of overall survival estimates from US to UK populations receiving first-line treatment for advanced non-small cell lung cancer. Transportability is indeed an important topic when local data is not available or sufficient, which is especially true for rare diseases or, in this context, rare biomarker alterations. However, upon careful examination of the present paper, several points emerge that demand attention and reflection.
  First, since only aggregated data from the UK were available, the authors chose to use unanchored matching-adjusted indirect comparison (MAIC). This method, however, requires the very strong assumption that all effect modifiers and prognostic factors are accounted for [1], which is very improbable given that only few patient characteristics were available from the used comparator study. Additionally, MAIC analyses are subject to assumptions about the distribution of covariates in the population for which only summary measures are available. We strongly recommend discussing these issues in connection with the choice of method.
  Moreover, the extrapolation of results from one country to another without considering country-specific factors is quite problematic. Especially, fundamental differences between the US and UK health care systems were only addressed in a half-sentence in the Box on “strengths and limitations of this study“ but were not mentioned in the discussion. Health care c...

  Show More

  The recently published study investigates transportability of overall survival estimates from US to UK populations receiving first-line treatment for advanced non-small cell lung cancer. Transportability is indeed an important topic when local data is not available or sufficient, which is especially true for rare diseases or, in this context, rare biomarker alterations. However, upon careful examination of the present paper, several points emerge that demand attention and reflection.
  First, since only aggregated data from the UK were available, the authors chose to use unanchored matching-adjusted indirect comparison (MAIC). This method, however, requires the very strong assumption that all effect modifiers and prognostic factors are accounted for [1], which is very improbable given that only few patient characteristics were available from the used comparator study. Additionally, MAIC analyses are subject to assumptions about the distribution of covariates in the population for which only summary measures are available. We strongly recommend discussing these issues in connection with the choice of method.
  Moreover, the extrapolation of results from one country to another without considering country-specific factors is quite problematic. Especially, fundamental differences between the US and UK health care systems were only addressed in a half-sentence in the Box on “strengths and limitations of this study“ but were not mentioned in the discussion. Health care coverage or infrastructure and local availability of health services, but also agreements for cost reimbursements may have a large impact on survival rates.
  Furthermore, regarding overall survival, the differences in proportions and types of second-line treatments may have a profound influence on the overall survival time of patients. However, in this study, second line treatments in the two cohorts were only mentioned in the results. but not further considered in the discussion part. We recommend paying attention to this central issue in more detail and examining how this could be solved in the MAIC analysis by at least taking the proportion of second line treatments into account.
  Lastly, “similarity” of the OS curves was only assessed by visual inspection, without any further quantitative measure or prospectively defined criteria. For overall and chemotherapy, the course of the curves for UK and US deviated and even crossed, hinting at missing prognostic factors in the weighting [2]. Also, the comparison with the data from Pilleron et al. [3] showed barely any overlap of confidence intervals. Yet, the authors claim that the curves look similar, and that weighting worked well.
  Thus, clearly the major drawback of this study is that the authors make very strong and potentially misleading claims, while the data merely show some similarity of the curves. At the same time, limitations of the methodology are barely discussed and missing individual patient data from the UK is interpreted as no data being available, although data are only not available to the authors.
  In conclusion, while transportability of data from one population to another is an important topic, especially in comparison between real world evidence and evidence from RCTs, transporting knowledge into another country would require some careful consideration of country specific characteristics like health care systems, local patient characteristics, treatment guidelines and proper discussion of potential drawbacks from this analysis. We hope that the authors will pay attention to these concerns and take proactive steps to refine their work, thereby contributing to the progress of research in this vital area.

  Sources:
  1. Population-adjusted indirect comparisons (MAIC and STC). https://www.sheffield.ac.uk/nice-dsu/tsds/population-adjusted (2022).
  2. Austin, P. C. & Stuart, E. A. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat. Med. 34, 3661–3679 (2015).
  3. Pilleron, S., Morris, E. J. A., Dodwell, D. & Franks, K. N. Patterns of chemotherapy use and outcomes in advanced non-small cell lung cancer by age in England: A retrospective analysis of the population-based Systemic Anti-Cancer Treatment (SACT) dataset. J. Geriatr. Oncol. 14, (2023).

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top