Project Title IMPRESS: IMproving outcomes for ovarian cancer Patients by Refining Evidenced based tools that drive Surgical Standardisation
Project Lead & Pilot location Professor Richard Edmondson - University of Manchester
Inequality Criteria: Age, minority groups and location
Objective Criteria: Improve survival rates & access to care as well as improve patient experience
The IMPRESS (IMproving outcomes for ovarian cancer Patients by Refining Evidenced based tools that drive Surgical Standardisation) project, led by the University of Manchester and Professor Richard Edmondson, aims to address the significant variation in treatment outcomes for women with ovarian cancer across the UK. By leveraging machine learning and data-driven approaches, the project aims to develop personalised decision support tools and enable shared decision-making between patients and clinicians.
The project's goals include demonstrating the effectiveness of machine learning algorithms in identifying patients who may not be receiving optimal care, developing tools for personalised outcome predictions, expanding a comprehensive database across seven UK centres, improving understanding of decision-making in multidisciplinary teams, and designing a tool that facilitates shared decision-making based on personalised prediction of treatment outcomes.
The project faced challenges related to ethical approvals, data collection, and resource requirements. However, it successfully collected data from over 1,100 patients, revealing variations in treatment patterns and survival rates across centres. The project also conducted additional studies, including a Delphi questionnaire to gather patient and clinician input and an observational study on decision-making in multidisciplinary teams.
The findings highlighted the need for improved access to personalised evidence and informed decision-making tools. The project has generated significant data, which is being disseminated through publications and presentations, and has formed a unique data resource that will continue to benefit patients beyond the project's completion.
Future funding plans involve developing a grant application to further advance the predictive tool and assess its usability in clinical practice. The work will be divided into several work packages, including investigating shared decision-making, adding a decision support tool, expanding the database, addressing intellectual property and commercialisation, and analysing cost implications and training needs.
In conclusion, the IMPRESS project has made significant progress in improving outcomes for women with ovarian cancer through the development of personalised decision support tools and enhanced understanding of decision-making processes. Future funding aims to build upon these achievements, optimise resources, and further empower patients and clinicians in treatment decision-making.
Key Factors for success of IMPRESS are:
The success of the IMPRESS project can be attributed to these key factors:
Formation of a unique data resource:
The project has successfully formed a comprehensive database of detailed clinical information, representing the largest dataset of its kind in the UK. This data resource has immense potential for further analysis, research, and enhancing patient care and outcomes.
Collaboration among seven UK centres:
The project has fostered an important collaboration between seven centres in the UK. This collaboration has not only facilitated data collection but will continue to deliver benefits even after the project's completion.
Completion of "side" projects:
The project has successfully completed two "side" projects, namely the ADEPPT study and the MDT study. These projects have generated valuable data on current decision-making processes and future requirements, contributing to the advancement of knowledge in the field.
Overall, the project has made significant strides in data collection, collaboration, and generating valuable insights, laying the foundation for future advancements in personalised decision support tools and improved outcomes for women with ovarian cancer.
Key questions and limitations encountered in the IMPRESS project were:
Time management and regulatory approvals:
Delays in staff recruitment and obtaining regulatory approvals impacted the project's timeline. Clear understanding of required approval levels and efficient time management are essential for smoother project execution.
Data completeness and standardisation:
Challenges were faced in obtaining complete and standardiSed data sets, particularly due to patients receiving treatment at different centres. Emphasising data completeness and implementing measures for standardisation are crucial for robust research outcomes.
Ethical approval process:
Obtaining ethical approval proved to be more time-consuming than anticipated. Future projects should allocate sufficient time for the ethical approval process to avoid delays.
Resource requirements for data collection:
Variations in resource requirements for data collection were observed among sites. Thorough analysis of resource needs is important for efficient allocation of staff time.
Incomplete data from some centres:
Despite efforts to collect accurate and high-quality data, incomplete data sets were obtained from some centres. Ensuring comprehensive data collection across all participating centres is vital for comprehensive research outcomes.
By addressing these areas, future projects can optimise time management, enhance data collection and standardisation, streamline the ethical approval process, allocate resources effectively, and ensure comprehensive data collection, leading to more successful outcomes and advancements in personalised decision support tools.
IMPRESS: IMproving outcomes for ovarian cancer Patients by Refining Evidenced based tools that drive Surgical Standardisation
Full QI Project Summary (10- 15 minute read)
- Brief outline – identify issue & significance
- Focus of project
- Issues & background
- Significant facts
- Project Design and Planning
- Identification of Sites
- Ethical Approval
- Data Collection
- Data Risk
- Data Collection
- Number of Patients in Data
- Resource Requirement Analysis
- Current Practice Evaluation
- Validating Algorithms
- Additional Studies
- Consultation with patients - ADDEPT
- MDT Observational Study
- Data Collection Results
- Outcome Results
- MDT Data Results
- Delphi Data Results
- Lessons Learned – challenges and changes
- Limitations – what we would do differently
- Future Funding
- Work Package 1: Investigating Shared Decision Making (SDM) & Personalised Decision Support and Shared Decision Making (PDS-SDM)
- Work Package 2: Adding Decision Support Tool to Shared Decision-Making
- Work Package 3: Expanding and Refining the Database
- Work Package 4: Intellectual Property & Commercialisation
- Work Package 5: Analysing Cost Implications and Training Needs
Focus of the Project:
Ovarian cancer is a devastating disease that affects thousands of women in the UK. Unfortunately, the reality for these women is harsh and often filled with uncertainty. One of the most alarming aspects of ovarian cancer treatment is the significant variation in outcomes depending on where a woman lives. Shockingly, the chance of a woman being alive just one year after diagnosis can range from 63% to 76% based on her geographical location. This disparity in treatment outcomes is primarily driven by the differences in how the cancer is treated. It is clear that there is a pressing need to address this issue and ensure that all women diagnosed with ovarian cancer receive the optimal care they deserve, regardless of their location. Recognising the potential of machine learning in healthcare, the IMPRESS project has been initiated to explore its application in improving the treatment and outcomes for women with ovarian cancer.
Building upon preclinical work, the project aims to demonstrate that machine learning algorithms can effectively identify patients who may not be receiving optimal care;
• Develop tools for personalised outcome predictions for women with advanced ovarian cancer.
• Enable shared decision-making between women and their clinicians for optimised treatment decisions.
• Expand the database of patient cases to include 7 UK centres and gather detailed information on over 200 variables per patient.
• Improve understanding of decision-making in MDT for ovarian cancer treatment.
• Consult with patients to design a tool that facilitates shared decision-making, powered by personalised prediction of treatment outcomes this will empower women to advocate for the best treatment options for their specific needs.
Issues & Background Information:
The Ovarian Cancer Audit Feasibility Project (OCAFP) has confirmed significant variations in practice between Cancer Alliances in England, leading to potential inequality in treatment. Alarmingly, a considerable number of women with advanced ovarian cancer in the UK may be receiving suboptimal care due to their geographic location. Notably, older women are particularly at risk of not being offered treatment in some centres.
Currently, the decision about what treatment is received is made by an MDT. Whilst highest survival rates are associated with treatments that include both surgery and chemotherapy many patients are treated with other combinations including surgery alone, chemotherapy alone or no treatment at all. Worryingly the rates of these treatment pathways vary considerably around the UK, for instance, the rate of "no treatment" ranges from 15.8% to 27.6% between Cancer Alliances.
Disparities in treatment pathways are influenced by factors such as:
• patient age
• cancer stage/extent
• surgical expertise
• system resources e.g. theatre time
• the prevailing ethos within different MDTs
The absence of personalised evidence hampers teams from questioning alternative treatments. To address this, we propose a personalised predictive system, empowering patients and their clinical teams to freely explore treatment options. However, challenges include developing accurate predictive models, patient-friendly representation, and integrating the tools into clinical practice.
Solution - project design
The project design aims to leverage a data-driven approach to develop a comprehensive surgical data collection system. The goal is to determine the optimal number of patients for surgery, enhancing survival rates while minimising complications for unsuitable patients. Additionally, the project seeks to expand upon existing algorithms that utilise data to provide decision aids, enabling shared care between clinicians and patients through personalised outcome predictions. A full-time project manager was recruited at the University of Manchester and started in post in April 2022, to coordinate ethical approvals and to monitor project progress.
Identification of Sites:
We carefully selected diverse sites, encompassing both large and small centres with varying surgical resection rates. These participating centres represent a comprehensive range of practices across the UK. Among them, one centre surpasses the 99th percentile for resection rates, while two centres fall below the 1st percentile. Additionally, three centres represent mainstream practice. Experienced clinical leads from each site collaborated as co-applicants, and the existing staff undertook obtaining and uploading of the required data. All sites were allocated funding to facilitate data collection.
Obtaining ethical approval proved more time-consuming than anticipated. Initially classified as a service evaluation, it became clear that additional approval was necessary for data analysis and publication. Extensive ethical applications were submitted to the Health Research Authority (HRA) and the University of Manchester. Despite delays and uncertainties, ethical approval was granted without changes. Data collection and queries were completed by six initial sites.
Data and Measures:
Collecting accurate and high-quality data was identified as a major risk in the project, given the complexities associated with obtaining data from NHS systems. Within the NHS, data can be stored in various formats and locations, such as electronic patient records, operating theatre management systems, and laboratory results systems. To mitigate this risk, a tightly defined data dictionary, a predefined case report form, and ongoing discussions with data managers were implemented to ensure accurate data collection and uniformity between centres.
Data from participating sites was transferred securely to the University of Manchester using the approved REDCap system. This allowed data collectors and clinical leads at each site to enter information securely using a predefined case report form. Anonymisation of data eliminated the need for additional confidentiality advisory group approval. At the time of writing, data has been received from 6 out of 7 sites, with a total of 1139 patients included in the dataset. Detailed information about the collected data fields can be found in the project's protocol.
Number of Patients in Data:
The original funding application estimated data collection for 1500 patients. However, as of now, data has been received for 1139 patients from the 6 participating sites.
Resource Requirement Analysis:
As part of the data collection process, sites were requested to provide resource requirements information to understand the time required for data collection at each site. Manchester's data collection was excluded from this analysis as it was completed prior to the funding application.
The analysis aimed to determine the amount of staff time needed if data collection expands to additional sites in the future. While it was assumed that the time needed to collect each record would decrease with familiarity, there was no clear evidence to support this assumption. Additionally, no correlation was found between the number of patients and the time required for data collection. The number of systems required to obtain the data varied across the five centres that provided resource requirements information.
Current practice evaluation:
Data is being used to understand current practice across the six participating centres. Initial validation studies confirm data quality and reveal variations in treatment patterns and survival rates between centres.
Ongoing work is focused on externally validating the previously described predictive algorithms.
Consultation with patients - ADDEPT:
As well as creating a predictive algorithm that provides information on the best treatment to clinicians, we also would like the predictive tool to involve and empower patients to ensure they are receiving the treatment that is best for them, rather than allowing the clinician to steer treatment towards the sites preferred method.
In order to understand what information patients would like to see in a personalised predictive tool, we conducted a Delphi questionnaire involving 25 patients and 20 clinicians to gather input on a personalised predictive tool. Patients emphasised the importance of the following:
- survival prediction
- surgical morbidity
- long-term quality of life
These insights will inform the design of our predictive algorithm. Results are forthcoming for publication.
MDT Observational Study:
In order to better understand our current practice of decision making for patients, and thus how this could be improved we carried out a study of clinical decision making in MDTs. A manuscript of the study is accepted for publication in European Journal Surgical Oncology.
• Trained observers attended MDTs virtually at five participating cancer centres
• Used the GO-MDT-MODe tool to measure input and quality of discussions
• Significant differences were found between centres in meeting structure and format
• Varied input from different staff members to each case reviewed by the MDT
• Notable variation in surgical input, reflecting diverse surgical practices in England
• Highlighted the need for an independent tool to predict patient outcomes and enhance decision- making.
Given the project's main goals were to develop a consortium to generate a data resource, the key endpoints were as follows:
- Data collection: Determine the proportion of available data collected within the project's timeframe and resource allocation.
- Data quality: Assess the data quality by evaluating the proportion of missing data and the length of follow-up data.
Other outcome metrics specific to individual side projects will be defined in results.
Data Collection Results:
Data were collected from 1139 patients across six centres, with one centre yet to report. This represents the largest dataset of detailed clinical information in the UK. The average time taken for data collection per patient was 27 minutes. These findings inform resource requirements for future projects and enable comprehensive analysis of patient characteristics and treatment outcomes.
Analysis of the collected data revealed variations in treatment patterns across the centres. The "ideal" treatment, combining surgery and chemotherapy, ranged from 29% to 76% among the centres. Survival probabilities also varied, highlighting differences in treatment effectiveness and patient outcomes.
MDT Data Results:
During 17 MDT meetings, 870 cases were observed, including 145 advanced ovarian cancer cases. Analysis of discussions for advanced cases showed variations in quality, indicating differing rates of low-quality discussion among the centres.
Delphi Data Results:
Two rounds of questionnaires were conducted with patients (n=22) and clinicians (n=21). The analysis confirmed the importance of predicting survival and perioperative mortality to patients. These insights guide the development of personalised predictive tools and patient-centred decision-making processes.
Lessons learned – challenges & changes:
The project findings revealed significant variation in one-year survival rates (ranging from 63% to 86%) and the proportion of patients receiving ideal treatment (ranging from 29% to 76%) across different centres.
Additionally, some centres exhibited a high proportion of "low quality" MDT discussions, reaching as high as 45%. It was evident from patient feedback that they desired more information to aid in their decision-making process regarding their care, including estimates of overall survival and surgical mortality.
The project demonstrated that high-quality data collection from multiple sites is possible. However, it was also recognised that such an endeavour requires significant resources to ensure accurate and comprehensive data collection.
Limitations and Changes for Future Project:
By addressing these limitations and implementing changes, such as improved time management, enhanced data collection, and continued collaboration, future projects can optimise resources and provide valuable insights to benefit patients.
- Time limitations: Delays in staff recruitment and regulatory approvals were encountered, necessitating consideration of the time required for document preparation, submission to the Health Research Authority (HRA), and receiving confirmation from sites' Research and Development (R&D) departments. Clear understanding of required approval levels is crucial from the project's inception.
- Data completeness challenge: Incomplete data sets were obtained from most centres, particularly due to patients receiving treatment at different centres for various stages of their care. Standardisation and consistency in data collection across patients and centres should be emphasised to minimise variations in completed data fields.
Limitations – what we would do differently now:
Improvements and Ongoing Benefits:
- Exploiting the data resource: The successful formation of a unique data resource resulted from collaboration among seven UK centres, which should continue beyond the project's completion to deliver sustained benefits for patients. This entails utilising the data for further analysis, research, and enhancing patient care and outcomes.
- Side projects and dissemination: Completed "side" projects, including the ADEPPT study and the MDT study, have generated significant data on decision-making processes and future requirements. Actively presenting and publishing these findings contribute to expanding knowledge in the field.
Overall, the project has been very successful with the formation of a unique data resource which is currently being exploited to be deliver benefits for patients. This has come about through the formation of an important collaboration between seven centres in the UK which will continue to deliver benefits even after this project has been completed.
The “side” projects including the ADEPPT study and the MDT study have both completed and are being presented and published at present. These have generated really important data regarding current decision making and what is required in the future.
The progress achieved during the IMPRESS project has paved the way for the consortium to develop a future grant application aimed at further advancing the predictive tool and assessing its usability in clinical practice.
All current sites involved in the IMPRESS project will continue their participation in the future work, while new connections have been established to strengthen the application and project design.
Notable additions include experts in shared decision making, Darren Flynn, a health economist, Joanne Gray, and an expert in machine learning, Glen Martin.
Additionally, the Institute of Cancer Research Clinical Trials unit has contributed to project design and statistical powering, and the governance team at the University of Manchester has provided guidance on the necessary regulations for conducting a trial involving a machine learning algorithm as a medical device within the NHS.
The grant application is being developed for submission in 2023, with the aim of fully utilising the collected data to develop a personalised shared decision-making tool and conducting a clinical trial to evaluate its acceptability among patients and clinicians. The trial will also assess if the tool significantly contributes to ensuring patients receive the most suitable treatment, regardless of the treatment centre. To achieve these goals, the future funding project will be divided into the following work packages:
WP1 – Investigating Shared Decision Making (SDM) & Personalised Decision Support and Shared Decision Making (PDS-SDM): Demonstrate the feasibility and acceptability of shared decision making and a personalised decision support tool for managing advanced ovarian cancer, addressing both patient and clinician perspectives.
WP2 – Adding Decision Support Tool to Shared Decision-Making: Conduct a phase II clinical trial to evaluate the efficacy of integrating a personalised decision support tool into the pathway for patients with advanced ovarian cancer. Stepped-wedge cluster randomisation will be used, comparing shared decision-making (SDM) to personalised decision support and shared decision making (PDS-SDM).
WP3 – Expanding and Refining the Database: Expand and refine the existing database from the IMPRESS project to support the personalised decision support tool.
WP4 – Intellectual Property & Commercialisation: Facilitate the registration of the personalised decision support tool, incorporating a machine learning algorithm, as a medical device with the Medicines and Healthcare products Regulatory Agency (MHRA).
WP5 – Analysing Cost Implications and Training Needs: Analyse the cost implications of practice changes resulting from the personalised decision support tool. Assess the need for additional training for surgeons due to an increase in potentially complex cases.
These work packages aim to further advance the personalised decision support tool, its implementation in clinical practice, and the overall benefits for patients with advanced ovarian cancer.
We are considering using a separate box on the case study webpage rather than place within the main body text. Then we will link items listed below to the resource library.
The project produced several outputs, including:
- MDT practice determines treatment pathway for patients with advanced ovarian cancer: a multi-centre observational study. T Khassan, E Smitten, N Wood, C Fotopoulou, J Morrison, M MacDonald, K Baxter, RJ Edmondson. Accepted for publication Eur J Surg Oncol
- Defining optimal rates of surgery in advanced ovarian cancer. K Baxter, B Russell, A Hawarden, M Gee, N Khalil, D Jones, VN Sivalingam, RJ Edmondson. Accepted for publication Int J Gyn Cancer
- A Delphi Questionnaire to Engage Patients and Clinicians in the Design of a Personalised Predictive Decision Tool K Baxter, J Booth, RJ Edmondson. Presented at the Blair Bell Meeting London Feb 2023
- Treatment regimens for ovarian cancer determine one-year survival: a study of 6 UK cancer centres. K Baxter, N Wood, M Macdonald, J Morrison, J Yap, C Fotolopoulou, J Booth RJ Edmondson