Skip to main content
Download PDF
Download ePub

Evaluating the implementation of PROMs and PREMs in routine clinical care: co-design of tools from the perspective of patients and healthcare professionals

Health and Quality of Life Outcomes volume 23, Article number: 15 (2025) Cite this article

Abstract

Background

Implementation of patient-reported measures (PRMs) is an integral element for patient-centered models; however, there is still hardly any quantitative evidence regarding its impact in routine care settings. The objective of this study was to codesign two concise tools that allow for a standardized and longitudinal assessment of the implementation of PRMs in routine care in terms of acceptability and perceived value from the perspective of both patients and healthcare professionals.

Methods

A list of constructs and items to be presented, separately, to patients and healthcare professionals was created from evidence gathered through a narrative literature review. Focus groups, composed of either patients or healthcare professionals from different chronic conditions, were conducted for the co-design of independent assessments. Once agreement was reached, the content validity was examined in separate consensus meetings.

Results

A total of 10 patients and 10 healthcare professionals participated in the focus groups. After 7 focus groups, the PRMs Implementation Assessment Tool for patients (PRMIAT-P) was developed with 33 items in 9 constructs, and the tool for healthcare professionals (PRMIAT-HP) had 33 items in 16 constructs. Content validity was confirmed for both tools.

Conclusions

The perspective of patients and healthcare professionals regarding the implementation of PRMs in routine care can be evaluated quantitively with the PRMIAT tools. These tools are understandable, concise and comprehensive, and can be used in multiple settings and for different chronic conditions. They have been codesigned as a standard set to facilitate both longitudinal assessments and performing benchmarking among different initiatives.

Background

International organizations increasingly advocate for the shift toward patient-centered care models [1, 2], with the use of patient-reported measures (PRMs), both health outcomes (PROMs) [3] and experiences (PREMs) [4], being crucial to integrate patients’ voices into health services [1, 5, 6]. The systematic use of PROMs in clinical settings has been shown to improve survival [7], symptom control [8, 9], detection of unrecognized problems [10, 11], rates of emergency room visits and hospitalizations [12], communication between patients and healthcare professionals, shared decision making, and patient satisfaction [11]. On the other hand, PREMs are performance indicators that aim to collect objective and reliable information about experience in clinical encounters, allowing for quality improvement actions [13,14,15], and facilitating the standardization of data for long-term monitoring and benchmarking [16].

Owing to the numerous stated benefits, implementations of PROMs are underway at different levels, from micro (individual patient management) to macro (regional programs) [17, 18]. Examples of large implementations include Australia’s New South Wales region [17], Alberta’s cancer care program (Canada) [19, 20], and the UK’s National Health Service (NHS) PROMs program [21, 22]. PREMs have been implemented since 1995 through the US Consumer Assessment of Healthcare Providers and Services (CAHPS) across the Medicaid and Medicare programs [23,24,25]. The successful adoption of PROMs and PREMs in routine clinical care can change the organization and delivery of healthcare services. However, some requirements at different levels need to be considered, such as time, cost, response rates, or appropriate outputs; as well as an evaluation that takes into account patients’ and healthcare professionals’ needs, minimizing burdens, and understanding barriers [18].

There are several initiatives to guide healthcare systems to select and implement PROMs or PREMs in different settings [1, 23, 26,27,28,29] including some of them focused on routine clinical care [23, 29]. A systematic review has synthesized evidence on the perceived benefits and limitations of using PROMs in clinical practice as perceived by patients and healthcare professionals from qualitative studies [30].

Nevertheless, there is scarce quantitative evidence regarding the impact that PRMs implementations in routine clinical care has on patients and healthcare professionals. The ISOQOL Clinical Practice Implementation Science Work Group recommended indicators, such as the response rate of PRMs, or the percentage of healthcare professionals that attend training activities, for evaluating PRMs implementations [31], which are based on the main Implementation Science constructs (acceptability, adoption, fidelity, appropriateness, sustainability or feasibility). In addition to ad-hoc qualitative studies [30, 32], and the aforementioned proposal on potential indicators, a structured quantitative method has been used in some studies to evaluate the degree to which the routine implementation of PRMs has been successful [33, 34]. Recently, a study aimed at identifying barriers and facilitators postimplementation from the perspectives of healthcare professionals and patients, with a codesigned an ad-hoc survey, revealed the necessity of ongoing efforts to ensure a successful hospital-wide PROM implementation [35]. Although, these studies did quantitively measure the impact of their implementations, the tools employed were only used for PROMs implementation, they were not designed for longitudinal use and their development was not properly described. Moreover, a narrative review of the use of PREMs highlights, not only the scarcity of longitudinal information from PREM initiatives, but also the lack of evidence of its impact [15].

To the best of our knowledge, standardized tools that measure the impact of both PROMs and PREMs implementations in routine clinical care from the perspective of patients and healthcare professionals do not exist. Thus, the objective of this study was to codesign concise tools that allow for a standardized and longitudinal assessment of the implementation of PRMs in routine clinical care, from the patients’ and healthcare professionals’ perspectives independently, in terms of acceptability and perceived value. The tools aim to be applied at different institutions and regions, to evaluate the implementation of PRMs regardless of the setting or condition.

Methods

In the absence of specific procedures, international guidelines for PROMs development [36,37,38] were followed. First, to identify relevant areas in which PRMs implementation may impact patients and healthcare professionals, a narrative literature review [39] on the assessment of PRMs implementations was conducted. Second, the identified areas were used to initiate a codesign process that included focus groups with patients and healthcare professionals to define, select, and properly formulate the content of the tools for PRMs implementation assessment. Content validity was examined through consensus meetings with patients and PRMs experts.

The study adhered to the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [40], a tool for reporting qualitative studies.

Literature review

A narrative literature review of published evidence was conducted to identify existing evaluation strategies for PRMs implementation initiatives through either qualitative or quantitative methodologies. This search was performed in PubMed in November 2022, without restrictions on the publication dates, using terms such as: quality of life, PROM, PREM, implementation, routine use, evaluation, assessment, and impact. The backward and forward citation tracking method was also used to capture literature that did not appear directly in the search. The inclusion criteria were as follows: description of a PRM implementation via an Implementation Science framework, evaluation of a PRM implementation through surveys or qualitative methodologies, exploration of barriers and facilitators of a PRM implementation in routine practice, and implementations of PRMs at any level of the health system or in any pathology. The exclusion criteria were implementations of PRMs without an evaluation being described and implementation initiatives that were not specific for routine clinical practice.

Codesign of the assessment tools

The relevant areas identified in the literature review were extracted and edited into a content list to be presented in the focus groups, as the next step of the development process [38]. Focus groups took place at the hospital’s research institute and started with a brief explanation of the PROMs, PREMs and project objectives, and were moderated by three of the researchers following a semi structured format [37].

The participants were patients in active management for breast or prostate cancer, chronic kidney disease, or bariatric surgery purposively selected, and health professionals from hospital departments participating in existing research studies with PROMs. All participants gave prior written informed consent. The focus groups were stratified by condition and conducted separately for patients and healthcare professionals. Patients’ age, gender, basic clinical variables and housing area codes were collected from their medical records. Healthcare professionals’ age, gender, job/role, years of experience, and years working in the institution were also collected.

Phenomenological inquiry was used to understand the participants’ perspectives and detect new constructs or items, modify or remove those already detected in the literature, and assess their relevance and importance through discussion until agreement and saturation were reached [41]. The researchers made the final decision at the end of each focus group if no agreement among the participants was reached. The response options were also discussed in each group. This process was replicated for all focus groups to ensure that the content represented a wide range of health conditions. An updated version was produced by the research team after each focus group and presented to the following group. The items of both tools (patients and healthcare professionals’ versions) were developed simultaneously in Catalan and Spanish during the focus groups. All focus groups were audio recorded and transcribed with the software HappyScribe Ltd.© [42] to ensure that the distinct versions of the item lists gathered all the updates agreed upon by the participants. Researchers took notes during the focus groups to guide the changes to be made to the content list.

Content validity

Content validity was examined through two consensus meetings that were organized to judge the clarity, comprehensiveness, relevance, and redundancy of each item of the tools [43, 44]. In the case of the patients’ tool, one participant from each condition was selected to participate, all of whom were expert patients. For the healthcare professionals’ tool, a consensus meeting was organized with the PRMs experts, none of whom had participated in the focus groups. The guide developed beforehand for each meeting included 4 questions regarding comprehensibility, adequacy, and importance, together with a final question about whether participants would change anything in the item. The resulting tools were sent to the participants of the focus groups for their feedback.

Ethics

This study was approved by the Ethics Committee of Parc de Salut Mar (Barcelona, Spain).

Results

Among the articles produced by the search in PubMed, 50 were considered relevant for the present study. From those finally included in the narrative review of the literature, several domains were selected that focused on patients’ and healthcare professionals’ perspectives (7 from the Consolidated Framework for Implementation Research [45], and 8 from the Proctor framework [46]): patients’ needs and resources [45], structural characteristics [45], sustainability [46], knowledge and beliefs about the intervention [45, 46], adoption [46], self-efficacy [45], and individual state of change [45]. Moreover, after also reviewing the qualitative evidence from the literature, 15 items covering 7 different constructs were presented to patients in the focus groups; and 24 items, covering 14 constructs, were presented to healthcare professionals [30, 32, 45,46,47,48,49,50,51,52,53,54,55,56]. These areas along with a study on the application of Implementation Science theories to inform the use of PROMs in routine clinical care, guided the initial content list for the focus groups [48].

In total, 10 patients and 10 healthcare professionals participated in the focus groups (the participants’ demographic information is shown in Table 1). A total of 50% of the patients were female, aged 40–77 years, and had diverse socioeconomic backgrounds [57]. The average time since diagnosis of their main pathology was 9 years (range 1–42 years), and most of the participants (8/10) had undergone at least one surgery. Most of the healthcare professionals who participated were female (7/10), and half of them were < 50 years old. On average, the healthcare professionals had 19.2 years of experience in their profession (ranging from 8–37 years). The main reason for not participating was time availability by both patients and healthcare professionals.

Table 1 Participant characteristics
Full size table

Figure 1 shows the steps followed for the development of the two PRM Implementation Assessment Tools (PRMIAT) and the number of constructs and items at the end of each of these steps. After the 4 focus groups with patients, the instrument for patients contained 23 items in 7 constructs. After the 3 focus groups with healthcare professionals, the instrument for healthcare professionals contained 34 items in 14 constructs. Saturation was reached in the last focus groups, as the same themes had come out repeatedly previously. The focus groups lasted between 80 and 120 min.

Fig. 1
figure 1

Development process of the PRMs Implementation Assessment Tools

Full size image

The content validity evaluation of the patients’ tool was completed in a 60-min consensus meeting with 8 participants (one patient with each condition and 4 investigators). Six items were reformulated for better comprehension. Alterations included editing terminology such as changing “healthcare center” to “hospital”, incorporating examples within the items, modifying the verbal tense, changing the direction of the item, and shortening the items. It was ensured that the wording remained adequate for patients with different conditions; items were aligned with the concepts they were intended to measure and maintained relevancy. Redundancy and completeness were evaluated, resulting in the addition of 6 items related to the constructs of ‘suitability for all patients’, and ‘value of the implementation’. Separate sections were designated within the tool to accommodate the different aspects of PROMs and PREMs independently. Additionally, four items addressing usability were included.

The content validity of the healthcare professionals’ tool was appraised in a 90-min consensus meeting of 15 PRMs experts (epidemiologists, public health researchers, psychometricians, and medical residents). As a result, two items were excluded due to low relevance and one new item was added from splitting an item into two, focusing on PROMs’ and PREMs’ implementation separately. Five items were edited for better comprehensibility by adjusting the verbal tense or restructuring the question, whereas seven items required substantial revisions to clarify ambiguous language or provide additional detail. Four open-ended questions were appended to elicit insights into the advantages, disadvantages, barriers encountered, and suggestions to improve the usability of the platform being used to complete the PRMs.

The final PRM Implementation Assessment Tool – Patients (PRMIAT-P) is composed of 33 items that cover 9 constructs (Table 2). It is divided into 4 sections: 1) to be answered independently by all patients regardless of previous exposure to PRMs as part of their routine clinical care (11 items), 2) on the implementation of PROMs (13 items), 3) on the implementation of PREMs (5 items), and 4) usability questions (4 items). Sections 2–4 have been developed to be administered after patients have answered PROMs, PREMS, or both (Fig. 2).

Table 2 Constructs and items of the PRMs Implementation Assessment Tools for patients
Full size table
Fig. 2
figure 2

Constructs and administration timeline for the PRMs Implementation Assessment Tools for patients and healthcare professionals

Full size image

The PRM Implementation Assessment Tool – Healthcare Professionals’ (PRMIAT-HP) is composed of 33 items representing 16 constructs (Table 3). The tool closes with 4 open questions to be answered once the PRMs have been implemented. The open questions were added to identify the advantages and disadvantages of the implementation that have yet to be detected or may be specific to a department or service, the usability issues they encountered, and integration problems with the current visualization software. It is divided into time sections: before the implementation program has begun (24 items), once PRMs have been implemented, or at both time points (29 items) (Fig. 2).

Table 3 Constructs and items of the PRMs Implementation Assessment Tools for healthcare professionals
Full size table

Both tools use a 5-point Likert scale ranging from ‘Strongly disagree’ to ‘Strongly agree’ [58]. The participants confirmed the adequacy of this response scale when presented at each focus group and at the two consensus meetings. See supplementary file 1 for the full version of the questionnaire.

Discussion

The objective of this study was to codesign concise tools with relevant stakeholders, that allow for a standardized and longitudinal assessment of the implementation of PRMs in routine clinical care in terms of acceptability and perceived value. The developed PRMIAT tools (PRMIAT-P and PRMIAT-HP) differ from previous assessment strategies [33,34,35, 48, 55, 59] by providing independent and different information from patients and healthcare professionals; being applicable in different settings, institutions, and regions; evaluating the impact of administering PROMs and/or PREMs; and measuring change in the stakeholders’ perception throughout their implementation.

Among the PRM implementation initiatives that have previously been quantitively evaluated, three [33, 35, 48] stand out for meeting several of the abovementioned criteria. One of them applied the Proctor implementation framework [46] to develop indicators to evaluate the implementation of PROMs in several conditions [48] from the perspective of patients and clinicians. However, these indicators were not codesigned with patients or healthcare professionals, PREMs were not included, and the evaluation of this implementation was not published. The second initiative developed the Medical Care Questionnaire (MCQ) to evaluate PROMs implementation in a randomized control trial of oncology patients [33, 60]. The MCQ measures 3 domains (communication, coordination, and patient preferences) at different timepoints, administering an end-of-study questionnaire to patients and clinicians to evaluate usability, content, relevance, usefulness, and reliability of the use of PROMs in routine care. However, since the MCQ also covers the experience of the patient with the healthcare system, it may not be suitable for evaluating an implementation that also includes PREMs in routine care [60]. The third initiative developed a survey based on interviews with patients and healthcare professionals focused on identifying enablers and barriers to be administered only after the implementation of PROMs [35].

Previous studies have identified different barriers for patients (time burden, poor usability, lack of feedback) than for healthcare professionals (lack of knowledge, integration in their workflow, difficult access and interpretation of results) [47, 49, 61,62,63,64]. In the present study, several constructs were included in both the PRMIAT-P and the PRMIAT-HP, indicating that certain areas of interest and key features of a PRM implementation, such as the focus of the consultation or knowledge and beliefs about the implementation, are shared between the patients and the healthcare professionals. However, the return of information to the patient [11], an important indicator of a successful implementation, was relevant only for patients in our study, whereas logistic support for the implementation program and training were constructs relevant only to healthcare professionals, thus were only included in their questionnaire.

The PRMIAT tools make it possible to measure patients’ and healthcare professionals’ perspectives before and after PRMs have been implemented. For example, healthcare professionals are asked about the degree to which they think that PRMs benefit their patients, which should change over time if PRMs are being used routinely. To our knowledge, there is only one ad-hoc questionnaire [34] that allows for the evaluation of healthcare professionals’ perspectives before and after the implementation of PROMs; however, it does not evaluate patients’ perspectives. The majority of PRMs implementation assessments (qualitative or quantitative) only assess patients or healthcare professionals once the implementation has begun [12, 30, 35, 51]. A major difficulty in measuring change is related to presenting patients with questions about hypothetical concepts and tools that they have yet to use and understand, which requires that some constructs be presented in the conditional tense. This difficulty could explain why previous evaluations did not measure change. However, it is particularly important to be able to measure longitudinal changes in perspective toward implementation by stakeholders before and after PRMs are integrated into routine workflows.

Because of this aspect of the tool, being able to detect changes before and after PRMS implementation, together with the fact that it can be used to assess the impact of both PROMs and/or PREMs’ use, the PRMIAT is divided into 4 sections that can be used independently according to the characteristics of the implementation program. A recently published review emphasized the lack of longitudinal and quantitative data describing the impact of the use of PREMs [15]. In fact, the PRMIAT is remarkable for enabling also the evaluation of PREMs’ implementation, in contrast to most published evidence, which focuses predominantly on PROMs [34, 35, 48, 56].

From a methodological standpoint, codesigning with the participation of different stakeholders — in this case patients, healthcare professionals and PRMs experts — is a key factor to ensure the content validity of the new instrument [34, 55, 56]. Taking healthcare professionals’ and patients’ voices into account is crucial [28, 51, 63, 65] to be aware of the concrete benefits of using PRMs routinely, not just assuming the positive results obtained from clinical trials, but also identifying enablers and barriers for all the actors involved. Nevertheless, none of the previous studies that used a survey or questionnaire to assess the impact of a PRM implementation evaluated the content validity of the tool as part of their development, nor assessed its psychometric properties once the tools had been administered.

In the present study, the COSMIN methodology for evaluating the content validity of PROMs [66] was used to drive the consensus meetings. Even though that methodology was developed to be used for systematic reviews in which the content validity of several PROMs is evaluated and compared, it can also be used as a guide when developing a structured questionnaire. In that sense, researchers reviewed the COSMIN-proposed criteria to ensure good content validity, summarizing the relevance, comprehensiveness, and comprehensibility of the PRMIAT-P and PRMIAT-HP.

Once the PRMIAT tools are administered to evaluate a PRMs implementation in routine clinical care, they should be complemented with process indicators for institutional strategic purposes depending on the level of the implementation, in addition to cost measures as recommended in previous studies [31, 48]. All the complementary indicators, including those related to the use of e-platforms to collect PRMs, should be specifically selected for each program to gain a comprehensive understanding of the impact of that specific implementation.

Limitations

While this study provides a valuable tool for PRMs implementations, which are increasingly common worldwide, several limitations should be acknowledged. First, the tools developed in the present study could have been composed of other items if participants had belonged to other health units or if their participation order had been different. Nevertheless, the selected conditions represent a wide range of healthcare processes, and initial constructs were maintained throughout the development process; thus, all groups had the chance to discuss and complement them. Second, the researchers leading the focus groups and consensus meetings potentially had a bias toward the benefits of the PRM implementation, due to their background, that could have steered the discussions. To avoid this bias, participants were informed of the reason why this study was designed, and of the most prevalent benefits and hindrances found in the literature. Moreover, the research team did not have any relationship with the patients prior to the study; however, they did have a relationship as collaborators with the healthcare professionals who participated in the focus groups. Third, the tools presented here were developed simultaneously in Catalan and Spanish; thus, for their proper utilization outside of the Spanish healthcare context, a cultural and linguistic adaptation should be performed.

Additionally, the focus groups were all recorded, but the transcription was only used by the research team to support changes in the content of the tools derived from paper notes; they were therefore not analyzed. Lastly, each focus group with patients and healthcare professionals in each condition had a limited number of participants, though the total number of participants was in line with qualitative studies (10 patients and 10 healthcare professionals). The working team of the project includes 3–5 expert patients and 2–5 healthcare professionals per group, and all were invited to the focus groups, although not all could attend on the date agreed upon. However, in the process of recruiting and selecting patient participants, it was ensured that the patients had undergone different types of treatments, to achieve heterogeneity within each focus group. Similarly, the healthcare professionals who participated were from different disciplines within the care of the same health condition. Separating patients based on their clinical condition for the focus groups was a conscious decision by the research team, to allow the different experiences of patients and healthcare professionals to influence the development of tools.

Conclusions

The PRMIAT tool is the first approach that allows for standardized evaluations of PROMs and PREMs implemented in routine clinical practice, considering the perspective of the main actors and beneficiaries of these programs. Both instruments, PRMIAT-P and PRMIAT-HP, are understandable, appropriate for their intended use, comprehensive, and can be easily used in multiple settings and for different chronic conditions. They have been designed to become a standard set for the assessment of the implementation of PRMs, facilitating repeated evaluations in the same center (to assure improvement), and comparisons of different implementation frameworks, approaches, and administration or visualization platforms, becoming a benchmark to compare institutions or regions. The real-world implementation of PRMs can impact patients’ empowerment, health monitoring, or health care quality. This impact needs to be confirmed through its assessment, and the PRMIAT can be the tool to achieve it.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

CAPHS:

Consumer assessment of healthcare providers and systems

MCQ:

Medical Care Questionnaire

NHS:

National Health Service

PREMs:

Patient-Reported Experience Measures

PRMs:

Patient-Reported Measures

PROMs:

Patient-Reported Outcome Measures

PRMIAT:

PRMs Implementation Assessment Tool

PRMIAT-HP:

PRM Implementation Assessment Tool-Healthcare Professionals

PRMIAT-P:

PRM Implementation Assessment Tool-Patients

References

  1. Coulter A. Measuring what matters to patients. BMJ. 2017;356.

  2. World Health Assembly, 69. Framework on integrated, people-centred health services: report by the Secretariat. World Health Organization; 2016. Report No.: A69/39. Available from: https://iris.who.int/handle/10665/252698.

  3. Speight J, Barendse SM. FDA guidance on patient reported outcomes. BMJ. 2010;340:518.

    Article  Google Scholar 

  4. Kingsley C, Patel SP. Patient-reported outcome measures and patient-reported experience measures. BJA Educ. 2017;17:137–44.

    Article  Google Scholar 

  5. Santana M-J, Ahmed S, Lorenzetti D, Jolley RJ, Manalili K, Zelinsky S, et al. Measuring patient-centred system performance: a scoping review of patient-centred care quality indicators. BMJ Open. 2019;9:e023596.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Langberg EM, Dyhr L, Davidsen AS. Development of the concept of patient-centredness – A systematic review. Patient Educ Couns. 2019;102:1228–36.

    Article  PubMed  Google Scholar 

  7. Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment. JAMA. 2017;318:197.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Strasser F, Blum D, Von Moos R, Cathomas R, Ribi K, Aebi S, et al. The effect of real-time electronic monitoring of patient-reported symptoms and clinical syndromes in outpatient workflow of medical oncologists: E-MOSAIC, a multicenter cluster-randomized phase III study (SAKK 95/06). Ann Oncol. 2016;27:324–32.

    Article  CAS  PubMed  Google Scholar 

  9. Basch E, Schrag D, Henson S, Jansen J, Ginos B, Stover AM, et al. Effect of Electronic Symptom Monitoring on Patient-Reported Outcomes Among Patients With Metastatic Cancer: A Randomized Clinical Trial. JAMA. 2022;327:2413–22.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Denis F, Lethrosne C, Pourel N, Molinier O, Pointreau Y, Domont J, et al. Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients. J Natl Cancer Inst. 2017;109(9):djx029.

  11. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211.

  12. Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol. 2016;34:557.

    Article  CAS  PubMed  Google Scholar 

  13. Beattie M, Murphy DJ, Atherton I, Lauder W. Instruments to measure patient experience of healthcare quality in hospitals: A systematic review. Syst Rev. 2015;4:1–21.

    Article  Google Scholar 

  14. Bull C, Byrnes J, Hettiarachchi R, Downes M. A systematic review of the validity and reliability of patient-reported experience measures. Health Serv Res. 2019;54:1023–35.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Jamieson Gilmore K, Corazza I, Coletta L, Allin S. The uses of Patient Reported Experience Measures in health systems: A systematic narrative review. Health Policy. 2022;128:1–10.

  16. De Rosis S, Cerasuolo D, Nuti S. Using patient-reported measures to drive change in healthcare: The experience of the digital, continuous and systematic PREMs observatory in Italy. BMC Health Serv Res. 2020;20:1–17.

    Article  Google Scholar 

  17. Rutherford C, Campbell R, Tinsley M, Speerin R, Soars L, Butcher A, et al. Implementing Patient-Reported Outcome Measures into Clinical Practice Across NSW: Mixed Methods Evaluation of the First Year. Appl Res Qual Life 2020 163. 2020;16:1265–84.

  18. Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346.

  19. Watson L, Delure A, Qi S, Link C, Chmielewski L, Photitai É, et al. Utilizing Patient Reported Outcome Measures (PROMs) in ambulatory oncology in Alberta: Digital reporting at the micro, meso and macro level. J Patient-Rep Outcomes. 2021;5:97.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Cuthbert CA, Watson L, Xu Y, Boyne DJ, Hemmelgarn BR, Cheung WY. Patient-reported outcomes in Alberta: rationale, scope, and design of a database initiative. Curr Oncol. 2019;26:e503–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. NHS Digital. Patient Reported Outcome Measures (PROMs). 2023. Available from: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/patient-reported-outcome-measures-proms.

  22. NHS Foundation Trust. Patient reported outcome measures (PROMs) - South West Yorkshire Partnership. 2022. Available from: https://www.southwestyorkshire.nhs.uk/service-users-and-carers/useful-information/patient-reported-outcome-measures-proms/.

  23. Bull C, Teede H, Watson D, Callander EJ. Selecting and Implementing Patient-Reported Outcome and Experience Measures to Assess Health System Performance. JAMA Health Forum. 2022;3: e220326.

    Article  PubMed  Google Scholar 

  24. Consumer Assessment of Healthcare Providers & Systems (CAHPS) | CMS. Available from: https://www.cms.gov/data-research/research/consumer-assessment-healthcare-providers-systems.

  25. About CAHPS. Available from: https://www.ahrq.gov/cahps/about-cahps/index.html.

  26. Snyder C, Crossnohere N, King M, Reeve BB, Bottomley A, Calvert M, et al. The PROTEUS-Trials Consortium: Optimizing the use of patient-reported outcomes in clinical trials. Clin Trials. 2022;19:277–84.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Kelley TA. International Consortium for Health Outcomes Measurement (ICHOM). Trials 2015 163. 2015;16:1–1.

  28. Di Maio M, Basch E, Denis F, Fallowfield LJ, Ganz PA, Howell D, et al. The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO Clinical Practice Guideline 5 on behalf of the ESMO Guidelines Committee. Ann Oncol. 2022;33:878–92.

    Article  PubMed  Google Scholar 

  29. van der Wees PJ, Verkerk EW, Verbiest MEA, Zuidgeest M, Bakker C, Braspenning J, et al. Development of a framework with tools to support the selection and implementation of patient-reported outcome measures. J Patient-Rep Outcomes. 2019;3:75.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Campbell R, Ju A, King MT, Rutherford C. Perceived benefits and limitations of using patient-reported outcome measures in clinical practice with individual patients: a systematic review of qualitative studies. Qual Life Res 2021 316. 2021;31:1597–620.

  31. Stover AM, Haverman L, Van Oers HA, Greenhalgh J, Potter CM. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings On behalf of the ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Qual Life Res. 2021;30:3015–33.

    Article  PubMed  Google Scholar 

  32. Carfora L, Foley CM, Hagi-Diakou P, Lesty PJ, Sandstrom ML, Ramsey I, et al. Patients’ experiences and perspectives of patient-reported outcome measures in clinical care: A systematic review and qualitative meta-synthesis. PloS One. 2022;17.

  33. Velikova G, Keding A, Harley C, Cocks K, Booth L, Smith AB, et al. Patients report improvements in continuity of care when quality of life assessments are used routinely in oncology practice: Secondary outcomes of a randomised controlled trial. Eur J Cancer. 2010;46:2381–8.

    Article  PubMed  Google Scholar 

  34. Roberts NA, Janda M, Stover AM, Alexander KE, Wyld D, Mudge A. The utility of the implementation science framework “Integrated Promoting Action on Research Implementation in Health Services” (i-PARIHS) and the facilitator role for introducing patient-reported outcome measures (PROMs) in a medical oncology outpatient department. Qual Life Res. 2021;30:3063–71.

    Article  PubMed  Google Scholar 

  35. Huberts AS, Koppert LB, Benschop JAM, Meerstra JW, van ’t Spijker A, van Lint C, et al. Facilitators and barriers in the implementation and adoption of Patient Reported Outcomes Measurements in daily practice. Value Health. 2024;S1098–3015(24)02407–0.

  36. SubCommittee S. PROMIS® Instrument Development and Validation Scientific Standards. 2013;72.

  37. Johnson C, Aaronson N, Blazeby JM, Bottomley A, Fayers P, Koller M, et al. Guidelines for Developing Questionnaire Modules. EORTC Quality of Life Group; 2011 p. 51.

  38. Ravens-Sieberer U, Gosch A, Abel T, Auquier P, Bellach B-M, Bruip J, et al. Quality of life in children and adolescents: a European public health perspective. Soz Praventivmed. 2001;46:294–302.

    Article  CAS  PubMed  Google Scholar 

  39. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18:143.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care J Int Soc Qual Health Care. 2007;19:349–57.

    Article  Google Scholar 

  41. Cheng KKF, Clark AM. Qualitative Methods and Patient-Reported Outcomes: Measures Development and Adaptation. Int J Qual Methods. 2017;16:1609406917702983.

    Article  Google Scholar 

  42. HappyScribe Ltd.. Available from: https://www.happyscribe.com/.

  43. Rothman M, Burke L, Erickson P, Leidy NK, Patrick DL, Petrie CD. Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force Report. Value Health. 2009;12:1075–83.

    Article  PubMed  Google Scholar 

  44. Aaronson NK, Alonso J, Burnam A, Lohr K, Patrick DL, Perrin E, et al. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002;11.

  45. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 2009;4:1–15.

    Article  Google Scholar 

  46. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.

    Article  PubMed  Google Scholar 

  47. Wheat H, Horrell J, Valderas JM, Close J, Fosh B, Lloyd H. Can practitioners use patient reported measures to enhance person centred coordinated care in practice? A qualitative study. Health Qual Life Outcomes. 2018;16(1):223.

  48. Ahmed S, Zidarov D, Eilayyan O, Visca R. Prospective application of implementation science theories and frameworks to inform use of PROMs in routine clinical care within an integrated pain network. Qual Life Res. 2021;30:3035–47.

    Article  PubMed  Google Scholar 

  49. Thestrup Hansen S, Kjerholt M, Friis Christensen S, Hølge-Hazelton B, Brodersen J. Haematologists’ experiences implementing patient reported outcome measures (PROMs) in an outpatient clinic: a qualitative study for applied practice. J Patient-Rep Outcomes. 2019;3(1):74.

  50. Dronkers EAC, Baatenburg de Jong RJ, van der Poel EF, Sewnaik A, Offerman MPJ. Keys to successful implementation of routine symptom monitoring in head and neck oncology with “Healthcare Monitor” and patients’ perspectives of quality of care. Head Neck. 2020;42:3590–600.

  51. Talib TL, Dechant · Paige, Kean · Jacob, Patrick ·, Monahan O, Haggstrom DA, et al. A qualitative study of patients’ perceptions of the utility of patient-reported outcome measures of symptoms in primary care clinics. Qual Life Res. 2018;27:3157–66.

  52. Primdahl J, Jensen DV, Meincke RH, Jensen KV, Ziegler C, Nielsen SW, et al. Patients’ Views on Routine Collection of Patient-Reported Outcomes in Rheumatology Outpatient Care: A Multicenter Focus Group Study. Arthritis Care Res. 2020;72:1331–8.

    Article  Google Scholar 

  53. Stern BZ, Pila S, Joseph LI, Rothrock NE, Franklin PD. Patients’ perspectives on the benefits of feedback on patient-reported outcome measures in a web-based personalized decision report for hip and knee osteoarthritis. BMC Musculoskelet Disord. 2022;23:1–12.

    Article  Google Scholar 

  54. Lai CH, Shapiro LM, Amanatullah DF, Chou LB, Gardner MJ, Hu SS, et al. A framework to make PROMs relevant to patients: qualitative study of communication preferences of PROMs. Qual Life Res. 2021;31:1093–103.

    Article  PubMed  Google Scholar 

  55. Depla AL, Pluut B, Lamain-de Ruiter M, Kersten AW, Evers IM, Franx A, et al. PROMs and PREMs in routine perinatal care: mixed methods evaluation of their implementation into integrated obstetric care networks. J Patient-Rep Outcomes. 2023;7:26.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Teela L, van Muilekom MM, Kooij LH, Gathier AW, van Goudoever JB, Grootenhuis MA, et al. Clinicians’ perspective on the implemented KLIK PROM portal in clinical practice. Qual Life Res. 2021;30:3267–77.

    Article  PubMed  Google Scholar 

  57. Generalitat de Catalunya - Agència de Qualitat i Avaluació Sanitàries de Catalunya. Nou indicador socioeconòmic del model d’assignació de recursos de l’atenció primària. Barcelona; 2017;1–17.

  58. Østerås N, Gulbrandsen P, Garratt A, Šaltytë Benth J, Dahl FA, Natvig B, et al. A randomised comparison of a four-and a five-point scale version of the Norwegian Function Assessment Scale. Health Qual Life Outcomes. 2008;6.

  59. van Oers HA, Teela L, Schepers SA, Grootenhuis MA, Haverman L. A retrospective assessment of the KLIK PROM portal implementation using the Consolidated Framework for Implementation Research (CFIR). Qual Life Res. 2021;30:3049–61.

    Article  PubMed  Google Scholar 

  60. Harley C, Adams J, Booth L, Selby P, Brown J, Velikova G. Patient experiences of continuity of cancer care: development of a new medical care questionnaire (MCQ) for oncology outpatients. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2009;12:1180–6.

    Article  Google Scholar 

  61. Howell D, Molloy S, Wilkinson K, Green E, Orchard K, Wang K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol Off J Eur Soc Med Oncol. 2015;26:1846–58.

    Article  CAS  Google Scholar 

  62. Nguyen H, Butow P, Dhillon H, Sundaresan P. A review of the barriers to using Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs) in routine cancer care. J Med Radiat Sci. 2021;68:186–95.

    Article  PubMed  Google Scholar 

  63. Foster A, Croot L, Brazier J, Harris J, O’cathain A. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient-Rep Outcomes. 2018;2.

  64. Roberts NA, Alexander K, Wyld D, Janda M. What is needed by staff to implement PROMs into routine oncology care? A qualitative study with the multi-disciplinary team. Eur J Cancer Care (Engl). 2019;28.

  65. Nordan L, Blanchfield L, Niazi S, Sattar J, Coakes CE, Uitti R, et al. Implementing electronic patient-reported outcomes measurements: Challenges and success factors. BMJ Qual Saf. 2018;27:852–6.

    Article  PubMed  Google Scholar 

  66. Terwee CB, Prinsen C a. C, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27:1159–70.

Download references

Acknowledgements

The authors would like to thank all patients, healthcare professionals, and PRMs experts that participated in the focus groups and consensus meetings. The authors would like to acknowledge the help of the patient co-researchers: Armando Garcia, Ana Maria Hidalgo, Alejandra Cabre, Mercè Leal, Ramón Noró, Roser Carvajal, Josep Gasch, Tamara Comas, Alejandro Roig, and Manuel Martínez. The authors thank Áurea Martin Morris for her support in proofreading and preparing this manuscript for submission.

Hospital del Mar Patient-Reported Measures Group: Clara Amat-Fernandez (Hospital del Mar Research Institute); Renata Briseño-Diaz (Hospital del Mar Research Institute); Montse Ferrer (Hospital del Mar Research Institute); Olatz Garin (Hospital del Mar Research Institute); Yolanda Pardo (Hospital del Mar Research Institute); Ángels Pont (Hospital del Mar Research Institute); Víctor Zamora (Hospital del Mar Research Institute); Guillermo Bosch (Hospital del Mar-ASPB-UPF); Nuria Argudo (Hospital del Mar); Anna Bach (Hospital del Mar); Marc Beisani (Hospital del Mar); David Benaiges (Hospital del Mar); Xavier Castells (Hospital del Mar); Lluis Cecchini (Hospital del Mar); Betty Chamoun (Hospital del Mar); Lluís Fumadó (Hospital del Mar); Marta Hurtado (Hospital del Mar); Alicia Noguera (Hospital del Mar); Maribel Pérez-Piñero (Hospital del Mar); María José Pérez-Sáez (Hospital del Mar); Dolores Redondo-Pachón (Hospital del Mar); Sonia Servitja (Hospital del Mar); Cristina Siles (Hospital del Mar); Maria Vernet-Tomas (Hospital del Mar); Montserrat Villatoro (Hospital del Mar); Betty Chamoun (Hospital del Mar); Jose Maria Valderas (University of Singapore); Catalina Lizano-Barrantes (Universidad de Costa Rica).

Funding

Financial support for this study was provided through grants by Instituto de Salud Carlos III (ISCIII), co-funded by the European Union (PI21/00026) and Generalitat’s de Catalunya Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR2021 SGR 00624). The following researchers have worked in this manuscript while funded by grants: CAF (AGAUR FI-2 00266), CLB (University of Costa Rica OAICE-85–2019).

Author information

Authors and Affiliations

  1. Universitat Pompeu Fabra, Barcelona, Spain

    Clara Amat-Fernandez, Montse Ferrer, Renata Briseño-Diaz & Olatz Garin

  2. Health Services Research Group, Hospital del Mar Research Institute, Doctor Aiguader 88, Barcelona, 08003, Spain

    Clara Amat-Fernandez, Yolanda Pardo, Montse Ferrer, Catalina Lizano-Barrantes, Renata Briseño-Diaz & Olatz Garin

  3. CIBER en Epidemiología y Salud Pública, CIBERESP, Madrid, Spain

    Clara Amat-Fernandez, Yolanda Pardo, Montse Ferrer & Olatz Garin

  4. Universitat Autònoma de Barcelona (UAB), Barcelona, Bellaterra, Spain

    Yolanda Pardo

  5. Teaching Unit of Preventive Medicine and Public Health, Hospital del Mar-ASPB-UPF, Barcelona, Spain

    Guillermo Bosch

  6. Department of Pharmaceutical Care and Clinical Pharmacy, Faculty of Pharmacy, Universidad de Costa Rica, San José, Costa Rica

    Catalina Lizano-Barrantes

  7. Breast Diseases Unit, Hospital del Mar, Barcelona, Spain

    Maria Vernet-Tomas

  8. Urology Department, Hospital del Mar, Barcelona, Spain

    Lluís Fumadó

  9. Gastrointestinal and Bariatric Surgery Unit, Hospital del Mar, Barcelona, Spain

    Marc Beisani

  10. Nephrology Department, Hospital del Mar, Barcelona, Spain

    Dolores Redondo-Pachón & Anna Bach-Pascual

Authors
  1. Clara Amat-Fernandez
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yolanda Pardo
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Montse Ferrer
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Guillermo Bosch
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Catalina Lizano-Barrantes
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Renata Briseño-Diaz
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Maria Vernet-Tomas
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Lluís Fumadó
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Marc Beisani
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Dolores Redondo-Pachón
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Anna Bach-Pascual
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Olatz Garin
    View author publications

    You can also search for this author inPubMed Google Scholar

Consortia

Hospital de Mar Patient-Reported Measures Group

  • Clara Amat-Fernandez
  • , Renata Briseño-Diaz
  • , Montse Ferrer
  • , Olatz Garin
  • , Yolanda Pardo
  • , Ángels Pont
  • , Víctor Zamora
  • , Guillermo Bosch
  • , Nuria Argudo
  • , Anna Bach
  • , Marc Beisani
  • , David Benaiges
  • , Xavier Castells
  • , Lluis Cecchini
  • , Betty Chamoun
  • , Lluís Fumadó
  • , Marta Hurtado
  • , Marta Hurtado
  • , Alicia Noguera
  • , Maribel Pérez-Piñero
  • , María José Pérez-Sáez
  • , Dolores Redondo-Pachón
  • , Sonia Servitja
  • , Cristina Siles
  • , Maria Vernet-Tomas
  • , Montserrat Villatoro
  • , Betty Chamoun
  • , Jose Maria Valderas
  •  & Catalina Lizano-Barrantes

Contributions

CAF contributed to conceptualization, investigation, methodology, project administration, analysis and writing (original draft, review and editing). YP contributed to conceptualization, investigation, methodology, supervision, project administration, and writing (review and editing). MF contributed to conceptualization, investigation, methodology, and writing (review and editing). GB contributed to conceptualization, investigation, methodology, and writing (review and editing). CLB contributed to investigation, and writing (review and editing). RB contributed to investigation, and writing (review and editing). MVT contributed to conceptualization, investigation, and writing (review and editing). LF contributed to conceptualization, investigation, and writing (review and editing). MB contributed to conceptualization, investigation, and writing (review and editing). DR contributed to conceptualization, investigation, and writing (review and editing). AB contributed to conceptualization and investigation(review and editing). OG contributed to conceptualization, funding acquisition, investigation, methodology, supervision, project administration, and writing (review and editing). All co-authors critically revised the manuscript and approved the submitted version.

Corresponding authors

Correspondence to Yolanda Pardo or Olatz Garin.

Ethics declarations

Ethics approval and consent to participate

Informed consent was obtained from all participants involved in this study. This study was approved by the Ethics Committee of Parc Salut Mar (2021/10182/I).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amat-Fernandez, C., Pardo, Y., Ferrer, M. et al. Evaluating the implementation of PROMs and PREMs in routine clinical care: co-design of tools from the perspective of patients and healthcare professionals. Health Qual Life Outcomes 23, 15 (2025). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12955-025-02333-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12955-025-02333-7

Keywords

Health and Quality of Life Outcomes

ISSN: 1477-7525

Contact us