Impact of adoption of patient-reported outcomes in clinical practice on the accuracy of symptom reporting in medical records of cancer patients

Impact of adoption of patient-reported outcomes
in clinical practice on the accuracy of symptom
reporting in medical records of cancer patients

Donatella Marino1*, Chiara Baratelli1*§, Giovanni Guida1,2, Carmela Giovanna Cleopatra Turco1, Gaetano Lacidogna1, Elisa Sperti1, Francesca Vignani1, Emmanuele De Luca1,3, Clizia Zichi1,3, Marco Audisio1,3, Daniela Ballaminut1, Annalisa Bellezza1, Paola Chiotto1, Giovanna Ciriolo1, Rossella Comite1, Fulvia Codegone1, Santina Florio1, Luisa Fusco1, Laura Polimeno1, Donatella Pozzi1, Eliana Zilio1, Sabrina Terzolo1, Massimo Di Maio1,3

1Division of Medical Oncology, Ordine Mauriziano Hospital, Turin, Italy; 2Degree course in Nursing, University of Turin, Italy; 3Department of Oncology, University of Turin, Italy. *Equally contributing first authors; §Present address: Division of Medical Oncology, San Luigi Gonzaga Hospital, Orbassano (TO), Italy.

Received: October 18, 2020. Accepted: October 27, 2020.

Summary. Purpose. Medical records are a relevant source for real-world evidence. We introduced patient-reported outcomes (PROs) in clinical practice, demonstrating a significant quality-of-life improvement, compared to usual visit. In this secondary analysis, we describe the agreement between patients’ and physicians’ reports of 5 symptoms. Our hypothesis was that adoption of PROs questionnaire could significantly improve the agreement. Methods. Eligible patients were receiving active anti-cancer treatment. Patients in the control group underwent usual visits (group A), while patients of group B, before each visit, filled a PROs paper questionnaire, to provide information about symptoms and toxicities. No specific instructions were provided to physicians to integrate such information in medical records. Agreement between patient and physician evaluations was assessed by Cohen’s κ, calculating under-reporting as proportion of toxicities reported by patients but not recorded by physicians. Results. 211 patients (412 visits) have been analyzed. For all symptoms, Cohen’s κ was better for group B: emesis (0.25 group A vs. 0.36 group B), diarrhea (0.16 vs. 0.57), constipation (0.07 vs. 0.28), pain (0.22 vs. 0.42), fatigue (0.03 vs. 0.08). For all symptoms, although under-reporting was relevant in both groups, it was lower for group B: emesis (75.49% vs. 60.0%, p=0.031), diarrhea (82.89% vs. 50.0%, p<0.001), constipation (92.11% vs. 69.57%, p<0.001), pain (59.57% vs. 42.31%, p=0.01), fatigue (82.11% vs. 64.10%, p<0.001). Conclusion. Adoption of paper PROs allowed a significant reduction in under-reporting of symptoms, but agreement remained suboptimal. Direct integration of electronic PROs could minimize the issue of under-reporting of medical records, increasing their accuracy.

Key words. Patient-reported outcomes, real-world evidence, toxicity, under-reporting.

Impatto dell’impiego dei patient-reported outcomes nella pratica clinica sull’accuratezza della registrazione in cartella clinica dei sintomi dei pazienti oncologici.

Riassunto. Introduzione. Le cartelle cliniche rappresentano un’importante sorgente di dati per la cosiddetta “real-world evidence”. Recentemente, abbiamo adottato l’impiego dei patient-reported outcomes (PRO) nella pratica clinica oncologica, dimostrando un beneficio significativo in termini di qualità di vita rispetto alla modalità di visita tradizionale. In questa analisi secondaria dello studio, descriviamo, per 5 sintomi, la concordanza tra quanto riferito dai pazienti nei questionari e quanto riportato dai medici in cartella. La nostra ipotesi era che l’adozione sistematica dei questionari comporti un miglioramento dell’accuratezza delle cartelle cliniche. Metodi. Erano eleggibili per lo studio pazienti in trattamento antitumorale attivo, in regime di day hospital. I pazienti nel gruppo di controllo sono stati sottoposti alle visite usuali (gruppo A), mentre i pazienti del gruppo B, prima di ogni visita, compilavano un questionario cartaceo per la registrazione dei PRO, allo scopo di fornire informazioni su sintomi e tossicità del trattamento. I medici non hanno ricevuto istruzioni specifiche sul riportare le informazioni contenute nei questionari nella cartella clinica. La concordanza tra quanto riferito dai pazienti e quanto riportato dai medici in cartella è stata valutata mediante il κ di Cohen, calcolando l’under-reporting come la proporzione di sintomi riferiti dal paziente ma non riportati in cartella dal medico. Risultati. L’analisi è stata condotta su 211 pazienti, per un totale di 412 visite. Per tutti i sintomi, il coefficiente κ di Cohen è risultato migliore per il gruppo B: emesi (0,25 gruppo A vs 0,36 gruppo B), diarrea (0,16 vs 0,57), stipsi (0,07 vs 0,28), dolore (0,22 vs 0,42), fatigue (0,03 vs 0,08). Per tutti i sintomi considerati, sebbene l’under-reporting sia risultato rilevante in entrambi i gruppi, esso è risultato inferiore nel gruppo B: emesi (75,49% vs 60,0%, p=0,031), diarrea (82,89% vs 50,0%, p<0,001), sti­psi (92,11% vs 69,57%, p<0,001), dolore (59,57% vs 42,31%, p=0,01), fatigue (82,11% vs 64,10%, p<0,001). Conclusioni. L’adozione di un questionario cartaceo per la raccolta dei PRO ha consentito una significativa riduzione dell’under-reporting dei sintomi nelle cartelle cliniche, ma la concordanza è rimasta subottimale. L’integrazione diretta dei PRO, raccolti mediante strumenti elettronici, nella cartella clinica potrebbe migliorare significativamente il problema dell’under-reporting, migliorando l’accuratezza delle cartelle cliniche come fonte di informazioni per la real-world evidence.

Parole chiave. Patient-reported outcomes, real-world evidence, tossicità, under-reporting.

Introduction

Patient-reported outcomes (PROs) are considered as the preferable instrument to report subjective symptoms1,2. PROs allow the description of a toxic effect caused by a treatment or a medical condition directly referred by the patient, without revision or interpretation from a clinician3. A rapidly growing number of studies has compared the report of symptoms coming from patients to those interpreted and reported from health professionals like a physician (in most cases) or a nurse4-9. The great majority of these studies demonstrated a limited agreement between patients and clinicians, with the latter significantly under-reporting the number and the entity of toxicities. In fact, in the “traditional” modality of visit, the collection and description of symptoms comes from an unstructured interview guided by the physician, and even if it does not necessarily imply an under-treatment, it causes an under-reporting of many symptoms, compared to the use of PROs3. The importance of integrating data coming from patients about symptoms and side effects of treatments has been largely demonstrated6-8. Several methods can be applied to reconciliate patients and clinicians’ report of toxicities; however, if reports from patients are shared with investigators in real time during the patients’ visit to the clinic, investigators can make use of this information to improve patients’ clinical management3.

In fact, PROs allow a more accurate description of treatment toxicity and symptoms, improving not only the predictive accuracy in outcome prognostication compared to clinicians’ reports, but also directly improving overall survival10,11. For instance, Basch et al. recently demonstrated a 5-month advantage in median overall survival when collecting symptoms directly from the patients compared to the group of patients who received the traditional modality of visit10. This aspect is probably the consequence of a precocious report of a clinical problem by the patient, which allows earlier interventions preventing more serious consequences potentially coming from a late or inadequate management of the problem. Consequently, an early proper management of symptoms could determine a longer prosecution of active treatments, with adequate doses and consequent possible increased efficacy of a treatment.

In recent years, the importance of real-world data has emerged; consequently, the utilization of PROs in the description of side effects and toxicities coming from treatments, thus directly communicated by the patients, is increasingly indicated not only to better clarify the symptoms caused by the disease itself or the toxicity profile of a well known treatment, but also as a valid instrument to produce labelling claims for new medical products. In fact, the point of view of the patient represents a crucial element in the definition of the safety profile of a drug; as an example, the PROSPER Consortium12 produced a document focalized on the importance of employing PROs to describe toxicity data both during clinical trials and post-approval drugs uses.

In order to improve the management of our patients receiving active anti-neoplastic treatments and consequently their outcomes, in 2018 we introduced in our daily out-patients practice a paper questionnaire, administered by a dedicated nurse to the patient in order to provide a direct description of the reported symptoms before the visit with the physician13. Although non-randomized, the comparison of a group receiving the questionnaire with a previous group, undergoing the standard “traditional” medical visit, demonstrated a significant improvement in quality of life (QoL) for the patients receiving the dedicated questionnaire. In addition, this study demonstrated the importance of the role of the nurses considering the more direct relationship with the patients. Overall, our experience confirmed that the use of PROs is easily feasible in daily clinical practice, and it is also capable of improving patients’ QoL, determining a high grade of satisfaction.

While the improvement in patients’ QoL has already been described in our previous paper, we decided to perform a new analysis exploiting the same database, in order to describe the concordance between physicians and patients in the description of symptoms, focusing on the entity of under-reporting by physicians into medical health records.

Materials and methods

Patients and procedures

As detailed before13, all the patients included in the analysis have been treated with an active anti-cancer treatment, as outpatients, at the Day Hospital of the Division of Medical Oncology, Mauriziano Hospital in Turin, Italy. Patients included in the control group (who were treated in 2017) underwent only “usual” medical visits, while patients treated in 2018 received by a dedicated nurse, before each visit, a specifically designed questionnaire, in order to systematically acquire patient-reported information about their symptoms and toxicities experienced during the treatment. The nurse explained to the patient the correct way of answering the questionnaire; subsequently, the nurse collected the questionnaire and delivered it to the physician, who could consult it before the visit.

Within the project, all patients received the European Organization for Research and Treatment of Cancer (EORTC) quality of life questionnaire (QLQ) C-30 at two specific time points14. The first assessment was performed when the patients had already received at least one administration of therapy, in order to be suitable for toxicity data collection. The second assessment was scheduled approximately one month after the first: a certain variability in timing of administration of questionnaires was due to the interval between visits determined by clinical practice. All patients signed a written consent before filling questionnaires.

Study objectives

The primary objective of the study, the comparison between the two patients’ groups in terms of QoL changes, is not discussed in this manuscript, as results have already been published13.

Aim of this secondary analysis was to describe patients’ and physicians’ reporting of five symptoms occurring during anticancer treatment, focusing on the agreement between patients’ and physicians’ reports and the rate of possible under-reporting by physicians into medical health records. Our hypothesis was that the adoption of the nurse-administered questionnaire could significantly improve the reporting of symptoms in health records by physicians.

Questionnaires

The systematic collection of symptoms and toxicities, based on the administration of the dedicated paper questionnaire, was started in January 2018. The questionnaire had been specifically designed by physicians and nurses. Although patients were not specifically involved in the design of the questionnaire, items were chosen according to the most common side effects observed across systemic therapies. In detail, the questionnaire contains 13 questions, corresponding to 13 symptoms/toxicities (mouth problems, nausea, vomiting, constipation, diarrhea, dyspnea, skin problems, nail problems, itching, hand/foot problems, fatigue, pain, other issues). Patient was asked to refer to the period elapsed since previous therapy, and a final question interrogated about the persistence of problems at the moment of the visit. All the questions had the same five response categories: “Not at all”, “A little”, “Quite a bit”, “Much” and “Very much”. When patients reported any pain, they had to fill a 11-point visual analogue scale, in order to better describe its intensity.

In both groups, the EORTC QLQ-C30 questionnaire was administered twice as mentioned above. EORTC QLQ-C30 is a 30-item questionnaire composed of five multi-item functional subscales (physical, role, emotional, social, and cognitive functioning), three multi-item symptom scales (fatigue, pain and emesis), a global health status subscale, and six single items to assess financial impact, dyspnea, sleep disturbance, appetite, diarrhea, and constipation, during the previous week. 

Nausea, vomiting, diarrhea, constipation and pain are assessed by one item each in the QLQ-C30 questionnaire: items 14 (have you felt nauseated?), 15 (have you vomited?), 16 (have you been constipated?), 17 (have you had diarrhea?), 9 (have you had pain?). Fatigue is assessed by two items: 12 (have you felt weak?) and 18 (were you tired?). These questions specifically refer to the previous week. The items are scored in four categories (not at all, a little, quite a bit, or very much).

Analysis of patients’ and physicians’ reporting of symptoms

For each patient, 2 visits were theoretically eligible for our analysis. Each visit was included if both health record of the visit and QoL questionnaires were available. In order to get a picture of symptoms reporting that could be similar to real life conditions, physicians received the 13-items questionnaire during the visit, and the record of patient’s symptoms into the health record was left to their judgment and routine practice. It is important to point out that physicians did not receive any specific training, nor were instructed to pay more attention to patient’s symptoms recording.

For the analysis of patient-reported symptoms, for each symptom, all patients’ responses in QoL questionnaire different from “not at all” (i.e., a little, quite a bit, or very much) were pooled together as “symptom reported by the patient”. In the case of fatigue, the worst response to items 12 and 18 was considered. Considering that nausea and/or vomiting had been generically reported by physicians as “emesis” in some health records, the 2 patient-reported responses were combined into “emesis”, considering the worst response to item 14 and 15.

Similarly, for the analysis of physician-reported symptoms, for each symptom, any severity reported by the physician in the health record of the visit was deemed “symptom reported by the physician,” whatever the severity and the description.

Agreement between patient and physician evaluations was assessed by Cohen’s κ15. Although there is no universal definition of the interpretation of κ values, according to Fleiss, κ values <0.40 can be interpreted as poor agreement, values between 0.40 and 0.75 as moderate to good agreement, and values >0.75 as excellent agreement16.

Under-reporting was calculated as the rate of cycles where physicians did not report the symptom in the medical record, out of cycles where patients reported any severity of the symptom in the QoL questionnaire7. Under-reporting was compared between group A and group B by chi-square test. All statistical tests were two-tailed and p-values less than 0.05 were considered statistically significant.

Analyses were performed with SPSS for Windows, version 25.0.

Results

Overall, out of the 211 patients theoretically eligible (422 visits), 10 visits were not retrieved due to a technical problem of the electronic health record and 412 visits were eligible for the analysis, 233 in group A and 179 in group B (figure 1). The main characteristics of patients included, in the whole series and separately in group A and group B, are summarized in table 1.

Table 2 describes patient reporting and physician reporting of symptoms, for all the visits in the whole series. In all cases, percentages of symptoms reported by patients were higher than those reported by physicians. Emesis (nausea or vomiting) of any severity was reported by patients in 172 (41.75%) of 412 questionnaires, diarrhea in 120 (29.13%), constipation in 206 (50.0%), pain in 219 (53.16%) and fatigue in 374 (90.78%). On the other hand, physicians reported emesis of any grade in 63 (15.29%) of 412 visits, diarrhea in 43 (10.44%), constipation in 40 (9.71%), pain in 133 (32.28%), and fatigue in 98 (23.79%). For the five symptoms considered in the analysis, Cohen’s κ ranged between 0.04 (fatigue) and 0.33 (diarrhea), which can be interpreted as poor agreement.

Table 3 describes the agreement between patients reporting and physician reporting, scattered by group (group A receiving “classic” visit and group B receiving the nurse-administered questionnaire for description of symptoms). For all symptoms, Cohen’s κ was better for group B, receiving the nurse-administered questionnaire, compared to group A, receiving “classic visit”. In detail, Cohen’s κ improved from 0.25 to 0.36 for emesis, from 0.16 to 0.57 for diarrhea, from 0.07 to 0.28 for constipation, from 0.22 to 0.42 for pain and from 0.03 to 0.08 for fatigue.

As shown in figure 2, in the whole series, the proportion of under-reporting by physicians (i.e., patients reported the symptom in the questionnaire, but physicians did not report the symptom in the health record of the visit) was 69.19% for emesis, 70.83% for diarrhea, 82.04% for constipation, 53.42% for pain and 74.60% for fatigue. For all symptoms, however, although under-reporting was numerically relevant in both groups, reporting was improved for group B compared to group A. In detail, under-reporting improved from 75.49% to 60.0% for emesis (p=0.031), from 82.89% to 50.0% for diarrhea (p<0.001), from 92.11% to 69.57% for constipation (p<0.001), from 59.57% to 42.31% for pain (p=0.01) and from 82.11% to 64.10% for fatigue (p<0.001).

Reporting was improved for group B compared to group A independently of the severity of the symptom referred by the patients. In detail, when limiting the analysis to the subgroup of visits when patients referred a symptom as “quite a bit” or “very much”, under-reporting improved from 57.1% to 42.9% for emesis, from 78.6% to 38.9% for diarrhea, from 87.2% to 43.8% for constipation, from 41.2% to 29.4% for pain and from 77.0% to 57.4% for fatigue. On the other hand, when limiting the analysis to the subgroup of visits when patients referred a symptom as “a little”, under-reporting improved from 80.2% to 64.3% for emesis, from 85.4% to 57.7% for diarrhea, from 94.7% to 83.3% for constipation, from 70.0% to 52.3% for pain and from 87.6% to 68.4% for fatigue.

Discussion and conclusions

In this study, we confirm that underreporting of symptoms by physicians in patients’ health records is common when the traditional standard visit is performed, even for some symptoms (such as pain or diarrhea) that clinicians are generally more concerned of. A noteworthy degree of disagreement was evident also when patients reported severe symptoms. Through the use of a simple 13-item questionnaire, without a specific training of the medical staff, we were able to improve the agreement and the accuracy of health medical records, for all the symptoms analyzed. Furthermore, as we have previously described, we could demonstrate through a satisfaction survey that the adoption of the questionnaire was well accepted by the patients. Despite this improvement, however, for none of the symptoms the agreement was optimal, drawing attention to the necessity of further improvement.

The adoption of PROs is an established and effective method to directly report patient experience. In fields such as oncology, where the impact of treatments can strongly affect subjects’ QoL, the integration of PROs is of major importance. As a consequence, there is growing interest in implementing this field, both into clinical practice and clinical research17,18. As previously pointed out, several studies have described a low agreement between clinicians and patients when reporting subjective toxicity, resulting in a significant underestimation of incidence and severity of toxicities4-9.

In some cases, while filling health records, physicians may have deliberately omitted some symptoms reported by patients in the PROs questionnaires, based on the subsequent discussion occurred during the visit. For instance, this could apply to some symptoms, such as diarrhea, where sometimes the patient’s subjective experience does not meet the technical definition of the adverse event. However, we judge highly unlikely that such a deliberate omission could justify the wide amount of underreporting observed.

As we already discussed reporting primary objective of the project13, we recognize that our observational analysis is methodologically weaker than a theoretical randomized trial comparing 2 different modalities of patient-physician communication. However, following the results of the trial by Basch10 and other similar experiences, we convinced ourselves that randomization was no more acceptable, because we believed that adoption of PROs should have been considered part of clinical practice, and not experimental anymore. Consequently, we decided to perform an observational analysis, describing the introduction of PROs in our clinical practice, using a cohort of patients treated immediately before as control group. The second group was treated just few months after the first, by the same group of physicians and nurses, and the 2 groups of subjects were quite similar in terms of age, type of tumor and type of treatment.

In recent years, the use of real-world evidence has been increasingly considered, not only for description of treatment effectiveness in clinical practice after the conduction of randomized clinical trials, but also for description of treatment toxicity. From this perspective, after the introduction of new treatments in clinical practice, real-world data could play a substantial role in the optimal and accurate description of tolerability and adverse events associated with administration of anticancer treatments. It is not surprising that, in recent years, safety analyses based on real-world evidence have been increasingly cited in the National Comprehensive Cancer Network and the American Cancer Society practice guidelines19. However, our data suggest that the accuracy of data sources potentially used for the acquisition of real-world evidence (patients’ health records) – at least in terms of description of symptoms and subjective adverse events – could be limited. On one hand, our results suggest that the adoption of PROs in clinical practice can significantly increase the accuracy of health records in terms of description of symptoms and toxicities. On the other hand, however, the improvement observed with the adoption of PROs is still suboptimal and, for all symptoms considered in our analysis, the real burden declared by patients is much higher than the description made by clinicians in health records.

How could further, substantial improvement in accuracy of data sources be obtained? While in our experience the paper-based PROs were read by physicians but not necessarily transcribed into patients’ health records (no specific instruction was given to physicians to pay more attention to symptom recording), the direct incorporation of PROs into electronic health records could completely resolve the problem of under-reporting. In the past decades, the advances of technology have been rapidly changing the landscape of possible intervention in medicine20. Internet-based eHealth technologies, smartphone apps and telemedicine are not only means to reach unserved populations or to empower patients to actively participate in their own care21,22, but they should be exploited by clinicians to facilitate doctors-patient communication, symptoms management and ameliorate assistance.

In this perspective, many health centers worldwide are integrating PROs into clinical practice, often on a national/regional-based extent and with the support of the Ministry of Health or other institutions23-25. Some challenges can be identified from these experiences: the need of secure and robust informatic tools, the identification of patient’s pathway in order to select the most suitable questionnaire, patients and personnel training, data entry, definition of a clinical algorithm for severity-tailored advices, clinicians’ perception of a higher burden of work.

Some instruments are already accessible, such as in the case of PROMIS® (Patient-Reported Outcomes Measurement Information System), a NIH- funded initiative officially launched in 200426. It includes over 300 measures of physical, mental, and social health for use with the general population and with individuals living with chronic conditions. PROMIS can be assessed through a short form (basically a standard questionnaire), or through computer adaptive testing (CAT); in the latter, the first question asked is typically the same for all patients, then an algorithm chooses the next item based on the previous response. CAT maximizes efficiency by reducing the burden of responding and progressively reduces the standard error after every question administered, thus being more accurate.

Guides about integrating PROs into clinical practice are available and they consider opportunities and limitations of different issues (such as degree of integration with the electronic health record, access to information, legal issues, candidate PROs measures, clinical algorithms, role of personnel), in order for the user to choose the option that best fits each specific situation27,28.

In conclusion, in our study we used a dedicated paper questionnaire that was cost effective, associated with a significant improvement in QoL and positively accepted by the patients; this allowed a significant reduction in the under-reporting of symptoms by clinicians in patients’ health records, compared to traditional visit. However, the degree of agreement was still not optimal. We believe that an electronic tool directly integrated in the health record would minimize the issue of underreporting, allowing a more proactive management of symptoms, transforming patients’ health records in a more accurate source of data about symptoms and treatment tolerability29, and finally allowing a better care for the patients.

Acknowledgments: the authors wish to thank all the patients who were included in the analysis, and their caregivers.

Funding and conflict of interests: Massimo Di Maio is recipient of a research funding from the CRT Foundation (Turin, Italy) for this project on the impact on quality of life of the systematic evaluation of toxicity with patient-reported outcomes in patients with solid cancer (CRT grant number 46333, “Richieste ordinarie 2015”). This research funding allowed the attendance of Donatella Marino, Chiara Baratelli and Carmela Giovanna Cleopatra Turco at Ordine Mauriziano Hospital. Other authors declare no conflict of interests.
Authors have full control of all primary data and agree to allow the journal to review their data if requested.

References

1. U.S. Food and Drug Administration. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. (2009) Available on: https://bit.ly/369J8jP [accessed 17 October 2020].

2. European Medicines Agency. Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products. (2005) Available on: https://bit.ly/3o45Dgd [accessed 17 October 2020].

3. Di Maio M, Basch E, Bryce J, Perrone F. Patient-reported outcomes in the evaluation of toxicity of anticancer treatments. Nat Rev Clin Oncol 2016; 13: 319-25.

4. Petersen MA, Larsen H, Pedersen L, et al. Assessing health-related quality of life in palliative care: comparing patient and physician assessments. Eur J Cancer 2006; 42: 1159-66.

5. Grossman SA, Sheidler VR, Swedeen K, et al. Correlation of patient and caregiver ratings of cancer pain. J Pain Symptom Manage 1991; 6: 53-7.

6. Fromme EK, Eilers KM, Mori M, et al. How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30. J Clin Oncol 2004; 22: 3485-90.

7. Di Maio M, Gallo C, Leighl NB, et al. Symptomatic toxicities experienced during anticancer treatment: agreement between patient and physician reporting in three randomized trials. J Clin Oncol 2015; 33: 910-5.

8. Cirillo M, Venturini M, Ciccarelli L, et al. Clinician versus nurse symptom reporting using the National Cancer Institute-Common Terminology Criteria for Adverse Events during chemotherapy: results of a comparison based on patient’s self-reported questionnaire. Ann Oncol 2009; 20: 1929-35.

9. Basch E, Iasonos A, McDonough T, et al. Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: results of a questionnaire-based study. Lancet Oncol 2006; 7: 903-9.

10. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017; 318: 197-8.

11. Basch E, Bennett A, Pietanza MC. Use of patient-reported outcomes to improve the predictive accuracy of clinician-reported adverse events. J Natl Cancer Inst 2011; 103: 1808-10.

12. Banerjee AK, Okun S, Edwards IR, et al. Patient-Reported Outcome Measures in Safety Event Reporting: PROSPER Consortium guidance. Drug Saf 2013; 36: 1129-49.

13. Baratelli C, Turco CGC, Lacidogna G, et al. The role of patient-reported outcomes in outpatients receiving active anti-cancer treatment: impact on patients’ quality of life. Support Care Cancer 2019; 27: 4697-704.

14. Aaronson NK, Ahmedzai S, Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993; 85: 365-76.

15. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20: 37-46.

16. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd Edition. Hoboken, NJ: Wiley, 1981.

17. Basch E, Rogak LJ, Dueck AC. Methods for implementing and reporting Patient-reported Outcome (PRO) measures of symptomatic adverse events in cancer clinical trials. Clin Ther 2016; 38: 821-30.

18. Mooney K, Berry DL, Whisenant M, Sjoberg D. Improving cancer care through the patient experience: how to use patient-reported outcomes in clinical practice. Am Soc Clin Oncol Educ Book 2017; 37: 695-704.

19. Wu TH, Yang JC. Real-world or controlled clinical trial data in real-world practice. J Thorac Oncol 2018; 13: 470-2.

20. Tarver WL, Haggstrom DA. The use of cancer-specific patient-centered technologies among underserved populations in the United States: systematic review. J Med Internet Res 2019; 21: e10256.

21. Ventura F, Ohlen J, Koinberg I. An integrative review of supportive e-health programs in cancer care. Eur J Oncol Nurs 2013; 17: 498-507.

22. Dedding C, van Doorn R, Winkler L, Reis R. How will e-health affect patient participation in the clinic? A review of e-health studies and the current evidence for changes in the relationship between medical professionals and patients. Soc Sci Med 2011; 72: 49-53.

23. Dudgeon D, King S, Howell D, et al. Cancer Care Ontario’s experience with implementation of routine physical and psychological symptom distress screening. Psychooncology 2012; 21: 357-64.

24. Holch P, Warrington L, Bamforth LCA, et al. Development of an integrated electronic platform for patient self-report and management of adverse events during cancer treatment. Ann Oncol 2017; 28: 2305-11.

25. Girgis A, Durcinoska I, Arnold A, Delaney GP. Interpreting and Acting on the PRO Scores From the Patient-reported Outcomes for Personalized Treatment and Care (PROMPT-Care) eHealth System. Med Care 2019; 57 Suppl 5 Suppl 1: S85-S91.

26. HealthMeasures: Promis. Available on: https://bit.ly/2V25oFE [accessed 29 January 2020].

27. Patient-Centered Outcomes Research Institute. User’s Guide to Integrating Patient-Reported Outcomes in Electronic Health Records. (2017) Available on: https://bit.ly/2HHaKTY [accessed 17 October 2020].

28. Aaronson N, Elliott T, Greenhalgh J. International Society for Quality of Life Research. User’s guide to implementing Patient-Reported Outcomes assessment in clinical practice. (2015) Available on: https://bit.ly/2J8msY1 [accessed 17 October 2020].

29. Calvert M, Thwaites R, Kyte D, Devlin N. Putting patient-reported outcomes on the ‘Big Data Road Map’. J R Soc Med 2015; 108: 299-303.