How to generate better patient-reported outcomes data

PRO instruments must have “content validity,” meaning they must be relevant and specific to the disease being studied. That requires a clear understanding of the natural history of the disease, the key symptoms that may improve from treatment, and an appropriate length of time to see a clinical benefit. Specify and define concepts—such as signs, symptoms, and impacts—important to the target patient population and likely to demonstrate meaningful and interpretable changes in clinical trials. Seek input from patients and expert clinicians.

It takes time—sometimes years—and effort to do this right. At Parexel, we are currently working with a sponsor to design a PRO for a rare pediatric central nervous system disorder. Clinical trials of this disease traditionally use the primary endpoint of reducing seizures. But seizure control is not the only symptom critical to quality of life in this setting: these children often cannot feed themselves or verbalize, functions that families, caregivers, and patients prize. So, we are developing a new PRO to quantify changes in self-feeding and verbalization. We have worked closely with patients and caregivers and scoured the medical literature to determine what would represent a meaningful change in vocalization and how to measure it precisely. We are combining modules from multiple already-validated PROs into a novel instrument we can validate in this disease and use in clinical trials.

Initially, we will use the new PRO as a secondary or exploratory endpoint in a Phase 1/2 trial. If successful, we will use it for the pivotal trial. We hope to gain FDA guidance and acceptance of this approach during Type B and C meetings. Developing a new PRO requires careful planning, strong rationale, and high-quality evidence. 

Sponsors must also ensure that the patient population can validly and reliably self-report (some patients may be too young or sick) and select a PRO that accommodates a spectrum of cognition and mobility levels. Also, make sure the recall period for a PRO is at most two weeks. Otherwise, the FDA may question the validity of the results. 

For statistical validity and comparability, the mechanism used to report PRO data should be consistent throughout a trial. A drug recently approved by the FDA for an orphan metabolic disorder used a daily questionnaire to measure patients’ hunger levels. However, partway through the pivotal study, patients switched from answering the questions on paper to using an e-diary. The sponsor did not subject the new e-diary device to usability testing or conduct patient cognitive interviews to assess its functionality, questionnaire comprehension, and ease of use. The FDA concluded there was insufficient data to demonstrate that patients understood one of the questions. Although the PRO data, in this case, made it onto the label (the PRO was the primary endpoint of the study), it’s a cautionary tale for other sponsors.

FDA reviewers who analyzed and rejected PRO data contained in orphan drug applications from 2017-2022 frequently noted there was no prespecified statistical analysis plan (SAP) or correction for Type 1 errors (results due to chance). One way to avoid this problem is to agree with the agency on the SAP before initiating a pivotal trial. When a PRO is designated as a secondary endpoint, a hierarchical testing framework with an alpha level (the level of significance required to show that the result is not due to chance) adjusted for multiple comparisons can help ensure valid results. 

Another considerable problem cited by the FDA was missing data due to death, disease progression, or a high dropout rate for the trial. Sicker patients and those who do not respond to treatment are less likely to complete patient-reported quality-of-life instruments as a trial proceeds, skewing results. In one orphan cancer trial that enrolled heavily pretreated patients with advanced disease, patient completion of a 36-item health-related quality of life (HRQoL) questionnaire dropped so precipitously from baseline (about 90%) to week 48 (about 30%) that the data were deemed invalid by FDA reviewers.

During a 2022 FDA conference on patient-focused drug development (PFDD), senior clinical analysts from the agency noted that sponsors should have a plan to understand why data is missing and how to handle it.³ To address potential bias and interpretability issues due to missing data, sponsors can plan a worst-case scenario analysis in which they manage patients with missing HRQoL data differently. More sophisticated statistical methods, such as pattern mixture models, may further elucidate the effect of missing data on study findings.

Although missing data is inevitable in trials, sponsors can take steps to prevent it. If the burden of filling out questionnaires and e-diaries for multiple PROs in a single study is too high, patients will not complete them. Sponsors thus need to ensure that study designs make sense, are practical, and will achieve the goals outlined in the protocol.

Of the 21 orphan drug labels from 2017-2022 that contained PRO data, just four (19%) reported data from open-label studies.

Most PRO data the FDA approved for inclusion on product labels were generated in blinded, controlled studies.

PRO data from open-label or single-arm studies has limited interpretability. That’s because treatment effects can be systematically over- or under-estimated by patients who know the treatment assignment. 

Randomized, controlled trials (RCTs) are the most efficient way to neutralize confounding factors and, therefore, the best way to study risks and benefits. Double-blinded assignment to treatment and control can provide unbiased comparisons for known and unknown factors. Sometimes, a comparator arm is neither feasible nor ethical for some modalities, such as cell and gene therapies with remarkable activity and indications, such as advanced cancers or ultra-rare diseases with no existing treatments. Use of active controls, external control arms, natural history studies, and historical controls as comparators can help.

However, even a single-arm open-label trial can collect useful PRO data. In our analysis of orphan drugs, we found a CAR-T cell therapy for cancer that the FDA approved based on a single-arm, open-label Phase 2 study. This pivotal study captured three different PROs as secondary endpoints. Although the PRO data did not meet the FDA’s standard for inclusion on the product label, the data did appear on the EMA label and was later published in a peer-reviewed journal.⁴ An editorial accompanying the journal report stated: “…the authors should be commended for their efforts to provide the scientific community with unprecedented PROs information on the burden of this CAR T-cell therapy. Despite several open questions, which should be elucidated in further studies, the results reported in their article are highly encouraging and hopefully will stimulate other high-quality research initiatives in this area.⁵