Rater Training and Surveillance: Reducing Alzheimer’s Research Failure Rates

Alzheimer’s disease research often fails due to problems with the drug target or the agent being tested.

In some clinical cases, however, methodological imprecision is the culprit.¹ To avoid pitfalls in methodology, Worldwide implements lessons learned in clinical trials from our long history of tenaciously fighting Alzheimer’s disease.

Imprecision manifests itself in several ways, including diagnostic inaccuracy during trial patient enrollment. Clinical trials focusing on early-stage Alzheimer’s disease have particularly high screen failure rates.² This is especially disturbing, given that Alzheimer’s disease research increasingly targets early-stage patients.

Unfortunately, the issues that contribute to diagnostic inaccuracy at patient screening may also detract from assessments and data quality during study conduct through inconsistent approaches to administering and scoring outcome measures. Failed clinical trials, although costly for pharmaceutical companies and heartbreaking for patients and families, do yield important lessons for Alzheimer’s disease research – lessons that may ultimately reduce the failure rate.

3 Rater Reliability Lessons in Alzheimer’s Disease Clinical Trials

Here are three lessons that relate directly to the diagnostic decisions made and overall data quality in an Alzheimer’s clinical trial.

1. Rater variability is a significant source of unpredictability in Alzheimer’s disease research. In particular, assessments that are repeatedly administered are vulnerable to both practice effects and increased variability due to subject responses, idiosyncratic administration, scoring interpretations and errors, as well as various rater and patient biases and expectations.
2. It’s accepted that rater training is crucial for the success of a clinical trial, both in terms of appropriate patient eligibility and quality data outcomes. Initial training procedures for inclusion and efficacy measures must be thorough and clear to ensure that raters have the experience and knowledge to administer and score assessments correctly. But it’s less commonly understood that ongoing surveillance and refresher trainings throughout the life of the Alzheimer’s clinical trial is equally as essential. This rater monitoring and retraining is to address concerns about rater drift, which is the tendency to deviate from standardized scale administration and scoring conventions. Rater drift is a common phenomenon, especially in global clinical trials.
3. Rater training should be focused on all study scales but needs to be detailed enough to provide item-by-item administration and scoring guidance.

Worldwide has found a way to turn these insights into action through our rigorous multi-stage rater training, certification, and surveillance process.

Rater Training Reduces Clinical Trial Failure Rates

Worldwide’s Rater Training and Certification program assesses rater experience and knowledge with study indication and scales. We then build on these foundations with comprehensive rater training at the start of a clinical trial as well as ongoing supervision and training through our data surveillance program.

As a result, we’ve successfully reduced screen failure rates in Alzheimer’s disease research. In one case, an ongoing study in prodromal Alzheimer’s patients, we brought it down to 50 percent, well below the 80 percent average.⁴

In another case – a multinational study evaluating an investigational drug for mild cognitive impairment – overly strict inclusion criteria led to a high screen failure rate. Worldwide deployed a therapeutic team to the sites and conducted regular teleconferences and web-based seminars. These interventions resulted in improved adherence to protocol- and scale-specific instructions, which led to a dramatic increase in enrollment.⁵

Finally, we have found that expert clinical feedback on scoring, protocol adherence, and administration competency can decrease errors and potentially reduce rater drift and data variability. Worldwide researchers looked at 56 raters at 46 U.S. centers across 285 screening visits and discovered that 67 percent of the source documentation included errors. However, over the course of the trial, errors decreased by 20 percent after ongoing feedback, remediation, and refresher training. Next, researchers plan to determine which methodology leads to the greatest reduction in error variance and rater drift.⁵

These lessons apply to any Alzheimer’s disease clinical trial. There’s always the possibility that a rater will screen subjects inappropriately or too strictly. Ongoing rater training and surveillance can improve patient recruitment and reduce rater errors, leading to more successful clinical trials.

Learn More About Worldwide’s Rater Services and Alzheimer’s Disease Research

Worldwide’s uncommon approach to patient enrollment and rater training helps our clients avoid the pitfalls of an underperforming study. Learn more about our Clinical Assessment Technologies services, rater-training and certification services, and our Alzheimer’s disease research or contact Worldwide Clinical Trials directly.

For more clinical trial operations insights, please download our eBook, 10 Steps to Better Alzheimer’s Disease Research.

About the author

Rolana Avrumson has over 19 years of clinical experience, including 10 years in clinical trial research. As the Director of Clinical Projects for the Worldwide Clinical Trials Clinical Assessment Technologies team, her primary responsibilities include oversight of the operational and clinical conduct of assigned studies, including protocol review, rater and CRA training, subject eligibility reviews, data surveillance, sponsor reporting, and scale management.

Ms. Avrumson has held significant roles in global and domestic phase Ib/II/III trials in a number of CNS indications with pediatric, adult, and geriatric populations. These indications include neurocognitive disorders (e.g. Alzheimer’s Disease, MCI), psychiatry (e.g. addiction, mood disorders, psychotic disorders, anxiety disorders, ADHD), movement disorders (e.g. Parkinson’s), and various pain indications (e.g. neuropathic pain and arthritis trials). As a Clinician in applied practice, she has specialized in areas of Mood, Anxiety, Psychotic, Movement and Neurocognitive Disorders. Ms. Avrumson also has over 15 years’ experience administering, scoring, and analyzing psychiatric and cognitive measures for these indications.

Ms. Avrumson has a Master’s Degree in Rehabilitation Counseling Psychology with a specialty in Substance Dependence from the University at Albany, Albany, NY and a Bachelor’s Degree in Clinical Psychology from Tufts University, Medford, MA.

References

1. Posner H, Curiel R, Edgar C, et al. Outcomes assessment in clinical trials of Alzheimer’s disease and its precursors: Readying for short-term and long-term clinical trial needs. Innovations in Clinical Neuroscience. 2017; 14(1–2):22–29.
2. Babic T, Riordan H. Improving screen failure and recruitment rates in Alzheimer’s disease clinical trials. Journal for Clinical Studies. Oct. 2016
3. Cummings J, et al. Re-engineering Alzheimer’s clinical trials: Global Alzheimer’s platform network. The Journal of Prevention of Alzheimer’s Disease. 2016; 3.2: 114–120.
4. Friedmann B, et al. The impact of site rater monitoring and refresher training on enrollment in a MCI phenotype trial. European Neuropsychopharmacology. 20(3). August 2010.
5. Avrumson R, et al. “The utility of data surveillance in Alzheimer’s trials: Reducing variability and rater drift.” Poster presented at: The Annual CTAD Meeting; November 2017; Boston, MA.