Imaging endpoints are used in clinical trials to determine drug safety and efficacy. Central imaging reviewers (e.g., radiologists and nuclear medicine physicians) evaluate imaging biomarkers by selecting and measuring lesions over time, according to guidelines described in their facility’s Independent Review Committee Charter.
However, reading an image is inherently subjective and—even with standards, automatic procedures, and artificial intelligence—measurement differences can occur between two or more readers. Still, reader variability can be reduced in several ways. Here we describe one of them: the Calibration Meeting.
What is a Calibration Meeting?
A calibration meeting is a gathering that usually occurs before the start of the central imaging review. Central readers, key staff from the imaging core lab (ICL), and representatives of the sponsor and/or CRO confer either face to face or via teleconference.
The aim of the calibration meeting is to reduce reader variability by harmonizing the reading process among the readers. Training cases are provided to the readers in attendance to review their understanding of the read criteria. The differences in the reads of these cases are discussed as a team to help bring the readers into alignment on the read criteria, selection of lesions, etc. The meeting is also an opportunity to highlight common questions and troubleshoot challenging imaging issues. Attendees discuss the “what-if’s” of certain types of findings, ultimately leading to more accurate reading assessments. The goal is to build consensus among the readers to reduce read discordance and variability.
An Example in the Spotlight
A recent study involving neuroendocrine tumors (NETs) highlights the importance of calibration meetings. For a NET, lesions can be found in multiple tissues and organs. RECIST 1.1 guidelines require only 4 or 5 lesions be selected, but at least 5 tissue types are involved in NETs. The issue during the calibration read was how the tumor location should be prioritized for the identification of target lesions. After significant discussion involving more review of the eligibility criteria, the readers chose a prioritized sequence of anatomical locations to be selected for the identification of target lesions. Without the calibration meeting, each of the readers would have assumed their approach was the correct one and the determination of target lesions would have varied between the readers.
Additional Benefits
- Study specifics are taken into account
Even when following the same guidelines, readers can assess images differently from one study to another depending on the pathology or the drug’s mechanism of action. The calibration meeting is the opportunity for everyone to discuss their experience and to build a consensus that is based on the best way to reach the study’s imaging endpoints.
- Readers are no longer “lost in translation”
Multisite trials can involve international readers. Protocols and other study-related documents (e.g., training documents, review charters, etc.) can be misunderstood due to varying definitions and nuanced meanings of words that can differ from country to country, even if all the readers speak the same language. Subtle misinterpretation can cause reading errors. A calibration meeting held early on during the study prevents data from the first few subjects from being misinterpreted and having to be re-read. According to a blog on this topic by The Bracken Group, “it is especially important to conduct a calibration meeting in the case of internationally mixed studies as there are nuances in the terminology of protocols and image read documents that can be missed due to language barriers.”
- Adjudication rates are minimized
One of the key aspects that the US Food and Drug Administration (FDA) considers is the adjudication rate in a calibration read. The FDA expresses its opinion that “Adjudication is an element of a successful development program.” It is critical to ensure the adjudication rate in a clinical trial is minimized and correctable errors of interpretation or use of the software are resolved before the start of the clinical reads. This goal can be met in the calibration meeting.
How Best to Conduct Calibration Meetings?
Scheduling a meeting of the readers, the staff of the ICL, and the sponsor can be a logistical nightmare. Use of current technology to conduct the meeting is the optimal solution to ensure that this integral part of the trial occurs.
Here is a typical agenda for a calibration meeting:
- Introductions
- Presentation by the sponsor of the protocol and trial design
- Presentation by the sponsor of the key imaging endpoints and the rationale and nuances that have been discussed
- Overview of the read system and the software
- Breakout session for the readers to read and evaluate 5+ practice cases independently
- Regroup to discuss the images read and any disagreements in interpretation
- Document conclusions and read nuances in a reader reference manual for the study
Conclusion
Calibration meetings can eliminate errors, and thus tremendously aid in meeting the appropriate endpoints and in collecting more data for your trial. Reducing variability in clinical trials is important. An expert imaging company such as Keosys has experience in organizing these meetings and can provide you with all the necessary advice and support.
